F3: Decoupled Layout and WASM Enable Extensible Data Format

Key Quotes

"The full name of F3 is future proof file format and as the name indicates we aim to build a next generation columnar file format that is future proof in order to achieve this goal when we designed the format at the beginning we tried to make it efficient interoperable and extensible at the same time that's the basic principle when we designed the format or the high level goals."

Xinyu Zheng explains that the F3 project aims to create a next-generation columnar file format designed for future adaptability. The core principles guiding its design are efficiency, interoperability, and extensibility, addressing limitations in existing formats.

"The first thing is that many previous assumptions simply do not hold anymore for example in the in the old hadoop era when all your storage is hard drives connected by slow network when you design a file format what you what you want is better compression ratio and smaller files because that typically means better query performance and the reason is that the bottleneck of the whole query stack is on io right and smaller files just save io bandwidth and right now since they are different you cannot only optimize for for smaller files for better compression ratios you have to also think about the overhead when you are doing the decompression like how much computational cost you have to pay when you are reading the data back from the file right because now the cpu is the bottleneck instead of the io therefore the encodings and compression inside the file format has to be changed so basically which means it has to be lightweight by lightweight i mean the decoding and the decompression to consume two more cycles to make the process a bottleneck."

Xinyu Zheng highlights how hardware evolution, specifically faster storage and stagnant single-core CPU performance, necessitates a shift in file format design. Zheng argues that optimizing solely for compression and smaller file sizes, as done in older formats, is no longer sufficient; lightweight decoding that minimizes CPU overhead is now crucial.

"The recent focus on vector data is perhaps the most visible change in storage requirements. What are some of the other custom types of data that might need to be supported in the file storage layer? ... Besides vectors i think there are also other data types like images audio and video those data types actually impose a challenge on the file format if you really want to store them in place in a file format so first those data pads are much larger than the traditional ones so think of like integer is just 8 bytes right and strings are maybe 10 bytes or like tens of bytes but the image is in the level of megabytes and video is in hundreds of megabytes or even gigabytes so when you fit them into a single file you will have to make sure those data with a large variety of bits do not mess up with each other like when you are storing those images those videos those audios you are going you are also going to store some metadata like some descriptions some feature flags you you your model got from those images and videos right and those features those flags those metadata are just traditional data it's integers or like string or float but you you are going to want to store them together and then there comes a problem that how to align those small data together with those large data and and how you should partition the file so that when you are querying those two types of data together your query performance will not get affected and also when you are storing those two type of data together you still have good compression ratio."

Xinyu Zheng discusses the challenges posed by multimodal data, such as images, audio, and video, in file storage. Zheng explains that these large data types, alongside traditional metadata, require careful organization within a file to maintain query performance and compression efficiency.

"The key challenge of introducing a new format is that of adoption. What are the provisions in F3 that might simplify the adoption of the format in the broader ecosystem? (e.g. integration with compute frameworks) ... I think first the the self decoding ability I just mentioned is actually a very very attractive feature and is actually a strings for F3 to be integrated into the broader ecosystem because once you you'll be integrated it's much easier than other formats to get evolved right for other formats you for example when you are proposing a new encoding you have to go go through the process of like submitting a pull request on github getting it merged and then you have to make sure all the readers are upgrading to the new version so that it can use the new feature but right now in F3 we have this kind of like forward compatible ability where old readers can even adopt the new features the new encodings so I think that's actually a very attractive feature for F3 to to be get adopted."

Xinyu Zheng identifies the self-decoding capability as a key feature for F3's adoption. Zheng explains that this allows older readers to utilize new features and encodings, simplifying the adoption process compared to formats that require universal upgrades to leverage new functionalities.

"The second thing is that we want to make the encoding what which is that I just mentioned like inside the data block you are going to need to have some encoding and compression to make the data block as small as possible right but we found that in parquet although in the past few years parquet also added some new encodings in the file format and in the academia we see that there are many different research on proposing new encodings but none of those new encodings are getting used by users today so that's the problem and we are thinking about how to solve it right how to solve it in a way that is forward compatible and then we came out to came up to the idea that we can actually make the file format self decoding."

Xinyu Zheng addresses the challenge of new encodings not being adopted in existing formats like Parquet. Zheng proposes making the file format "self-decoding" as a solution, enabling forward compatibility and easier integration of new encoding algorithms.

"The biggest gap is that when people are designing software or technology they should really try to make it future proof as the as the name in the F3 project right it has to be forward looking because when we are designing of software we don't want it to to be only fit in the current workload or the current hardware right we want it to to be reused in future or like survive for a long time yeah I think that's that's the biggest lesson I learned from this project and I also think that's a gap for many technologies."

Xinyu Zheng concludes by stating that the most significant gap in current software and technology design is the lack of a "future-proof" approach. Zheng emphasizes the importance of forward-looking design that allows technologies to remain relevant and reusable across evolving workloads and hardware.