In this video I speak with Andrew Lamb, Staff Software Engineer @Influxdb. We discuss FDAP (Flight, DataFusion, Arrow, Parquet) stack for modern OLAP database system design. Andrew shared some insights into why the FDAP stack is so powerful in designing and implementing a modern OLAP database.

Chapters:00:00 Introduction01:48 Understanding Analytics: Transactional vs Analytical Databases04:41 The Genesis and Goals of the FDAP Stack09:31 Decoding FDAP: Flight, Data Fusion, Arrow, and Parquet12:40 Apache Parquet: Revolutionizing Columnar Storage17:18 Apache Arrow: The In-Memory Game Changer23:51 Interoperability and Migration with Apache Arrow27:10 Comparing Apache Parquet and Arrow28:26 Exploring Data Mutability in Analytic Systems29:19 Handling Data Updates and Deletions29:24 The Role of Immutable Storage in Analytics30:42 Optimizing Data Storage and Mutation Strategies34:20 Introducing Flight: Simplifying Data Transfer35:02 Deep Dive into Flight's Benefits and SQL Support39:20 Unpacking Data Fusion's SQL Support and Extensibility46:12 The Interplay of FDAP Components in Analytics51:49 Future Directions and Innovations in Data Analytics56:04 Concluding Thoughts on FDAP and Its Impact

FDAP Stack: https://www.influxdata.com/glossary/fdap-stack/FDAP Blog: https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/InfluxDB: https://www.influxdata.com/

Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator

If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet.

Database internals series: https://youtu.be/yV_Zp0Mi3xs

Popular playlists:Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-

Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17

Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d

Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN

Stay Curios! Keep Learning!

#datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign