In this video, @mehdio will do a walkthrough of DuckDB, Polars and Pandas. We will discuss the main features and dive into a pragmatic code example. ☁️🦆 Start using DuckDB in the Cloud for FREE with MotherDuck : 📓 Resources Github Repo of the tutorial : DuckDB getting started video: ➡️ Follow Us LinkedIn: Twitter : Blog: 0:00 Intro Why use DuckDB in Python with powerful libraries like Pandas and Polars? This video explores when to use DuckDB alongside these mainstays by comparing their core features for data analysis pipelines. We'll run a head-to-head performance benchmark on a large data engineering task to see which is fastest. Are they competitors or collaborators? 0:34 What is DuckDB How does DuckDB fit the Python ecosystem? It's a fast, in-process OLAP database running directly in your app—no server required. We explore its columnar engine, extension system for S3/JSON without extra Python packages, and native Apache Arrow support. This lightweight, single-file database simplifies your data stack while delivering incredible performance. 2:46 What is Pandas Pandas is the standard for data analysis in Python. We revisit this essential library, covering its features and the evolution to Pandas 2.0, which adopted an Apache Arrow backend for major performance gains. We also cover its strong support for formats like CSV and Parquet and its deep integration with visualization libraries, making it a go-to for data manipulation. 3:45 What is Polars Polars is a new DataFrame library built in Rust for high performance. It uses a multi-threaded architecture for speed, and its efficiency with larger-than-memory datasets comes from a powerful lazy evaluation engine. We explain how lazy evaluation optimizes query plans and reduces memory, making Polars a top contender for high-performance data manipulation. 5:12 Code project We're testing these tools on a real-world data engineering project: analyzing a 33 million row (5GB) Parquet dataset of Hacker News posts. We'll walk through the simple ETL pipeline that DuckDB, Pandas, and Polars will each execute. This practical example will serve as the basis for our performance and syntax comparisons for large dataset analysis in Python. 6:14 Install & dependencies A project's dependencies impact container size and maintainability. We compare the installation footprints of DuckDB, Pandas, and Polars. DuckDB stands out with minimal Python dependencies, relying on a self-contained extension system for features like S3 access. This design makes it a lightweight addition to build efficient data applications. 7:18 Versatility How flexible are these tools outside Python? We explore the versatility of DuckDB, Pandas, and Polars. DuckDB shines with client APIs for Java and Rust. We also discuss how Apache Arrow enables zero-copy data sharing between all three. Thanks to Arrow, you can convert a DuckDB result to a Polars or Pandas DataFrame with negligible performance cost. 8:18 Syntax We compare the developer experience: SQL vs. the DataFrame API, using code snippets for the same transformation. DuckDB is SQL-first but also offers a Pythonic relational API. Pandas and Polars are primarily DataFrame-oriented with intuitive APIs. We also show how DuckDB can run SQL queries directly on Pandas DataFrames, combining both paradigms. 9:26 Performance The performance benchmark results are in. We ran our Hacker News ETL pipeline on a 5GB dataset using DuckDB, Polars, and Pandas. DuckDB was the fastest. We break down why Pandas failed with an out-of-memory error and how Polars succeeded using its lazy DataFrame API. This highlights the importance of choosing the right tool and features for large datasets. 10:43 Takeaways What's the final verdict in the DuckDB vs. Pandas vs. Polars showdown? For our use case, DuckDB was the most versatile and performant tool. The biggest takeaway is that these tools are collaborators, not enemies. Thanks to Apache Arrow integration, you can easily combine them in a single data processing pipeline. Adding DuckDB to your Python workflow is a simple pip install that gives you a powerful analytical engine with minimal overhead. #duckdbvspandas #duckdbvspolars #dataengineering #polarsvsduckdb #polarsvspandas #pandasvsduckdb #pandasvspolars