DuckDB
Resources
- https://medium.com/striim/beyond-materialized-views-using-duckdb-for-in-process-columnar-caching-98b8387b8568 (Stream Processing/Ingestion Patterns )
- Smallpond – A lightweight data processing framework built on DuckDB and 3FS | Hacker News
- Practical Applications for DuckDB (with Simon Aubury & Ned Letcher) - YouTube
- Summing columns in remote Parquet files using DuckDB | Hacker News
- Querying Parquet with Precision using DuckDB - DuckDB
- Build sub-second data applications with MotherDuck’s Wasm SDK
- Implementing Hardware-Friendly Databases (with DuckDB co-creator, Hannes Mühleisen) - YouTube
- DuckDB Doesn’t Need Data to Be a Database | Hacker News
Streaming data and DuckDB
See Stream Processing/Ingestion Patterns
- In-Memory Analytics for Kafka using DuckDB – Robert Yokota
- https://github.com/mistercrunch/duckstreams
- Redpanda | Cloud Topics: Efficiently stream data through object storage
Plugins / Extensions
DBT
dbt-duckdb- DuckDB & dbt | End-To-End Data Engineering Project
- This additionally has extensions such as milicevica23/dbt-duckdb-delta-plugin-demo
- Eg. delta plugin uses the delta-rs package, which enables reading directly from different cloud object stores.
- You can write your own plugins
Writing to object store
- See httpsfs extension
Others
Others
- Merge
import duckdb duckdb.execute(""" COPY (SELECT * FROM '*.parquet') TO 'merge.parquet' (FORMAT 'parquet'); """)