Not every team needs a data warehouse. Many need something in between: durable event ingestion, ad-hoc SQL, and enough throughput for product analytics and feature-flag evaluation.
That was the brief on a Golang backend engagement. The client had outgrown Redis counters and batch CSV exports. They did not have the ops capacity for Kafka plus ClickHouse.
The shape that worked
Events published to NATS JetStream. A pull consumer batch-inserted into DuckDB. Feature flags evaluated from a parallel consumer on the same stream.
DuckDB handled the analytical queries — funnel analysis, flag usage, cohort breakdowns — without a separate cluster. Parquet export was a nice bonus for longer retention.
Where DuckDB fits
DuckDB is excellent when:
- Query patterns are analytical, not transactional
- Data volume is GB-scale, not PB-scale
- You want SQL without managing Postgres extensions or a cloud warehouse bill
It is less excellent when you need concurrent write-heavy OLTP or multi-tenant isolation at the database layer. Know which problem you have.
Batch inserts matter
The first version inserted events one at a time. It worked in development. Under load, the consumer fell behind.
Switching to batched inserts with a short flush interval fixed lag without changing the architecture. Embedded databases reward understanding their write patterns.
I wrote this up in more detail as a case study on the analytics project, including why JetStream won over Kafka.