Distributed analytics and feature flags with DuckDB
Built a Golang analytics backend using DuckDB and NATS JetStream for event ingestion and feature-flag evaluation.
Overview
Contract engagement to replace a brittle analytics setup with a distributed backend that could ingest events, run analytical queries, and serve feature-flag decisions — without standing up a full data warehouse.
Problem
The client needed near-real-time analytics and feature-flag evaluation across multiple services. Their existing approach mixed ad-hoc SQL, Redis counters, and batch exports. Queries were slow, flags were inconsistent, and there was no durable event log.
Constraints
- Small team — operations overhead had to stay minimal
- Event volume moderate but spikey; needed buffering not just fire-and-forget
- Analytical queries ad-hoc, not a fixed dashboard set
- Deploy on AWS with familiar tooling
Approach
Designed an event-driven pipeline: services publish to NATS JetStream, a Golang consumer writes to DuckDB for analytics, and a separate evaluation path serves feature flags from the same event stream.
Key Decisions
DuckDB for analytical storage
Embedded OLAP with excellent SQL and Parquet support — no separate warehouse cluster to run.
- ClickHouse
- PostgreSQL with Timescale
- S3 + Athena
Tech Stack
- Golang
- DuckDB
- NATS JetStream
- AWS
Result & Impact
Unified event ingestion and feature-flag evaluation behind one pipeline. Analytics queries that previously required batch exports ran directly against DuckDB. The client could add new flags and event types without redeploying consumers.
Learnings
- JetStream stream configuration needs upfront thought about retention and consumer lag
- DuckDB shines for embedded analytics but batch insert patterns matter at scale