Distributed analytics and feature flags with DuckDB

Principal Engineer · 2024 · 5 months · 2 min read

Built a Golang analytics backend using DuckDB and NATS JetStream for event ingestion and feature-flag evaluation.

Overview

Contract engagement to replace a brittle analytics setup. They needed event ingestion, ad-hoc queries and feature-flag decisions, without standing up a full data warehouse.

Problem

The client needed near-real-time analytics and feature-flag evaluation across multiple services. Their existing approach mixed ad-hoc SQL, Redis counters, and batch exports. Queries were slow, flags were inconsistent, and there was no durable event log.

Constraints

  • Small team, so operations overhead had to stay minimal
  • Event volume moderate but spikey; needed buffering not just fire-and-forget
  • Analytical queries ad-hoc, not a fixed dashboard set
  • Deploy on AWS with familiar tooling

Approach

Designed an event-driven pipeline: services publish to NATS JetStream, a Golang consumer writes to DuckDB for analytics, and a separate evaluation path serves feature flags from the same event stream.

Key Decisions

NATS JetStream for event ingestion, not Kafka

Reasoning:

JetStream gave durable streams, replay and consumer groups without a second message platform to operate. The team was small and already running NATS for service messaging, so this added no new infrastructure skill. Kafka's operational overhead was disproportionate to the event volume.

Alternatives considered:
  • Apache Kafka: industry standard, but the ops surface was wrong for the team size
  • Direct HTTP ingestion to DuckDB: simplest, but no buffering during spikes
  • Redis streams: already in the stack, but weak on long retention and replay

DuckDB for analytical storage

Reasoning:

Embedded OLAP with excellent SQL and Parquet support, and no separate warehouse cluster to run.

Alternatives considered:
  • ClickHouse
  • PostgreSQL with Timescale
  • S3 + Athena

Tech Stack

  • Golang
  • DuckDB
  • NATS JetStream
  • AWS

Result & Impact

  • None
    warehouse clusters to operate
  • 5 months
    engagement

Unified event ingestion and feature-flag evaluation behind one pipeline. Analytics queries that previously required batch exports ran directly against DuckDB. The client could add new flags and event types without redeploying consumers.

Learnings

  • JetStream stream configuration needs upfront thought about retention and consumer lag
  • DuckDB shines for embedded analytics but batch insert patterns matter at scale

← All projects