Assay

22 October 2026 · 5 min read

A Go harness that puts one task to coding agents from several providers, each sealed in its own container, and grades what they leave behind.

  • Golang
  • Docker

Assay gives a task to coding agents - Claude Code, Grok, Google's Antigravity and Codex, each driven through its own CLI - in throwaway containers, then grades what they leave behind with a checker they never see.

Motivation

I built it while writing adversarial engineering tasks for frontier models. A good task takes days to build properly, and most ideas die the moment a model solves them. I wanted to find that out in minutes, on the cheapest model that could kill the idea, before spending the days.

Approach

  • One task folder: a prompt, the input files, and an optional hidden checker
  • A fresh container per run, with each agent's CLI pinned at an exact version
  • The network fenced to the agent's own API, so the agent works offline the way a task environment would
  • Grading in a second container that sees only what the agent left behind
  • Every run kept on disk: transcript, raw events, files, verdict, cost
checker container, no networkagent container, fenced/app as lefteventsprompt + filesagent CLIhidden testsverdict + costlive transcript

Four CLIs, one interface

Each agent is one entry in a table: how to install it, how to launch it headless, which hosts it may reach, which effort levels it accepts, where its credentials live, and a decoder for its event stream. Adding an agent is adding an entry.

The CLIs all stream newline-delimited JSON events, and no two agree on the shape. Claude and Grok share an envelope; Antigravity reports each step as it starts and finishes and sends text as deltas; Codex reports each tool call as an item that starts and completes. Each decoder turns its own format into the same few kinds of line, so parallel runs from different providers read the same in one terminal. A tool call and its result in Claude's stream, trimmed:

{"type":"assistant","message":{"content":[{"type":"tool_use","name":"Bash","input":{"command":"wc -l < /app/data.txt …"}}]}}
{"type":"user","message":{"content":[{"type":"tool_result","content":"7"}]}}

and the whole recorded run as Assay renders it:

I'll count the lines and write the result.
→ Bash wc -l < /app/data.txt | tr -d ' ' > /app/answer.txt && cat /app/answer.txt
← 7
Done. /app/data.txt has **7** lines, and that number has been written to /app/answer.txt.
· success

Each agent has a recorded stream and its expected rendering in the tests. After a CLI upgrade, one re-recorded run shows exactly what the new format broke, as a diff.

Trust nothing the CLI says

Vendor CLIs are built for people at a keyboard, not for a harness, and their defaults fail quietly.

  • Effort levels are validated before a container starts. Claude treats an unknown effort as a warning and falls back to its default, so a typo would give a run I believed was pinned and wasn't
  • Versions are pinned in the image. Updating my own claude can't change a trial under me, and nothing but credentials comes from my machine
  • Credentials are copied in, never mounted, so nothing the agent writes can touch my home directory

The fence

By default a run has no internet, because task environments usually don't. The CLI still needs its own API, so each agent's API hosts are resolved on my machine and pinned into the container, and iptables rejects everything else. Rejected rather than dropped, so a blocked pip install fails at once instead of eating the timeout.

Some agents also have web tools that run on the vendor's servers, out of the firewall's sight. With internet off, Assay switches those off too: with a flag where the CLI has one, and for Antigravity, which doesn't, with a hook that refuses the calls.

It's a fence, not a sandbox: an allowed address behind a CDN reaches everything on that CDN, and the credential inside is live.

A ladder that stops on the first pass

A stage lists models cheapest first. If any run on a rung passes the checker, the stage stops: a cheap model solving the task is enough to kill the idea, so the dearer rungs never spend anything.

[stage.prune]
stop_on_pass = true
rungs = [
  { agent = "agy", model = "gemini-3.1-pro-high", effort = "high" },
  { agent = "grok", model = "grok-4.7", effort = "high" },
]
failsfailspassespassespassesall failcheapest rungnext rungdearest rungidea deadworth building

A stage like this refuses to start without a checker, because without one nothing can pass.

What a run leaves

  • The transcript as it was rendered, and the raw event stream for anything the rendering dropped
  • The agent's working directory as it left it - copied out even on a timeout, because what a stuck agent left behind is the interesting part
  • A list of what it added, changed and deleted
  • The checker's output, and a metadata file with the model, CLI version, image, duration, tokens, cost and verdict

A run cut short by a session limit, an API error or Ctrl-C is snapshotted with its credentials removed, and can be resumed later from where the agent stopped. A timeout isn't resumable - it's a result.

Second opinions

The same machinery doubles as a consultant. An ask stage mounts the repository read-only, opens the internet, picks the strongest model, and gets an answer back as a file.

← All articles