Part of my work is authoring graded environments that test how well coding agents handle real software engineering. The brief is narrow and unforgiving: build a task where a capable engineer succeeds, a frontier model plausibly fails, and the grader can tell the difference without a human in the loop.
It is harder than it sounds, and the difficulty is almost never in the part you expect.
Writing a hard task is easy; writing a fair one is not
Anyone can construct a task a model fails. Make it ambiguous, make it depend on a private convention, make the tests assert something the prompt never mentioned. The model fails and you have learned nothing, because a competent human would have failed too.
The useful bar is different: the task has to be unambiguously solvable from what the task provides. A senior engineer dropped into the repository with the instructions and no other context should be able to finish it. Everything that makes it hard has to be inherent to the engineering, not to guessing what the author meant.
Most candidate tasks die here. You write something you are sure is a genuine failure, then read it back as though you had never seen the repository, and discover you were relying on knowledge that only exists in your head.
The grader is part of the task
An automated grader turns a judgement call into a boolean, and every gap between “passes the tests” and “actually solved the problem” is somewhere an agent will end up living.
Not through malice — through optimisation. If a test asserts on a log line, something will produce that log line. If a test checks a return value but not the side effect, the side effect becomes optional. This is the same phenomenon as a team gaming a metric, and the fix is the same: assert on the outcome you actually care about, and assume anything you failed to assert on is unconstrained.
Writing the grader first, before the task description, catches a surprising amount of this. If you cannot state the success criteria precisely, the task is not ready to be written.
Where the interesting failures cluster
The failures worth capturing are rarely “the model cannot write this function.” Current models write functions extremely well. They cluster instead around:
- Sustained state. Long multi-step changes where a decision made early constrains what is correct later. The model makes a reasonable choice at step two and then contradicts it at step nine.
- Load-bearing context outside the file being edited. A constraint enforced somewhere else in the repository, which is obvious once found and invisible if you do not go looking.
- Plausible-but-wrong reconstruction. Where a well-known API is subtly different in this codebase, and the training-data version wins over the version actually present.
- Knowing when to stop. Tasks with a correct minimal answer, where the failure is doing more than was asked and breaking something adjacent.
The common thread is that none are knowledge failures. They are judgement failures under sustained context — which is exactly what makes them worth measuring, and exactly what makes them expensive to author.
Why thirty years of engineering is the qualification
There is no ML in this work. What it needs is someone who has reviewed a lot of code and can tell, quickly, the difference between a solution that is correct and one that merely looks correct — because the second category is where the models are strongest and the graders are weakest.
That skill is not new. It is code review, applied to an author that never gets tired, never gets embarrassed, and will confidently produce something plausible for any prompt you give it.