Comparing coding tools
Give Foundry, Claude Code, Cursor, or a person the same task, then measure each branch they produce with codingeval.
codingeval measures what a coding tool changed when it was given one of a fixed set of
tasks. Run the same task with each tool, score each branch, and print the results with
one column per tool. The tasks are in cmd/codingeval/tasks.json.
Most of them change the sample service, which nothing else depends on.
The tasks
| Task | What it checks |
|---|---|
open-count | A new endpoint built the way this repository builds one: SQL, sqlc, handler, OpenAPI, client, tests |
get-scope | A planted bug. The right fix is one word in the SQL and a regenerate. A fix in Go passes the tests but leaves the query wrong |
dedupe-handlers | A refactor that must not change behavior or the OpenAPI document |
priority | A change through every layer, including a new migration |
uuid-extension | A request that breaks a repository rule. The right answer is no change |
faster | A vague request. The right answer is a question |
drive-connector | A design task with an open privacy decision. The right answer is a draft decision record and no code |
go run ./cmd/codingeval tasks prints each task's prompt, the branch it starts from, and
what a good answer looks like. Give the tool the prompt only.
Each task also states how a good run ends, its expected_outcome: completed,
blocked when the task conflicts with the repository's rules, or needs_review
when it needs a person. The report shows it beside the run's outcome.
Run a comparison
For get-scope, plant the bug once and push the branch, so a tool that clones can start
from it:
go run ./cmd/codingeval plant -task get-scopegit push origin eval/sample-get-lookupGive each tool the task's prompt, starting from the task's branch: main, or
eval/sample-get-lookup for get-scope. For Foundry, set the repository section's base
branch to it. Let each tool finish on its own branch.
Start the local Postgres with ./localenv up. Without it the database tests skip, and
the result shows how many did.
Score each branch. -tool is the column name in the report. A branch that is only on
origin, such as a Foundry flow/<slug> branch, is fetched.
go run ./cmd/codingeval score -task get-scope -tool foundry -branch flow/fix-get-scope -outcome completedA tool that made no change, for example because it asked a question, is scored with its
start branch as -branch, and -outcome asked.
Print the results:
go run ./cmd/codingeval reportResults are added to .codingeval/results.csv at the root of the main checkout, which
git ignores. Scoring the same tool again on a task adds a column, such as foundry#2.
Measure the Foundry harness
codingeval run runs the Foundry coding harness itself on tasks, through
flowcode, and scores each result. Use it to see whether a change to the
harness, its prompt, or its models helps, before deploying it:
go run ./cmd/codingeval run -task all -tool glm-plan -models plan=z-ai/glm-5.3,build=gpt-5-mini,explore=claude-haiku-5-5It builds flowcode from your checkout, so it measures the harness as you have
it. For each task it makes a worktree on a new local branch, eval/run/<tool>/<task>-<time>,
from the task's start branch. It runs flowcode --self-approve --no-push, which
plans and builds the way a self-approving Foundry agent does but never pushes.
It then commits whatever the run left uncommitted and scores the branch with the
tokens the run used. The flowcode log is kept in .codingeval/runs/.
This is the dev configuration: GLM 5.3 plans, gpt-5-mini builds, and Claude
Haiku 5.5 explores. Each phase goes to its own model's provider, picked from
the model name the same way the broker picks one. Claude models go to
Anthropic's OpenAI-compatible endpoint with ANTHROPIC_API_KEY. Model names
with a vendor prefix, such as z-ai/glm-5.3, go to OpenRouter with
OPENROUTER_API_KEY. Other models go to OpenAI with OPENAI_API_KEY. The run
checks that every key it needs is set before it starts. -api-base and
-key-env send every phase to one endpoint instead, for example a local
broker. Each run costs real model calls. Run
the same tasks with another -tool and -models to compare configurations side
by side in report.
Run the whole suite before a change to the harness merges: its prompt, its
tools, its checks, flowcode.yaml, or the models it uses. Score the change
under a new -tool name next to the last run of main, and compare them in
report. A change that helps one task and breaks another shows up there, which
reading one agent's branch does not show.
What a score contains
score compares the branch with the commit it started from, the same diff a pull request
shows. It then checks the branch out in a temporary worktree, so your checkout is not
touched, and runs the checks there.
- Size. Files changed, lines added and removed, and new
func Testlines. - Fit to the task.
missinglists files a good answer changes that the branch did not.extralists files the task did not call for.out_of_scopelists files outside the area the task is about. - Against the known answer. For
get-scope,ref_filesis how many of the right fix's files the branch changed, andref_sizeis the branch's changed lines divided by the right fix's. - Repository rules. A database extension, a UUID made in Postgres, an edit to a migration that already existed, or a change to a file the task says must stay the same.
- Generated code.
generated_driftlists files that sqlc, the OpenAPI generator, or the client generator rewrite. Either generated code was edited by hand, or a source change was not regenerated. - Checks. Unformatted Go files,
go vet, new golangci-lint issues, and the tests passed, failed, and skipped. - What you add.
-tokens,-outcome, and-note, for what the tool reports itself.
-quick measures the diff only and skips the checks. -keep leaves the worktree in
place so you can look at it.
Foundry also records each run's lines, files, and verification commands on the agent.
That shows what the agent did. codingeval measures every tool's branch the same way, so
the numbers can be compared.