Canary does not. It scores coding agents on bugs published after the model's training cutoff, and verifies every answer by running the code — never by asking another model whether it looks right. The same agent that scores 90% on the usual kind of benchmark scores 0% here.
gemini-3.7-flash · 16 cases · training cutoff 2026-03-01
No section matches that search.
An engineering lead is about to let an AI agent open pull requests against a private codebase. All they have is a benchmark score, and that score carries two risks nobody prints beside it.
These models trained on essentially all of public GitHub — the bugs and the commits that fixed them. Evaluate an agent on well-known open-source defects and some part of what you measure is recall rather than reasoning. Their codebase is private; the model has never read it. The published number was measured under the one condition that does not hold for them.
“Found and fixed” often means a plausible diff nobody executed. In this run, 8 of 12 patches applied without a merge conflict and none of them made the failing test pass. Score on application and you report 67%. Only running the code tells those apart.
Same model, same twelve real defects, same run — scored three ways:
| How it is scored | Dataset | Success criterion | Reported rate |
|---|---|---|---|
| Textbook bugs only | 10 fixable synthetic bugs | One direct prompt | 90% |
| Patch-application check | 12 real published defects | Patch applies via git apply |
67% |
| Canary, sandboxed execution | 12 real published defects | Failing test runs and passes | 0% |
The bottom two rows are one run graded two ways. The top row also changes which bugs are asked about, which is the other half of how a benchmark reaches ninety.
One case is a canary: the security advisory naming the file and the defect is deliberately left in the source tree. The agent located it at 0.90 confidence, as it should — and still produced a patch that did not fix it. Across all twelve real defects it located three and repaired none. A benchmark scoring localisation alone would have called this run a partial success.
Of the findings stated at 90%+ confidence, 2 of 12 were correct. It raised 4 false alarms on clean control code, where the right answer was silence. An agent this sure and this wrong is more dangerous than one that admits doubt, because it gets believed.
This project was built to detect training-data contamination. On this sample it did not find any.
| Defects published | Located | Fixed | n |
|---|---|---|---|
| before the cutoff | 20% | 0% | 5 |
| after the cutoff | 17% | 0% | 6 |
| gap | 3 points | 0 | — |
Three points, on five cases against six — indistinguishable from zero, since one case landing either way moves it by twenty. Twelve defects cannot resolve an effect of the size worth caring about; that claim needs a hundred. What the instrument demonstrably does catch is the canary, which fired on every run, on both models.
The headline stands on its own evidence anyway: 90% of textbook bugs, 0% of real ones. You do not need a story about training data to find that alarming.
make claims and re-checked
by make verify, which fails on drift.
Canary has two baselines, and it matters which one a gain is measured against.
Same model, same cases, same grader: the reasonable basic approach, and what a developer actually does today. The fenced agentic runner, with its toolbelt and its multi-step loop, was built to beat it.
It did not. On the synthetic set it lost — 80% against 90%. On the real set both scored zero. Across everything measured here, the scaffolding never won.
Reported as a negative result rather than quietly dropped. On the real set the agentic
runner covered only 5 of 16 cases before an interrupted run, and a partial score is not
comparable with a complete one, so it is excluded from the headline table and marked
valid: false in the results JSON rather than averaged in.
This is where the gain is. Given the same twelve defects, the same model and the same run, the number you report depends entirely on how you score it: 90%, 67% or 0%.
The improvement is a reduction in measurement error, not an increase in agent capability. The number a buyer would act on stops being overstated by 67 to 90 points.
Only the last row survives the question an engineering lead is actually asking: will this fix my bug?
No model judges another model. The design replaces opinion with execution, audited redaction, and a control case that fails loudly when the harness itself is wrong.
The agent sees the repository as it stood at the buggy commit. Anything naming the fix — the advisory, the CVE id, the commit message — is redacted line by line, and every redaction is logged to an auditable manifest.
The agent gets list_files, read_file, search and
run_tests. Each task declares whether its test predates the fix or shipped
with it. The agent may run the first and is refused the second.
The patch is applied to a throwaway copy and the test re-run under
sandbox-exec, with the network denied, writes confined to the copy and
secrets stripped from the environment.
make ingest builds a task from two commits and writes nothing until it has
proved it: the test must fail on the buggy tree and pass once the fix is applied.
Every step is logged as a structured trajectory. Past runs can be re-scored under a revised rubric, locally and deterministically, without spending anything on APIs.
One case has the advisory deliberately left in the tree. If the harness reports that as a brilliant catch instead of flagging the leak, the harness cannot be trusted. It fires on every run.
The grader may know the future. The agent may not. Grading a fix against the regression test is sound science. Letting the agent read that test first is contamination wearing a lab coat.
Model output, patches and pytest transcripts below are quoted from the committed run
artefacts in trajectories/real_37/. Long tracebacks are elided and marked as
such; nothing is reconstructed. Each case links to the file it came from.
—
—
——Every command below runs on a clean clone. The default evaluation uses a deterministic offline stand-in model, so the full pipeline exercises end to end with no API key and no spend.
git clone https://github.com/seyimusibau-cyber/Canary.git
cd Canary
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txtmake test # 47 integrity tests: fence, sandbox, task sets (~16s)
make eval # full pipeline, deterministic offline stand-in (~25s)
make report # renders results/ to report/index.html[!] 1 contamination canary FIRED. If that line is missing, the run is
invalid — that is what it is for.
The same pipeline runs against a live model through Google AI Studio, behind a budget gate that stops the run rather than trusting you to notice.
# 1. Provide a Google AI Studio API key
cp env.template .env
set -a; . ./.env; set +a
# 2. Run with an enforced dollar ceiling
CANARY_AGENT_MODEL=gemini-3.7-flash \
python3 -m canary.cli eval --tasks tasks/real --cutoff 2026-03-01 \
--out results/real_gemini37.json --traj trajectories/real_37 \
--budget-usd 1.75 --timeout 90
# 3. Render the HTML report
python3 tools/render_report.py results/real_gemini37.json report/index.htmlCANARY_THINKING_BUDGET=512 cut one
call from $0.084 to $0.013, a 6.6x reduction.
The number that matters to you is the one measured on code the model has not read, and that
code is yours. make ingest builds a task from a bug already in your git
history: the buggy tree, the regression test lifted from the fixing commit, and ground truth
taken from the lines that fix actually touched.
# A real defect: buggy commit, fixing commit, and the test that separates them
make ingest REPO=~/src/your-project \
BUGGY=a1b2c3d \
FIX=e4f5g6h \
ID=auth_token_bypass \
TEST_CMD="python3 -m pytest -q tests/test_auth.py"
# A clean control, to measure false alarms on correct code
make ingest REPO=~/src/your-project \
BUGGY=main \
ID=clean_controller \
TEST_CMD="python3 -m pytest -q tests/test_controller.py" \
CLEAN=1Model responses are persisted in trajectory files, so re-scoring under a revised rubric or a corrected test happens locally, at no cost, and produces the same answer every time.
# One case, end to end, with live sandboxed test execution
make demo CASE=scitokens_1_9_6
# Re-grade complete runs from recorded trajectories ($0.00)
make replay| Target | What it does | API cost |
|---|---|---|
make test |
Runs the 47 integrity guards — fence, sandbox, budget, task sets. | $0.00 |
make eval |
Evaluates all three runners against the synthetic set with the offline stand-in. | $0.00 |
make live-eval |
Runs the evaluation against a live model via Google AI Studio. | live API |
make ingest |
Builds a verified task from two commits in your own repository. | $0.00 |
make report |
Renders a results JSON into a standalone HTML report. | $0.00 |
make claims |
Rewrites every claim token in the README from committed run data. | $0.00 |
make verify |
Fails if any README claim has drifted from the recorded results. | $0.00 |
make demo CASE=<id> |
Runs a single task end to end with live sandbox execution. | $0.00 |
make replay |
Re-grades recorded trajectories under the current rules. | $0.00 |
Building a measurement instrument, the dangerous defects are never the ones that crash. They are the ones that hand you a believable figure.
The fence's leak-marker list matched the bare substring "fix" — which also
matches prefix and suffix. On real packages that excluded
almost every source file. jinja2/lexer.py was fenced out entirely: the
agent was graded on 868 lines it had never been shown, scored a miss, and the miss
looked like an honest result.
The automated task builder produced a case whose ground truth was
__version__ = (1, 2, 4) — asking the agent to find the bug in a version
declaration. The fail-then-pass validation waved it through, because the old release
genuinely does fail the new test, for reasons unrelated to the graded line.
Spend was computed from answer tokens alone while the model billed its thinking as
output. Reported $4.33 against an actual $11. Thinking is counted now and capped, but
past runs cannot be corrected — thoughts_token_count was never recorded —
so replayed results carry cost_is_understated and the report refuses to
print those figures.
None of these were caught by ordinary correctness tests. All were caught by tests written to
ask a different question: could this pass for the wrong reason? Those are standing
guards in tests/test_realtasks.py now. The full account, including two ideas
that turned out to be wrong, is in
IMPROVEMENT_CHANGELOG.md.
sandbox-exec.
Linux and Windows have no equivalent path in this harness yet.
scitokens case
scored as located on a finding in an unrelated subsystem, and the fence had not shown the
agent the file the defect lives in. Both are open issues, and both are visible in the
case inspector rather than smoothed over.
cost_is_understated.
Canary is MIT licensed. The real task set vendors roughly 20MB of third-party open-source code — the packages exactly as published to PyPI before each fix — so the evaluation reproduces from a clean checkout with no network. Every package's own licence is vendored alongside it, and tasks/real/PROVENANCE.md lists each case: package, version, licence, advisory, and the path to that licence file. That table is generated by the builder, not maintained by hand.
Advisory data comes from the public OSV database; sources come from PyPI's public JSON API. No credentials are involved and none are in the repository.
Disclosed per the rules. Canary was scaffolded before kickoff — the fence, the grader, the
synthetic task generator, the CLI and a stubbed agent runner, about 900 lines, committed in
63a6132 and 5d68e2b, both dated 27 August 2026. Everything from
f2558a4 onward was built during the event: the sandbox and its approval gate,
trajectory recording, the toolbelt and the fenced agent loop, the budget guard, the real
task pipeline and its scanner, the HTML report, and every test beyond the original four.
Two methods are reused from earlier personal projects and disclosed as such: point-in-time evaluation with a leak guard, and recompute-every-claim-from-data verification. Both were reimplemented here from scratch; no code was carried over.
Model under test: Gemini, via Google AI Studio. Harness author's assistant: Claude Code was used throughout to write this project, disclosed here rather than hidden.