AdvisoryDrift

The problem

A security advisory says which versions of a package are affected, and every scanner downstream repeats it. When an advisory says a flaw was fixed in 26.6.0 and the fix actually shipped in 26.4.13 and 26.6.3, thousands of teams get the wrong answer, and somebody has to work out the truth by reading git history one maintenance branch at a time. The answer is not in the advisory. It is in the repository, spread across branches that were each patched separately, and 38% of corrected advisories name no fixing commit at all, so the obvious automation cannot take its first step.

Approach

The agent gets the advisory with its answer stripped out, a local clone of the project, and nothing else: six typed, read-only git functions, no shell and no network. That boundary exists first for the evaluation, because the answer key is one unauthenticated request away and any tool that could reach it would void every number. A hand-written loop asks for one tool call or one answer per turn. What comes out is not a score but a page a maintainer can approve or reject without trusting the model: each proposed boundary, the commit that fixed it on each maintenance line, and the git show commands that check it. It never publishes anything. The benchmark is built from 430 corrections merged into the GitHub Advisory Database, frozen as 112 hashed answer keys, 44 of them scorable and 38 scored on the run reported here. Scoring is strict exact-set match: no partial credit, an abstention counts as wrong, both arms see the same cases, and the test is paired.

Decisions

1. 01

   Compare against an upper bound, not a strawman

   The advisory as published scores zero by construction on a corpus built from corrections, so the comparison is against what a competent engineer would script in an afternoon: pull the corrected value out of the advisory text, or parse the fixing commit and ask git which tags contain it. The baseline is credited whenever either of its arms is right, which no real system could do without already knowing the answer. Against that upper bound the agent matched 15 of 38 cases exactly and the baseline 5, a gap a paired McNemar test puts at p = 0.0063.

2. 02

   The tool surface did the work, not the loop

   What moved accuracy was letting the agent search the repository's history itself, so it never needs the advisory to hand it a fixing commit. Of the 11 cases the agent gets right and the baseline gets wrong, 5 have no commit in the references for a script to parse. The agent goes and finds it. That is why the six git functions are the design decision worth defending, and the loop around them is not.

3. 03

   A gate that lost on the metric, reported anyway

   The agent was finding the true boundaries and then adding later releases it should have left out: recall was fine, precision was not. A boundary gate fixes that by emitting a version only when the repository shows the flaw present just before it and absent at it. On the defined metric the gate loses, 37% against 39%, so it ships off by default. It is reported because of what it buys: over-reports fall from 5 to 0 and wrong values from 9 to 2. For a correction a person signs onto a public advisory, "I cannot evidence this boundary" is a better failure than a confident wrong range.

4. 04

   A model anyone can afford

   The headline run used claude-opus-5 through a subscription nobody else can repeat, so the agent was also run against a pinned open-weight model. openai/gpt-oss-120b matched 12 of the same 38 cases, statistically indistinguishable from the headline model (paired p = 0.5488), at a measured $0.00231 per advisory. That is parity with the expensive model, not a second proof against the baseline, and the write-up says so. All model selection and the official run together cost $0.9214 across 551 provider calls.

5. 05

   Distrust the first number

   The first run looked like a headline and was noise: the agent and the baseline had been scored on different case sets, and on the same cases the gap disappeared inside its own interval. Every number believed halfway through the build moved once the full corpus ran, and the reason was always too few cases. An evaluation at 13 cases tells you which failure modes exist and nothing about how often they happen, which is why every comparison in the repository is same-case and paired.

Results

Won Best Demonstrated Improvement at the micro1 Frontier Engineering Challenge 2026. The headline reproduces offline from a clean clone in seconds, using only the Python standard library: no API key, no network, no repository clones. It is a competition entry with a working benchmark, not a deployed product, and the repository names the limitations it froze with instead of fixing them after the fact.