What the new engine can and cannot move on the benchmark
The published scores measure a model reading a window of vulnerable code. They cannot see the hunt, so they cannot see this engine. Here is which changes could raise a full-hunt score, which cannot, what we corrected in the published runs, and the one case we retired.
It would be easy to say the new allocation raises the benchmark score. We cannot say that, and the reason is worth setting out.
What counts as a hit
A case is a real CVE with a real fix commit. The vulnerable lines are the lines the fix changed. A hit is any finding, accepted or candidate, that lands within eight lines of them. Candidates count, which matters below.
What the published runs measure
Every published run uses focused mode: one model call over the window of code around the vulnerable lines, with no clone, no hunt and no verification. That makes it cheap and fair for comparing models. It also means the score is the model's ability to read a window of code. Nothing about how the engine allocates work can change it.
Until this release, the other mode, which runs the full hunt, could not score anything. It fetched findings from a route that did not exist, so every case came back empty. That is fixed, and one run can now compare two allocations over the same cases. No such run has been published yet.
Which changes could raise a full-hunt score
| Change | Could it move a hit? |
|---|---|
| More scout breadth | Yes. More independent hypotheses means more chances to land on the vulnerable lines |
| Frontier adjudicator overturning a refutation | Yes. A real bug the cheap verifier threw out becomes a candidate, and candidates count |
| Frontier second opinion on a dismissed lead | Yes. It can recover a lead the analyst gave up on |
| Frontier confidence scoring | No. A candidate counts whatever its score. This helps a human decide what to read first, not the hit count |
So three of the four changes act on recall. The fourth acts on triage. We would rather say so than list all four as score gains.
What we corrected in the published runs
While recomputing the scores for this release we found an error that predates it. Two runs were comparisons of several models, but the figure shown was pooled across all of them and labelled with only the first model.
| Run | Shown before | Correct |
|---|---|---|
| Engine 3.4.0, Haiku 5.5 | 28% | 80%, over the 5 of 8 cases scored |
| Engine 3.7.0, Haiku 5.5 | 69% | 63% |
Each run's figures are now rebuilt from the rows of the model it names. The other nine runs were already correct.
The case we retired
One case has been removed from the set. Its fix commit changes no source file we can score, so its ground truth cannot be derived, and it was skipped in every published run. It never counted toward any score, and removing it changed none. The set is now eight cases, and every run was recomputed without it.
Where this leaves the numbers
| Run | Cases hit | Per-run recall |
|---|---|---|
| Sonnet 5.5, engine 3.12.0 | 8/8 | 88% |
| Haiku 5.5, engine 3.12.0 | 7/8 | 75% |
Sonnet's 8/8 means every case was hit at least once. Its 88% counts every repeat: 21 hits in 24 attempts, because two cases were hit in only some repeats. Each case's own count is on the benchmark page.
These measure the models on engine 3.12.0, before the allocation work. The full-hunt comparison of the current allocation against the old one is the number that would show whether the engine itself got better. It has not been run, and whatever it shows, it will be published.