How recall is measured, and why 1 of 8 is on the site
Known CVEs, real fix commits, a score per engine version. The ground truth is derived from the fix, not labelled by hand, so the number cannot be argued with. The current number is low. That is the point of publishing it.
The cases
Each case is a real CVE in a real open-source project with a single, clean fix commit. The vulnerable version is the fix commit's parent. The vulnerable lines are the lines the fix changed. Nobody labels anything by hand, so the ground truth cannot drift toward what the engine happens to find.
Two kinds of case are tracked separately. An n-day case asks the engine to find the original bug. An incomplete-fix case starts from the state after a first patch and asks the engine to find the variant that patch missed.
What counts
- Found: a finding lands on the ground-truth lines.
- Found with class: it also names the right bug class.
- Repeats: a run may attempt each case several times, because models are not deterministic. A case shows found if any repeat hit it, and also its own count, such as 2/3. The run's recall is total hits over total attempts across every case and repeat, never best-of-N. So a run can have every case found and still score under 100%: eight cases found, but two of them in only some repeats, is 21 hits in 24 attempts, or 88%.
- Skipped: the fix commit touched no scoreable source, so the case cannot be derived. Skipped cases are not counted against recall and are listed as skipped.
When a case is retired
A case whose ground truth turns out to be underivable is retired, not left in as a permanent skip. One was retired in October 2026. It had been skipped in every run, so no score changed, and every run was recomputed without it.
What is recorded
Every run is stamped with the engine version, the commit it was built from, the model, the cost and the time. Runs are kept, so the benchmark page shows a trend rather than a single number.
The current run
| Cases | 8 |
| Scored | 8 |
| Found | 1 |
| Found with class | 0 |
| Cost | $0.08 |
| Engine | 3.1.0 |
The one hit was an incomplete-fix case: the variant of a regex denial-of-service that survived the first patch. The engine landed on the right lines and labelled the class differently.
Why publish 1 of 8
A benchmark that appears only when it looks good is marketing. The number exists to be moved, and the trend is the claim. Each engine release reruns the same cases and adds a row.