Benchmark(8)

Real CVEs with real fix commits. The vulnerable lines are the lines the fix changed, so the ground truth is derived, not labelled by hand. Pick a run to see its model, engine, caps, cost and every case. Rules are in How recall is measured.

Runs (13)

Selected run

75%

per-run recall: 6 of 8 attempts hit · 6/8 cases hit at least once · 4 with the right class

Model
claude-haiku-5.5
Engine
3.24.0 @ d0da6db+3.24.0
Mode
focused · confidence 60
Per-case cap
15m · $4.00
Spend
$0.0084 · 44,827 tokens · 8 calls
Wall time
2 min
Date
2026-10-21
  • n-day5/6
  • incomplete-fix1/2

Numbers as of 2026-10-26. Runs differ in model and caps on purpose; read each run's row before comparing two scores.

[ Fig. 2 ]10x
CVEClassCaseResult
  • CVE-2021-23337injectionn-daylodash/lodashCode injection via template sourceURLfound0.1 min · $0.0014
  • CVE-2020-8203prototype-pollutionincomplete-fixlodash/lodashVariant surviving the earlier prototype-pollution fixmissed0.1 min · $0.0004
  • CVE-2019-10744prototype-pollutionn-daylodash/lodashPrototype pollution in defaultsDeepmissed0.1 min · $0.0014
  • CVE-2017-16137redosn-daydebug-js/debugReDoS in the %o formatter regexfound0.2 min · $0.0010 · class ✓
  • CVE-2020-1747deserializationn-dayyaml/pyyamlCode execution via python/object/new in FullLoaderfound0.1 min · $0.0016
  • CVE-2021-3803redosn-dayfb55/nth-checkReDoS in the nth-child expression parserfound0.2 min · $0.0013 · class ✓
  • CVE-2020-28469redosn-daygulpjs/glob-parentReDoS in the enclosure regexfound0.1 min · $0.0006 · class ✓
  • CVE-2021-35065redosincomplete-fixgulpjs/glob-parentReDoS variant that survived the first patchfound0.1 min · $0.0008 · class ✓