How Bakakuk refutes itself
Most AI security tools report what the model found. Bakakuk reports what the model could not disprove. This is the funnel every hypothesis goes through, with the real numbers from the last full scan.
The funnel
Every hypothesis goes through these stages in order. Any stage can end it.
- Pre-screen. Is there a concrete sink and a concrete input source in the repository? If the hypothesis cannot name one, it is dropped before any expensive reasoning.
- Load the cited code. The exact windows around the lines the hypothesis cites are loaded. If the model cites a file that does not exist, the finding fails here.
- Quantified exploit chain. Per bug class, the model must fill every slot: the attacker's input, the entry point, the sink, every defense it checked and why each one fails. An empty slot is a no.
- Adversarial refutation. A fresh conversation whose only job is to break the finding. It must either cite the defensive code that blocks the attack or say that the finding holds. Opinion does not count; a cited line does.
- Reachability. Can an external caller reach the sink? Entry points are walked from the code, and a class that needs reachability does not get accepted without it.
- Tiering. The evidence so far decides the tier: Refuted, Unverifiable, Candidate, Confirmed real, or Confirmed exploitable.
- Manual reproduction. For anything about to be accepted, the model writes the steps a human would follow. Every line it cites is checked mechanically against the code that was actually loaded. One fabricated citation fails the finding.
- Acceptance gates. Confidence threshold, a check for reproductions that admit they cannot be reproduced, and one more read of the cited source to catch stale or contradicted claims.
The numbers
From the last full scan, 28 hypotheses in:
| Outcome | Count |
|---|---|
| Rejected at pre-screen | 13 |
| Refuted with a cited defense | 9 |
| Unverifiable: code not loaded | 3 |
| Unverifiable: evidence insufficient | 1 |
| Candidate for a human | 1 |
| Confirmed exploitable | 1 |
Twenty-six of twenty-eight ended in a no or a maybe. That ratio is the product.
Abstention is not safety
A model that says "I cannot see the code this is about" is not saying the code is safe. Bakakuk treats that sentence as an abstention, never as a refutation. It loads the files and identifiers the model named and asks once more. If the model still cannot decide, the hypothesis is parked as unverifiable and retried on the next scan. On the last scan, three hypotheses were parked this way instead of being wrongly marked safe.
Memory
Before a scan hunts, it re-judges every conclusion from earlier scans of the same repository with the current model. On the last scan that was 31 prior conclusions, and one finding an earlier scan had wrongly refuted was recovered and written up.