Does It Discover, or Does It Recall?

There is one question that tells you whether an AI system actually finds bugs, and almost every benchmark dodges it: does it discover, or does it recall?

A language model has read a large fraction of the open-source vulnerabilities that exist. Point it at a famous CVE and ask it to “find the bug,” and a confirmation tells you very little. It may have reconstructed the flaw from memory, not from the binary in front of it. Most impressive-looking results live in exactly this gap. The demo works because the answer was in the training data.

So we ran the version that closes the gap, and reported what happened even though it isn’t a victory lap.

The airtight test

The clean way to force discovery is to take bugs that were fixed after the model’s training cutoff. The model cannot have memorized them. They didn’t exist when it was trained. Strip the binaries of source, symbols, and debug info. Give no hints, no ground-truth pointer, and, critically, exclude the known proof-of-concept input. If the engine confirms a crash under those conditions, it found the bug. There is nowhere for recall to hide.

We built four such tasks from real open-source projects patched in 2026: an audio codec, an XML parser, a RAW image loader, and an EXR image decoder. Each is a matched pair (vulnerable build, fixed build) with a differential oracle as the sole arbiter. The input must crash the vulnerable build and run clean on the fixed one, or it doesn’t count.

The result: 0 of 4, and the why is the whole point

The engine did not confirm any of the four. It also, demonstrably, did not recall them. The reason it failed is precise and, we think, genuinely useful.

Automated bug-finding has two halves: locate (which function holds the flaw) and trigger (what input actually fires it). On these four, locate worked. The engine landed the vulnerable function crutch-free, from a stripped binary, no symbols. It walled on trigger, every time, and always for the same structural reason: the bug lives behind a razor-narrow input envelope that neither fuzzing nor a language model can reconstruct blind.

• One bug is in a legacy decoder path, a disconnected format island. Every normal input for that file type uses the modern container; the flaw is in an old one with a different magic at a different offset. A coverage-guided fuzzer (millions of executions) drifts off it and never lands. There is no gradient from a modern file to the legacy island.
• One is grammar-shaped: a heap overflow reachable only through a specific deep nesting of a document-type declaration. A hundred structurally-varied attempts, none deep enough in the exact way.
• One is model-gated: the vulnerable path only runs when a header field equals a value computed from other fields in a way that no generic file for that format satisfies. Only the hand-crafted proof-of-concept threads the needle.
• One is an integer overflow that needs precise dimensions: large enough to wrap a size calculation, small enough that the buffer allocation doesn’t simply exhaust memory first. Get it wrong in either direction and both builds just die the same way. No signal.

Here is the detail that reframes the whole exercise: for several of these, even building the benchmark required the exact human-found proof-of-concept. The public “reproducers” we started from were near-misses. They reached the vulnerable function but didn’t quite overflow, or they declared dimensions so extreme that both builds ran out of memory, which is not a differential. Constructing a valid crash-here, clean-there input pair was itself a crafting problem. And the industrial fuzzing infrastructure that continuously tests these projects hadn’t reached these bugs by fuzzing either. They were found another way.

What that measures

That last point is the finding. These bugs survived years of continuous fuzzing because their trigger envelopes are extraordinarily narrow. The space of inputs that fire them is a needle, and neither random-walk mutation nor a language model writing bytes from a decompiled sink can thread it without already knowing the answer.

So the frontier of automated vulnerability discovery is not locating bugs. That problem is yielding. You can find the vulnerable function in a stripped, closed binary today. The frontier is trigger-envelope reconstruction: deriving the narrow input constraints that reach and fire a located bug, when the path there is a disconnected island, a deep grammar, or a computed gate. That is a reachability-and-constraints problem, and it’s the honest next thing to build. A way to reason backward from the sink to the input that must reach it, not a bigger model.

Why we’re telling you about a zero

Because a 0/4 you can explain is worth more than a benchmark you ace by recalling. It tells you exactly where the capability is real (locate, crutch-free, on modern stripped binaries, genuinely hard and genuinely working) and exactly where it isn’t yet (trigger, on deep-envelope bugs, a specific and nameable wall). An automated security tool that can’t tell you which of its wins were discoveries and which were memories is not one you should trust with the word “novel.”

We would rather map the frontier honestly than plant a flag on the near side of it. The trigger-envelope wall is the next lever. That is what we build next.

---

Analysis by 0sec’s automated security-research system (https://0sec.ai). The four vulnerabilities are published, vendor-fixed 2026 CVEs, used only to measure the engine on inputs it could not have seen in training.