Finding the Bug Is the Easy Part. Knowing It's New Is the Hard Part.

There is a failure mode more dangerous than a false positive, and it hides in every automated vulnerability-research system: a true positive that the tool wrongly believes is new.

We hit it recently, and it’s worth telling straight.

What happened

Our pipeline took a stripped, closed-source Windows kernel driver: no source, no symbols, no debug info. It located a dangerous IOCTL handler, recovered the path to the sink, and drove it on a live kernel with a debugger attached. The result was clean and real. The driver mapped arbitrary physical memory with attacker-controlled address and length, we captured the operands at the sink, and we byte-bound the returned data to the physical page. A confirmed arbitrary-physical-read/write primitive, machine-verified, with no human in the confirmation loop.

That part we are proud of. Then the pipeline labeled it novel, and there it was wrong. The bug is CVE-2024-33228, published in 2024. The vendor had already shipped a fix.

Why it happened

The pipeline had a novelty check. It just wasn’t a novelty check.

It compared the driver against a public catalog of known-vulnerable drivers, by file hash. The driver we had wasn’t in the catalog, so the pipeline concluded “uncatalogued,” and we read that as “new.”

But a catalog of known-bad hashes is a blocklist, and a blocklist answers a different question than the one we were asking. A fresh build of already-vulnerable code has a fresh hash, so it isn’t on the blocklist, while carrying a bug that has been public for a year. “Not on the blocklist” is not the same as “not known.” We had conflated the two, and the model happily wrote “novel” on top of the confusion.

This is the shape of a mistake automated bug-finders make everywhere, not a Windows-driver quirk. The tool does the hard, impressive work (reach the sink, prove the crash, confirm the primitive) and then makes a claim about the world’s prior knowledge that it has no grounding for. The discovery is verified against a machine. The novelty is verified against a vibe.

The part nobody wants to say out loud

In automated security research, finding the bug is the easy 80%. The dangerous 20% is the two questions that come after: is it real, and is it new. The first we already answer mechanically: a proof-of-vulnerability that reproduces, or it doesn’t count. The second we were answering with a hash lookup and a hope.

“Is it new” is exactly where the entire industry’s credibility problem lives. It is the question that turns a re-discovery into a press release, that turns a known CVE into a “0-day,” that gets an AI security tool laughed off a bug bounty program for slop. The incentive is to claim novelty, and the check on that incentive has to be mechanical, because the model, like the researcher, wants the win.

What we built

So we built the second oracle: a novelty gate that every finding passes through before the word “novel” is allowed to exist.

It’s deliberately more than an embedding search, because the three questions it has to answer need three different tools:

• Is this a known CVE? Structured attribution first: match the affected product, file, function, and weakness class against the public record (the national vulnerability database, open-source advisories, vendor bulletins). Then a semantic pass for paraphrased descriptions. Then a live search fallback for the closed-source and firmware cases where the structured record is thin, which is precisely the gap the driver slipped through.
• Is it a duplicate of something we already found? Nearest-neighbor over a technical fingerprint (the sink, the bug class, the primitive shape) against known CVEs and our own prior findings.
• What’s already known about this target? For a given product and version, a deterministic lookup of every CVE that applies. The useful inverse, for scoping a target before you ever touch it.

Two rules are load-bearing, and both are about not trusting the machine’s optimism:

It never auto-dismisses. A gate that silently drops everything it thinks is “known” will, sooner or later, drop something that is actually new. So the gate flags and ranks; it never deletes. A human or a stricter check makes the call.

“No record found” is not a novelty claim. It is an absence of evidence, and we label it exactly that. Our local knowledge is only as fresh as its last sync. A clean result means “we found nothing public,” never “this is new.” The strongest thing the gate will ever say on its own is candidate, never novel.

The point

It would have been easy to publish the driver result and let “novel” stand. It reproduced. The evidence was real. Nobody would have checked for a while.

That is the trap. A smarter model doesn’t close it. The interesting engineering in automated security is the oracles you build to keep the smart model honest. One to prove the bug is real. One to keep you from claiming it’s new when it isn’t. The discovery is the part the machine is good at. The discipline is the part you have to build.

---

Analysis and tooling by 0sec’s automated security-research system (https://0sec.ai). CVE-2024-33228 is a published, vendor-remediated vulnerability, referenced here only as the example that motivated the work.