Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
scthornton 
posted an update 9 days ago
Post
2646
SecureCode update: we went back and fact-checked our own security dataset and corrected what didn't hold up.

The original claim was "complete incident grounding, every example ties to a documented CVE." An adversarial re-audit found that it was overstated: many CVEs were misattributed, and many "incidents" were representative scenarios carrying invented statistics. So we fixed it.

- Grounding: re-verified every reference. Removed 802 misattributed CVEs on the web side, corrected or honestly relabeled the incident narratives, and confirmed the AI/ML conversation CVEs are real (EchoLeak CVE-2025-32711, EmailGPT CVE-2024-5184, and others).
- Fix-correctness: reviewed whether each "secure" example actually eliminates the vulnerability. Removed 28 that did not (a "secure" secret scanner whose entropy check always returned zero, an Angular example still using bypassSecurityTrustHtml, and more).
- Leakage: re-split so near-duplicates stay on one side. Test contamination went from 11.6% to zero.
- Viewer, schema, and metadata: rebuilt as parquet under a shared schema. All three viewers are live.
- Models: retrained the whole family on the corrected data so the fix reaches the weights, not just the cards. Now ten open models (3B to 26B), including two new Gemma 4 variants, refreshed locally on a DGX Spark GB10. The paper (arXiv:2512.18542) was revised to match.

Counts moved from 2,185 to 2,372 unified (web 1,625 + AI/ML 747). A slightly smaller, fully-checked dataset beats a larger one you have to take on faith. Full writeup and links in the article.

Datasets: scthornton/securecode, scthornton/securecode-web, scthornton/securecode-aiml
scthornton

https://huggingface.co/blog/scthornton/securecode-audit-fixes
dipankarsarkar

The 802 is the safer number. The 28 is the one I would build around.

A misattributed CVE has an external referee. The database says yes or no, you re-check all 1,379, you drive it to zero, and it stays there. That work is finished.

The 28 are a different class. A SecretScanner that computes Shannon entropy and always returns 0.0 is not a wrong citation. It is code that looks like a fix and passes everything that never feeds it a real secret. Nothing outside the repo says it is wrong. A reader noticed.

You reviewed all 1,412, and setting the review up so a claim only survives if it can't be refuted is the right instinct. But zero disputed is a precision number. It says what review flagged held up. It says nothing about what review walked past. Full coverage is not full recall, and the one failure mode that matters has no false-negative estimate on it.

That part is mechanizable. Every secure example claims a specific vulnerability is gone. Make it prove it: feed the paired vulnerable input through the secure version and require the check to fire. The entropy-always-zero one dies on the first positive control, with no reviewer and no referee argument to have.

We ran the same move on LLM-generated GPU kernels. Mutation-scored the oracle instead of the artifact, and torch.allclose waved through buggy variants at a rate that made the oracle the actual problem.

So run positive controls across the examples you kept. How many fire? That number is the recall you don't have yet, and it's the only thing that tells you whether 28 was the count or just the start.

·
scthornton

Yep, you've named the real weakness, and you are right.

The 802 and the 28 are different epistemics. The CVEs had an external referee, so rechecking all 1,379 and driving it to zero is a final result that stays final. The 28 had no referee, and "zero disputed" is a precision number: it says what the review flagged held up, not what the review walked past. Human review cannot estimate its own false-negative rate, and "a reader noticed" is the honest account of how the entropy-always-zero scanner surfaced. A reader is not a guarantee.

Positive controls are the right fix, and I'm in the process of running them now. For each secure example that claims a specific vulnerability is eliminated, push the paired vulnerable input through the secure path and require the guard to fire. The entropy scanner dies on the first one with no argument to have. That turns "does the fix work" from a review opinion into an executable assertion, and the fraction that fires is the recall figure I do not currently have.

The honest constraint: a slice of the corpus is illustrative multi-turn snippets, not standalone programs with a defined input interface, so I can't mechanize it uniformly. But the failure mode you are pointing at lives in the executable subset, a scanner, a sanitizer, an auth gate, and those are exactly the ones I'm working to wrap in positive controls. Plan: isolate the runnable, secure paths; write a positive control per stated vulnerability; publish the fire rate as a real recall number; and call out the non-executable remainder separately rather than letting it hide inside a pass.

The oracle point is the sharpest part, and I hadn't framed it that way. Mutation-scoring the oracle instead of the artifact, torch.allclose waving through buggy variants, is the same disease as a review that flags nothing on a fake fix. Measuring the oracle is the move I skipped, and I am going to steal it. Thank you for that!

If 28 was just the count review that happened to catch, the positive-control pass will say so. I would rather publish that number than keep the precision one. Thank you sincerely for taking the time to share this.
dipankarsarkar

The non-executable slice is the part I would think hardest about, because it's about to inherit a number it didn't earn.

Once the positive-control pass gives you a fire rate on the runnable subset, that rate is going to read as the corpus's number. It isn't. It's the number for the examples that happened to be shaped like standalone programs.

And there's no reason to expect the remainder is better. If anything the opposite. An illustrative multi-turn snippet never had to run, so nothing ever pushed back on it. The entropy scanner survived because nobody fed it a secret. A snippet nobody can feed anything is that same condition, made permanent.

So I would publish it as two numbers, not one number plus a caveat. Measured recall on the executable subset. Then the remainder reported as unmeasured, carrying the executable rate over as the optimistic prior rather than leaving it as unknown. Unknown rounds to fine in a reader's head. An explicit prior doesn't.

How big is that slice, as a fraction of the 1,412?

·
scthornton

You called it before I had the number, and the number makes your point for you!

Positive controls, run for real: isolate each secure guard, push the paired attack through it, require it to fire. Every fire is gated so the rate can't be precision in disguise: a benign input must not fire, and a kill mutation (swap the guard for a pass-through) must make the fire vanish, which also proves the harness routes through the guard instead of hardcoding a verdict. I self-tested the runner on four known-good/known-bad cases and ran a blind check: a generator built a harness for the entropy scanner without being told it was broken, and the runner returned no-fire. Measuring the
oracle was the move I skipped, and you were right that it is the load-bearing one.

And it earned its keep immediately. One bulk pass I ran (a different model, generating harnesses at scale) reused six canned attack strings across every example; it "found" 106 no-fires. The mutation score gave it away: a 39% harness false-negative rate and a 15% fire rate where careful harnesses hit 100%. Those 106 were the oracle misfiring, not defects. I threw the batch out and regenerated it properly. So: the same disease you flagged, caught by the same medicine, on my own tooling.

Measured recall, a census of the executable subset rather than a sample: AI/ML 442 of 442 (100%, 95% CI 99.1-100%), web 77 of 77 (100%, CI 95.3-100%). Every executable secure guard in both configs, 519 in all, fired on its paired attack. Zero no-fires, zero new defects.

Now for the slice, which is the real story, imho. That rate is the number for guard-shaped examples, not the corpus. AI/ML is mostly guard-shaped, about 66% runnable, so the 100% means something. Web is not: positive-control-runnable is 79 of 1,249, roughly 6%. Not because web fixes are worse, but because most are a different object. 56% have no isolable callable at all: the fix is a Spring or Flask annotation, a security-header middleware, a TLS setting, a dependency pin, or secrets moved to env variables. 14% is a language I don't run, 12% is JavaScript I haven't wrapped. There's no input to feed.

So...I'm publishing two numbers, not one, plus a caveat: measured recall on the executable subset, and the remainder as unmeasured-by-positive-control. One tweak to the framing, though. The remainder isn't unpushed-on. It got the same adversarial review, which is how the dead-code defects got caught; it just lacks a mechanized recall estimate. So it keeps the human-reviewed status rather than inheriting a rate your own argument says is probably worse.

The limit I didn't expect: even on the executable subset, positive controls catch one class. I re-ran our three removed defects as known positives. The check caught one: the entropy scanner, broken guard logic. It missed two: correct guards that were dead code or never saw the data they were meant to scrub. Isolation cannot see reachability. One of three. The fire rate is broken-guard-logic recall, complementary to the review that caught the rest, not a replacement.

Where the fix is a guard, it holds up. Where it is config, positive controls are the wrong instrument, and I would rather say so than score it anyway. Thank you, and I mean that sincerely. This thread made the dataset more honest than the audit did.
dipankarsarkar

The two misses are the sharper result, not a limit of positive controls. They're a second oracle you haven't built.

A positive control asks one thing: does the guard fire when the attack reaches it. Both misses fail a different question: does the attack reach the guard at all. Dead code is a guard that's never on the path. "Never saw the data" is a taint gap, the scrubbed variable isn't the one that reaches the sink. Isolation is blind to both, because isolation is the act of deleting the path.

So the fire rate is reached-guard-works recall. The complement is reachability, a dataflow question on the original artifact, not the isolated guard. Feed the attack at the real entry point, then assert the guard's branch actually executed and the tainted value actually flowed through it. Dead code fails the first assert. The taint gap fails the second. Two controls, not one blind spot.

Which makes me ask: of your two misses, was one branch-never-ran and one wrong-variable? If they split that cleanly, you don't have a gap. You have your next two oracles.

·
scthornton

They split exactly the way you stated, I verified in the code.

The dead-code case is branch-never-ran. The knowledge-graph example defines three access-control functions, validate-cypher, filter-results, controlled-query, all correct, and the live query path calls none of them. Enforcement there is a sentence in the prompt telling the model to behave, plus a logger.warning that blocks nothing. Feed the attack at the real entry point and assert the guard's branch executed: it never does.

The other is wrong-variable. The Bedrock example has a working log scrubber, and it runs, but on a metadata string that is a version hash and a message length. The decrypted system prompt flows to the model-invocation sink without ever passing through the scrubber. Assert that the tainted value reached the guard: it did not. The scrubbed variable and the leaked variable are different variables.

So you're right, it's two oracles, not one blind spot. And the part I hadn't appreciated is that both are questions about the original artifact, not the isolated guard, which means they are static. Reachability is a call-graph query, and the taint gap is a source-to-sink-through-sanitizer query. Neither needs the example to run, so neither is bound by the executability limit that held positive controls to a fraction of the corpus. That will be the next build. Thank you for drawing the box around it so precisely.
dipankarsarkar

The reachability oracle has a blind spot, and it lands on your 56%.

A static call graph sees direct calls. It doesn't see framework dispatch. A Spring interceptor, a Flask before_request, a registered middleware: the guard runs, but nothing in the code calls it by name. The framework does, at runtime, by convention.

So the naive reachability check reports branch-never-ran on exactly the config-shaped fixes that positive controls already couldn't reach. Same slice, second time. The dead-code oracle's false positives cluster where the annotation is the enforcement.

Your knowledge-graph miss is honest dead code: a plain function defined and never called. That's the easy case. The hard case is a guard that IS wired, just wired by the framework, and a call graph that doesn't model the framework can't tell those two apart. It reads both as dead.

So the build needs a resolver for the dispatch conventions in play before it can call anything dead. Otherwise reachability inherits the config boundary as a false-positive rate instead of an honest unmeasured gap.

Which frameworks does the dead-code slice actually span? If it's one or two, the resolver is small. If it's ten, that's the real cost of the next oracle.
In this post