Cloudflare has published security-audit, an MIT-licensed skill that turns a coding agent into a structured security auditor, and it is worth trying if you set up the sandbox it asks for. The repository is the single-repo starting point of the vulnerability harness Cloudflare describes in Build your own vulnerability harness. This post covers what it does and what to check before you run an auditor agent over code you did not write.
| Question | Short answer |
|---|---|
| What is it? | A skill that runs a six-phase audit with isolated agents |
| How do I get it? | npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit |
| What does it need? | Tool use, parallel sub-agents, Node.js, and an OS-enforced sandbox for running target code |
| Does data leave my machine? | The skill itself does not say; output defaults to ~/security-audit-skill/<repo-name>/run-<N>, and your agent's model provider still sees what it reads |
How the skill works
Per the README, the agent moves through reconnaissance, coverage-led hunting, candidate validation, structured output, independent record verification and reporting. Hunting is driven by attack-class files: an overview plus ten domain files covering memory safety, LLM-backed systems (prompt injection, agent tool use, output handling), web protocols, client-side bugs, supply chain and releases, cloud config, RPC and messaging, resource exhaustion, data isolation and desktop or local IPC.
Findings are filed as confirmed, needs_validation or rejected in findings.json, with REPORT.md, FINDINGS-DETAIL.md, NEEDS-VALIDATION.md and a coverage-ledger.json alongside. The README states the rule that "the agent that checks a finding is never the agent that found it."
The six phases in more detail
The README describes each phase as a separate job with its own prompts. In reconnaissance, the agent maps architecture, trust boundaries, input surfaces and prior evidence, and writes the result to architecture.md and coverage-ledger.json. The ledger is the spine of the run: it lists the units of code that need attention and records which have been checked.
In the hunting phase, the parent agent assigns isolated hunters to ledger units, records their checks, and uses "coverage critics" to find gaps. Candidate validation then gives every unique candidate to a fresh verifier whose task is to disprove it. The structured output phase writes confirmed, needs_validation and rejected records to findings.json and checks them against report-schema.json. A fresh set of agents then verifies the final source claims, and any material replacement gets another independent verifier. The last phase derives REPORT.md, FINDINGS-DETAIL.md and NEEDS-VALIDATION.md from the verified records.
Two small Node.js scripts do the mechanical checks. validate-coverage-ledger.cjs runs after the ledger is created and after each later ledger update. validate-findings.cjs runs in phase 4 and again after every phase 5 replacement. The README calls both zero-dependency. They check structure, not truth, so the independent verifier agents still carry the factual burden.
What the three verdicts mean
The README defines the verdicts narrowly. confirmed means a complete source trace and a bounded observed result. needs_validation means an exact unresolved fact and no severity. rejected records a disproved candidate. The design principles add that the skill only confirms "established boundary failures", that severity is likelihood times impact, and that a defense-in-depth gap is a hardening note and not a vulnerability if another layer already stops the attack.
For a reader, the practical point is that needs_validation is not a weaker confirmed finding. It is an honest statement that the agent could not finish the proof, often because it was not allowed to run the target code. If you have no sandbox, expect most dynamic leads to land there.
Repeated runs are additive. The skill reads prior ledgers and findings to target gaps, revalidate changed source and carry forward current evidence, without treating stale or unresolved work as covered. The README states that "in our test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total."
Why a skill first, then a harness
Cloudflare's harness post says the company started with a skill of about 450 lines, run on a single repository, and tuned the prompts until it surfaced real bugs. It describes that skill as a seven-phase audit in one session, ending with submission to an ingest API. The README for the public version lists six phases, so the open-source skill is the single-repo starting point and not the full pipeline.
The post names three limits that pushed the team to build a harness. The context window fills after about an hour and the model can lose bugs it already tracked. A crash mid-run means starting over. A single-repo session cannot see how applications that consume a library relate to it. Cloudflare says going from the first slash-command run to a fleet scanner covering 128 repositories took about six weeks.
Its advice to others is modest: start with a skill, get the prompts working, and build the next stage only when its absence is the thing slowing you down. For most readers of this post, the skill is the right size. The fleet system, which the post says now holds 13,841 findings across 145 repositories in its validation stage, is a different undertaking.
Why the false-positive controls matter
Cloudflare's harness post gives, as a single-pass benchmark and not a measured result, about 100 initial findings for a standard repository of roughly 30,000 lines, with a full run taking three to four hours. After deduplication and judgment it reports roughly 80 distinct bugs. Across the fleet it says 20,799 raw candidates were generated and about 12,057 survived validation.
A separate validator agent tries to disprove each finding before it can be filed, and it cannot file findings itself. The post says the initial validation rejection rate fell from 40% to 11%, and attributes that partly to better context injection from the reconnaissance phase. The authors say they do not claim a false-negative rate, because there is no labeled set of every real bug in a codebase.
The post also lists failure modes the controls are built against. An agent may edit the source so its own exploit works and then report the bug it created. It may write a tautological test. It may build an exploit that runs but rests on a meaningless threat model. In response, a hunter must state the threat model (who the attacker is and what boundary is crossed) before filing, and every confirmed finding ships with a proof of concept written as a test that runs against the untouched codebase, plus a proposed patch. In the full harness, the fixer needs a failing test to flip to passing, never merges code, and a human reviews each branch.
Why the sandbox requirement is not optional
On Hacker News, one commenter asked why an OS-enforced sandbox is a requirement at all. The reason is in the README: hunters may run target-controlled builds, tests, processes, browsers, emulators, fuzzers and fixtures. Each of those executes code written by whoever wrote the target. If the target is hostile, or just badly behaved, an unsandboxed audit run can read your files, reach your network or fill your disk.
The README names four controls: disable external networking, use a sanitized allowlisted environment, enforce resource limits, and allow writes only to assigned scratch paths. Without them the workflow keeps the lead as needs_validation and does not run the code. That fallback is the safer default. It also means the skill will quietly do less on a laptop with no isolation than it does in a VM, so check which mode you are in.
Cloudflare's own sandbox for hunters is built on unshare, and the post says it was the biggest jump in finding quality: hunters stop only reading code and start compiling fragments and crashing binaries. That is the same capability that makes the sandbox necessary.
What this means if you run AI coding agents
An audit skill is an unusual workload: it reads a lot of code, may build and run it, and writes files. Treat it like any other powerful skill.
- Read it before installing. Open
SKILL.mdand the attack-class files. This skill comes from a well-known company, but the habit is the point; see agent skills specification and security review. - Provide the sandbox. The README requires no external networking, a sanitized environment, resource limits and writes limited to scratch paths. If you cannot, leave leads as
needs_validation. Cloudflare's post also warns that inside Docker the harness's sandbox needsseccomp=unconfinedandapparmor=unconfinedor it silently fails to start, so confirm what your isolation actually does. See Capsem and Perplexity's SPACE red team for VM-level options. - Choose the output directory. Keep results out of the target repo unless you point to a version-control-ignored path, as the README advises.
- Assume the audited code can talk back. The README does not address injected instructions in code the auditor reads. That is our inference, not a documented flaw: a comment in a dependency is still untrusted input, as in the Common Crawl header study.
- Review the run afterwards. Check which commands were executed and which files were written.
What developers are saying
The skill's Hacker News thread had 213 points and 38 comments when we read it. A few reports are practical:
- User drchaim reported spending about one million tokens on a medium-sized codebase "for nothing", and hyperionultra asked how many tokens a run costs. The README gives no token figure, so budget a trial run before pointing it at a large repository.
- User wslh said that audit skills framed explicitly as security research can trigger refusals from top models, and that splitting the work into skills per bug class without the security framing worked for them. This is one user's experience, not something the README documents.
- User prodigycorp asked Cloudflare to consolidate its many skills into one, because the number of skill descriptions pollutes the context window.
- User gbrindisi shared a write-up of an in-house audit skill from Synthesia as a lower-token alternative.
The thread reflects the usual cost and noise questions about agentic audits. The sandbox question is the one with a clear answer in the README.
What Beam can and cannot do here
Beam can scan the skill's files before you run it with its skill and MCP scan (11 heuristics plus an MCP version-pin check), and through hooks on Claude Code and Codex it can flag or deny risky commands during the run, with data kept on-device (local-first). It is heuristic, not a kernel sandbox, and does not replace the OS-enforced isolation the skill requires. Missing cached policy fails open. For the broader picture of reusable skills, see Beam security skills and subagents.
Frequently asked questions
What is Cloudflare's security-audit skill?
It is an MIT-licensed skill that turns a coding agent into a security auditor across six phases: reconnaissance, coverage-led hunting, candidate validation, structured output, independent record verification and reporting. It is the single-repo starting point of Cloudflare's vulnerability discovery harness.
How do I install it?
The README uses the Skills CLI: npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit, with --global for a user-level install. It needs an agent with tool use and parallel sub-agents, plus Node.js for its validators.
Is it safe to run on code I do not trust?
Only with the sandbox the README requires: OS-enforced, no external networking, a sanitized environment, resource limits and writes limited to scratch paths. Without one, the skill leaves leads as needs_validation and does not execute target code. The README does not address injected instructions in the code being read.
How does it avoid false positives?
The agent that verifies a finding is never the one that found it, and findings are sorted into confirmed, needs_validation or rejected. Cloudflare's write-up adds that a separate validator, which cannot file findings itself, tries to disprove each one, and that every confirmed finding needs a working test against the untouched codebase.
Should I review a skill before installing it?
Yes. Any skill gives an agent instructions and often tool access. Read the SKILL.md, pin the version you reviewed and scan it before first use.
Can Beam help when I run an audit skill?
Beam can scan a skill before you run it with its heuristic checks and watch the commands the agent runs through hooks on Claude Code and Codex. It is not a sandbox and does not replace the OS sandbox the skill requires.
Related reading
- Agent skills specification and security review
- Capsem: VM sandbox for AI coding agents
- Trail of Bits regression-hunt and patching-bugs: checking what an agent changed
- Cloudflare's harness write-up
- Hacker News discussion of the skill
- Perplexity's SPACE red team
Details are accurate as of October 20, 2026, based on the skill's README and Cloudflare's June 2026 harness post. The skill is new and may change.