DOC-foundry: Adversarial Agents That Prove Sigma Detection Rules Before Deploy

DOC-foundry is an open-source project that uses LangGraph agents to write Sigma detection rules—then forces every verdict through deterministic referee code before anything can deploy. Maintained on GitHub as DOC-foundry (public since July 2026; README and examples updated through November 2026), it targets the gap between “an LLM gave me YAML” and “I can prove this rule fires on attacks and stays quiet on benign traffic.” If you run coding agents that touch production credentials, the architecture is worth studying even if you never author a Sigma rule: it is the same agents propose, harness proves split security teams are adopting for offensive automation and now for defensive detection engineering.

TL;DR

What the project ships

According to the project README, DOC-foundry (Detection-as-Code Foundry) ingests arbitrary threat text and outputs three linked artifacts:

1. A Sigma detection rule that survived a Blue-vs-Red adversarial loop.
2. A threat report with round-by-round convergence history.
3. A proof bundle—positive corpus, benign baseline, adversarial mutation set, and a computed coverage label (real-coverage, brittle, or broken) with FP/FN numbers and a reproducibility hash.

The README stresses that large language models can draft Sigma YAML easily; what they cannot honestly supply alone is a proof that the rule works, a documented adversary that evades weak versions, and a record of how the rule hardened until it held.

Attack chain on bad detection rules (the problem DOC-foundry targets)

Detection engineering has a familiar failure mode that looks like a supply-chain issue for defenders:

1. Threat lands — a KEV entry, incident write-up, or ATT&CK technique becomes urgent.
2. Analyst or LLM drafts YAML — often anchored on brittle strings (single binary path, one keyword).
3. Rule deploys on optimism — no systematic benign baseline, no rename/obfuscation battery.
4. Attacker adapts — binary rename, shell obfuscation, or parent-process lineage tricks (the Joomla iCagenda example in the README's review-history walkthrough).
5. SOC drowns or misses real RCE — either noisy false positives or silent false negatives.

DOC-foundry automates steps 2–4 inside a loop, but keeps step 5's verdict out of model hands.

Mechanism: Blue proposes, Red attacks, code judges

The README diagrams a self-repair LangGraph loop:

ingest → extract → telemetry
  → blue_author → evaluate → battery → red_propose → red_verify → verdict
       ↑________________________________ repair if verdict ≠ real-coverage __|
  → gate → bundle

LLM agents (src/agents.py) — four personas. The Blue Detection Engineer authors behavior-plus-intent rules (not one brittle string) and re-authors on brittle or broken verdicts using concrete evasion evidence. The Red-Team Adversary commits to two real, documented attacks in round one and re-throws those same two every round so Blue closes a bounded gap rather than chasing a moving target.

Deterministic referee — evaluate, mutation battery, red_verify, verdict, and gate nodes. The README states explicitly: same input → same verdict → same reproducibility hash.

The proof core refuses to fake coverage

src/engine.py is described as a zero-dependency Sigma matcher with an honesty contract: it raises on constructs it cannot truly evaluate (|cidr, aggregations, invented |in operators) rather than silently passing. An un-evaluable rule becomes a broken round; Blue gets the engine's error message as repair evidence.

src/evaluator.py runs FP/FN evaluation plus a deterministic adversarial mutation battery (binary rename, casing, whitespace, caret/quote obfuscation, path relocation). red_verify replays red-team events through the engine—a bypass counts only if the engine confirms the rule failed to fire. Unverified adversary claims are discarded.

Research grounding without touching the hash

A lexical BM25 retriever over SigmaHQ (authoring) and MITRE ATT&CK plus LOLBAS (evasion) grounds proposals. The README says this RAG never enters the reproducibility hash and never touches the proof engine—keeping creative suggestions separate from scored truth.

Second pipeline: stakeholder review chain

python src/run.py --chain "…" runs a different question: what would Red Team and SOC Lead say, and what rule ships after both pressures?

The documented flow is draft → Red Team review → Blue revision (citing finding IDs) → SOC Lead review → final rule, packaged as final-rule.yml, review-history.md, and review.json. The README warns this chain is reviewed, not validated—no FP counts, no coverage label unless you also run the classic proof loop or --evals.

The worked Joomla iCagenda CVE-2026-48939 example (CVE-2026-48939) shows Red Team pushing Blue to broaden against evasion while SOC Lead pushes narrowing against alert volume—tension the authors treat as the product.

Governance: Kartikeya Gate and MCP deploy

Deploying a detection is modeled as a capability request (egress to a SIEM). src/governance.py scores Gate = Blast Radius × Capability Delta; the README states egress always requires a human and non-real coverage floors blast radius.

src/mcp_server.py exposes a governed tool boundary: the graph cannot fetch or deploy freely. deploy_detection refuses approval tokens not bound to the exact bundle hash. The harness holds privilege, not the agent—a pattern closer to how mature coding-agent setups treat shell and network tools than to “the model clicked deploy.”

Running it (from the README)

pip install -r requirements.txt

# Live — needs GEMINI_API_KEY or ANTHROPIC_API_KEY
python src/run.py "CVE-2026-45659 SharePoint deserialization RCE"
python src/run.py "T1218.011 rundll32 remote scriptlet execution via mshtml"

# Offline canned convergence demo
python src/run.py --threat fixtures/certutil_t1105.json

python src/run.py --evals          # deterministic engine + proof tests
python src/run.py --approve alice    # human approves → gated deploy succeeds

Bundles land under Outputs/<run>/ as rule.yml, report.md, and proof.json. The web UI (python -m uvicorn webui.server:app --port 8000) streams real LangGraph node events and token output; cached replays are labeled REPLAY (cached) so demos cannot masquerade as live generation.

What this means if you run AI coding agents

DOC-foundry is not a coding-agent harness. Nothing in the README integrates with Claude Code, Codex, or Cursor hooks. It still matters for the same reasons GTG-10007's autonomous exploit foundry mattered on the offensive side: standing agent loops with memory and deploy-adjacent tools scale what one operator can ship.

Concrete parallels for developer and security teams:

1. Do not treat fluent YAML or shell as proof. The Qwen CTI-REALM detection-rule task showed a model asked only to write a rule still probing the host. DOC-foundry's answer is a fixed corpus and a matcher that fails closed on unevaluable syntax—borrow the discipline, not necessarily the SIEM format.

2. Separate propose from prove in your own agent policies. If a coding agent drafts Terraform, CI workflows, or detection content, decide which steps require deterministic checks (linters, test harnesses, policy-as-code) before merge—mirroring red_verify replaying through engine.py.

3. Bind high-impact actions to human intent and artifact hash. DOC-foundry's MCP deploy gate is analogous to refusing to run terraform apply or push production rules unless a human signed a specific commit or bundle ID. That aligns with enterprise rollout checklists that separate experiment from production credentials.

4. Expect false confidence from review without execution. The review chain's explicit disclaimer—that four professional reviewers can make a rule look finished without firing it on telemetry—matches Andy Hall's finding that high-recall AI auditors drown humans in false alarms. Automated proof and human review solve different parts of the problem.

5. Hunt agent-generated egress, not just attacker LLM API calls. FBI FLASH guidance on weaponized ARTEX lists unexpected calls to frontier model APIs alongside burst enumeration. DOC-foundry's governed egress is the defensive mirror: know when your agents request outbound capability.

Checks you can run without installing DOC-foundry:

• Inventory which teams already use chat-to-YAML for detections or playbooks, and whether those flows record who approved deploy and which test corpus was used.
• Audit whether any repo workflow lets an agent open a PR that merges detection or infra changes without a required check job.
• List MCP servers your coding agents load; confirm none expose unpinned deploy or cloud-write tools without hook denies (MCP scanning docs).
• Run python src/run.py --evals in a sandbox if evaluating the tool—validates the referee independent of model quality.

How Beam handles this

Beam does not author or score Sigma rules. It is local, on-device observability for coding agents: PreToolUse hook denies for Claude Code and Codex where policy is set to enforce, heuristic scans of skills and MCP configs before run, and redaction before events persist under the collector data directory (workspace policy).

Where DOC-foundry gates detection deploy to SIEM egress, Beam gates day-to-day agent actions on the developer machine—credential reads, destructive commands, risky shell patterns. The overlap is architectural: privilege stays in the harness, not the model. Beam's limits are explicit: heuristic scan, not semantic malware analysis; missing cached policy is fail-open; enforce is hook-based, not a kernel sandbox. For session-level behavior rules across transcripts, see Cisco Project Quarry; for VM isolation of agents, see Capsem. Those layers stack; none replaces deterministic test corpora for detection YAML.

Frequently asked questions

What is DOC-foundry?

DOC-foundry is an open-source project on GitHub (mjha-arya/DOC-foundry) that runs an adversarial multi-agent workflow to turn threat text—such as a CVE, CISA KEV line, or ATT&CK technique—into a Sigma detection rule, a written threat report, and a proof bundle with positive, benign, and adversarial test sets plus a computed coverage label.

Do the LLM agents decide whether a rule is correct?

No. Four role-specialized agents (Threat Analyst, Blue Detection Engineer, Telemetry, Red-Team Adversary) propose rules and evasions, but FP/FN evaluation, adversarial mutation batteries, coverage verdicts, and the deployment gate are implemented as deterministic referee code. The README states the same input yields the same verdict and reproducibility hash.

What API keys does DOC-foundry need?

Live authoring requires GEMINI_API_KEY or ANTHROPIC_API_KEY per the README. Canned fixture runs work fully offline with python src/run.py --threat fixtures/certutil_t1105.json and python src/run.py --evals for engine tests.

How is the Review Chain different from the proof loop?

The classic pipeline proves correctness via engine scoring and issues labels such as real-coverage, brittle, or broken. The --chain pipeline runs draft, Red Team review, revision, SOC Lead review, and final rule packaging with review-history.md, but does not execute rules against telemetry or issue a coverage verdict unless you also run the proof loop.

Can an agent deploy a rule without a human?

The README states deploy_detection on the FastMCP server refuses any approval token not bound to the exact bundle hash, and that egress always requires a human under Kartikeya Gate scoring (blast radius times capability delta). The harness holds deploy privilege, not the graph.

Does DOC-foundry replace monitoring tools like Beam or Project Quarry?

No. DOC-foundry authors and validates SIEM-facing Sigma rules from threat intel. Beam observes local coding-agent tool use and can deny via hooks; Cisco Project Quarry applies session-level behavior rules to agent transcripts. They operate on different layers of the stack.

Related reading

• GTG-10007: inside an AI-run autonomous exploit foundry — offensive-side agent automation at campaign scale.
• Cisco Project Quarry: session-level agent behavior rules — Snort/YARA-style rules over whole agent sessions.
• AI auditors, false alarms, and agent log review — why review without measured FP rates fails at scale.
• Capsem: VM sandbox for coding agents — isolating where agents run versus what they do.
• Beam landing: what Beam watches and local-first storage.

Version numbers, repository layout, and API requirements are accurate as of November 14, 2026, per the DOC-foundry README on GitHub.