When an AI agent fixes a bug, the diff you review rarely shows what else broke. Trail of Bits looked at its own track record and found the problem is real: among more than 130 merged upstream pull requests, it confirmed 21 regressions, and has now released two agent plugins, regression-hunt and patching-bugs, meant to catch them. The write-up, Your AI agent fixed the bug, but what else did it change?, is a useful reference for anyone who lets coding agents write patches.
This is a tooling post, not an incident. It matters to readers of an agent-security blog because the unintended side effects of an agent's change are the quiet cousin of the loud failures we usually cover: no attacker, no injection, just a patch that fixed one thing and altered another.
Quick answers
| Question | Answer |
|---|---|
| What is it? | Two Claude Code-style plugins from trailofbits/skills: regression-hunt and patching-bugs. |
| What problem do they address? | Agent patches that fix the reported bug but introduce regressions elsewhere. |
| Does it run code? | Yes. Every claim needs a live proof of concept on a built, running instance, or the result is "inconclusive". |
| Does data leave my machine? | The post does not say; the plugins drive your own agent session, so that depends on the model provider you use. |
| Is it proven? | Preliminary: about 25% fewer false positives than a baseline prompt, on 21 known regressions. |
The data behind it
Trail of Bits reviewed every patch its engineers wrote for upstream open-source projects. Of more than 130 merged PRs, 21 contained confirmed regressions: 16 functional, one performance and four affecting build or tests. The authors state none broke a security protection, which is worth noting: this is a quality problem, not a vulnerability count. Agents also produced noisy output when asked to find these: in one run, 21 confirmed regressions came from 58 candidates, roughly 36% precision.
That noise is the practical issue. A reviewer agent that cries wolf trains people to ignore it. Trail of Bits concluded that patches needed both a standard way to be written and a standard way to be checked.
How the plugins work
The post says the plugins came from analysis of merged patches and thousands of Codex sessions, and lists four design principles that apply to both.
- Minimal, opinionated guidance. The agent gets wide latitude, constrained only by Trail of Bits' judgment of what a good patch and a real regression look like.
- Modular steps. Skills can run end to end or separately.
- Strict schemas. Each step takes and returns a defined structure: a claim, a code revision, a required check, raw evidence and open questions. This lets each step verify the previous one.
- A live proof of concept for every claim. Results come from a built, running instance of the project. If a step cannot produce a live PoC, its result is "inconclusive".
regression-hunt
It runs four stages:
- Enumerate candidate regressions by tracing the diff through callers to real interfaces.
- Reproduce each candidate on a live deployment.
- Replay the PoC with a fresh-context agent, so one agent's assumptions do not carry over.
- Adjudicate whether the proof meets the regression criteria.
The burden of proof sits with the adjudicating agent, which must rule out false positives, incomplete fixes, latent bugs and bugs that existed before the patch. If a threat model exists, the orchestrator hands it to every step.
patching-bugs
The post describes this as a distillation of best practices from Trail of Bits engineers' Codex sessions. It adds a three-arm proof: the reproduction must fail on the original code, pass on the patched code and fail again on a full revert. It guides invariant-based repair, fixing the violated security invariant rather than only the specific bug so that variants are covered. And it requires regression-hunt on every patch.
What the evaluation does and does not show
On the 21 known regressions, regression-hunt produced about 25% fewer false-positive findings than a baseline Codex prompt asking it to review the PR for regressions. The 95% confidence interval is 14% to 38%. The trade-off the authors name: strict evidence rules mean some known regressions were classified borderline because the plugin lacked enough evidence to confirm them.
Read that carefully. It is a vendor evaluating its own tool on its own data, it is preliminary (an evaluation of patching-bugs is still in progress), and the authors say they are analysing trace data to see which rules Codex skips and where its evidence is weakest. The honest reading is that requiring live proof makes the agent more disciplined, not that it makes patches safe.
Why agent patches regress, in our reading
The post does not claim a single cause, so the following is our interpretation of its design rather than something Trail of Bits states. An agent asked to fix one bug optimises for the bug it was shown. The test it writes usually exercises the reported path, and the diff it produces is judged by whether that test passes. Callers of the changed function, build scripts that assume the old behaviour and performance-sensitive paths are outside that frame unless something forces the agent to look.
The four-stage design reads like a response to exactly that. Enumerating by tracing the diff through callers pushes the agent out of the changed lines. Reproducing on a live deployment replaces "this looks wrong" with evidence. Replaying with a fresh-context agent removes the first agent's framing. Adjudicating with the burden of proof on the judge pushes back against the tendency of language models to agree with a confident-sounding claim.
For teams, the practical consequence is where to spend attention. A regression that breaks the build or slows a path is cheap to catch if you ask for it, and expensive if you find it in production. A security regression is rarer but costlier, and Trail of Bits reports none among its 21 confirmed cases, which is reassuring for that sample and says nothing about yours.
How to adopt it without over-trusting it
If you run agents that open pull requests, a sensible rollout is gradual. Start by running regression-hunt on patches that have already merged, so you can compare its findings with what you know went wrong and see its precision on your codebase before it gates anything. Treat "inconclusive" as a prompt for a human look, not as a pass. Keep the threat model short and current, because the plugins hand it to every step and a stale one will mislead all of them. And record which findings you accepted or rejected: the authors say they are analysing traces to find where Codex skips rules, and your own log gives you the same signal about your setup.
Also remember that a plugin which builds and runs your project executes code. Run it where a misbehaving build step cannot reach production credentials, ideally in a container or VM, for the same reason you would sandbox any agent that runs your test suite.
How to try it
The post lists these steps:
- Install
patching-bugsandregression-huntfrom the trailofbits/skills repository. The post does not give install commands; follow the README. - Add
THREAT_MODEL.mdto your repository. It defines what counts as a vulnerability or regression, covering assets, trust boundaries, access controls, supported configurations and the evidence a valid bug report needs. Trail of Bits links a GPAC example. - Add
PATCHING_GUIDELINES.md(orAGENTS.md) explaining how fixes should be written: build and test steps, fix size and placement, how to present changes, and what a regression test must show. - Run the plugins on your next bug report and send feedback through the repository's issue tracker.
Planned work includes configuration options such as minimising patch sets or loosening regression-hunt's rules to surface more candidates.
What this means if you run AI coding agents
Even without these plugins, these habits (our reading of the post's design, not Trail of Bits' advice) are worth adopting.
- Write down what "broken" means. A short threat model and patching guideline in the repository gives any agent, including a reviewer agent, something to check against. It is the same idea as an
AGENTS.md, aimed at review. - Demand reproduction, not assertion. Ask the agent to show a test that fails before and passes after, and fails again when you revert. That three-arm check is cheap to request manually.
- Use a fresh-context second pass. Having a separate session replay a claim catches errors the first session talked itself into.
- Treat plugins and skills as code you run. Installing a plugin from a repository gives it the same access as your agent. Read what a skill instructs before you enable it. Our guide to agent skills and their security review covers what to look for, and the Cloudflare security-audit skill post walks through running one safely.
- Keep humans in the loop for risky diffs. Research summarised in humans miss 1 in 3 AI agent command threats shows review fatigue is real, which is why narrowing what a reviewer must look at helps.
See also the vibe coding security checklist for a broader pre-merge list.
What Beam can and cannot do here
Beam does not review code diffs or find regressions, and it does not run PoCs. It is a local tool for what agents do while they work: it watches Claude Code and Codex sessions, flags risky actions such as credential exposure and destructive commands, and beam scan checks a skill or instruction file for risky patterns before you run it. If you adopt a third-party plugin like these, scanning its skill files first is a reasonable step. The scan is heuristic: it does not execute the skill and cannot judge whether its instructions are sound. See Beam's security skills and subagents for how Beam packages its own.
Frequently asked questions
What did Trail of Bits find about its own patches?
Reviewing every patch its engineers wrote for upstream open-source projects, Trail of Bits confirmed 21 regressions among more than 130 merged PRs: 16 functional, one performance and four build or test. It says none broke a security protection.
What are regression-hunt and patching-bugs?
Two plugins in the trailofbits/skills repository. regression-hunt enumerates, reproduces, replays and adjudicates candidate regressions in a diff. patching-bugs guides how a fix is written, with a three-arm proof and invariant-based repair, and runs regression-hunt on every patch.
How well does regression-hunt work?
In preliminary evaluation on the 21 known regressions it produced about 25% fewer false-positive findings than a baseline Codex prompt to review the PR for regressions (95% confidence interval 14% to 38%). Some known regressions were classed borderline because evidence was insufficient. It is a vendor evaluation of its own tool.
How do I try them?
Install the plugins from github.com/trailofbits/skills, add a THREAT_MODEL.md and a PATCHING_GUIDELINES.md or AGENTS.md to your repository, and run them on your next bug report. The post does not publish install commands, so follow the repository README.
Do these plugins make agent code safe to merge?
No. They raise the bar by requiring live evidence, but results are probabilistic, the evaluation is preliminary and strict evidence rules mean some regressions are reported with low confidence. They complement human review, tests and a sandbox for the agent.
Related reading
- Agent skills specification and security review
- Cloudflare security-audit skill: run it safely
- Humans miss 1 in 3 AI agent command threats
- Vibe coding security checklist
- How Beam works
Details reflect Trail of Bits' post and repository as of October 21, 2026. The plugins are evolving; check the repository for current behaviour.