Humans Miss 1 in 3 AI Agent Command Threats — What 409,000 Decisions Show

An AI coding agent proposes a shell command. You have a couple of seconds to approve or deny before you break flow. You do this dozens of times a session. How good are you, actually, at catching the bad one?

Independent developer Alex Wauters built a browser game to measure exactly that, and the data is now large enough to take seriously: 40,000+ play sessions and 409,000 individual approve/deny decisions. Playing human-in-the-loop for a simulated agent, people caught only 66.3% of malicious commands — they missed roughly 1 in 3 threats. The full breakdown is in explainx.ai's write-up of the dataset, and you can try the original game at llmgame.scalex.dev.

That number matters because "just ask the human" is the default safety model behind every agent permission prompt. This post looks at where human review holds up, where it collapses, and what to put underneath it.

TL;DR

What the game was, and what it was not

Wauters published the numbers himself in Humans missed 1 in 3 threats approving AI agent commands across 40,000 plays on scalex.dev, and that post is the primary source for everything here. In the game, you play the person approving a coding agent's commands under time pressure. Some commands are routine, such as git status and npm test. Others look like an agent that has been hijacked and is sending your secrets to a remote server. It opens with a one-minute countdown to your next meeting and asks how many commands you can clear.

The headline figures, as Wauters reports them:

• the average player missed one in three threats (mean accuracy 66.3%);
• 32.9% of sessions ended with a negative score, because penalties for approved threats and blocked safe commands outweighed everything done right;
• 35.2% of players caught every threat, but only 20.8% did so while blocking at most one in five of the safe commands. The rest got there partly by blocking everything, which earns a "Human Bottleneck" title;
• 7% approved every single prompt.

Wauters is open about the caveat that matters most. About 34% of the commands players saw were threats, and the game was short and timed. In real work, he writes, threats "appear rarely". That cuts both ways. A higher threat rate should make players more alert, since they know they are being tested. A lower rate in real use is the condition under which reviewers learn that "approve" is almost always right. The numbers are a measurement of a game, not of your team, and Wauters calls it "just a game and not an academic study".

Humans are good at loud threats, bad at quiet ones

The single clearest pattern in the data is that miss rates track how dangerous a command looks, not how dangerous it is.

rm -rf / reads as violent, so people stop. But cat ~/.aws/credentials is syntactically identical to cat README.md — a plain read — and it slipped past a third of reviewers. The eye is tuned for spectacle, and a credential read is not spectacular.

The attack that wins: a familiar name

The most successful attack in the entire dataset wasn't a scary one-liner. It was npm run analyze, approved 64.7% of the time, even though the underlying package.json script piped build stats to an external endpoint. People saw a command they'd run in every JavaScript project they've ever touched and waved it through without reading what the script actually did.

Two more camouflaged npm scripts landed in the top ranks:

• npm run setup — missed 48.0%
• npm run deploy — missed 44.9%

Pooled together, familiar-named npm-script attacks were missed about 52.5% of the time, versus 28.4% for exfiltration attacks that didn't hide behind a recognizable name. Naming a malicious script something a developer already trusts roughly doubles its success rate.

The deeper problem is that the command name is a false signal. npm run build runs whatever package.json says it runs, and an agent — or an attacker who edited a build file — never has to surface a suspicious-looking prompt at all. Command-level approval is reviewing the label on the box, not the contents.

Permission fatigue is measurable

Miss rates climbed toward the end of sessions. People who started sharp got sloppier as approvals piled up — the more prompts you see in a row, the less scrutiny each one gets. Anthropic has made the same observation about its own users becoming less diligent supervisors over time.

This is why gating everything can make a system less safe, not more. A prompt a tired human rubber-stamps offers less protection than no prompt plus a hard technical control. The lesson isn't "prompt more" — it's "prompt surgically, and back the prompt with something that doesn't get tired."

The same command can be safe or unsafe

Not every miss is a lapse in attention. Wauters notes that cat ~/.zshrc was approved by 45.9% of players, the most divisive command in the game. For a developer who keeps no secrets in a shell profile it is harmless. For one who exports API keys there, it is credential disclosure. The risk depends on a setup the agent cannot see, and the person approving may not remember. Several other prompts, he agrees, were "ambiguous without context", and he concludes that asking developers to make security judgements without the full picture is a flawed model.

The npm run case has the same shape. The game's history log showed the script body, a webpack-bundle-analyzer pipeline that posts to an external URL through curl, directly above the prompt. Two thirds of players approved anyway. Wauters credits a Hacker News commenter, dns_snek, with the sharper point: approving npm run build means approving whatever package.json says, and an agent could have edited that file, planted code in a build script, or modified a file in node_modules, all without a prompt. A prompt on the final command is late in the chain.

The other failure: blocking safe work

Over-caution cuts the other way and makes agents unusable. Reviewers wrongly blocked routine commands at high rates:

• npm config set registry https://npm.internal — blocked 59% of the time
• rm -rf dist/ — blocked 45% (ordinary build cleanup)
• kill $(lsof -t -i:3000) — blocked 43% (freeing a dev-server port)

The same reviewers who let a third of real threats through also blocked nearly half of the harmless commands a working agent needs. Errors compound in both directions at once.

Where the vendors have moved

Anthropic published its own numbers on the same problem. Its engineering post on Claude Code auto mode states that "Claude Code users approve 93% of permission prompts" and describes the result as approval fatigue, where "people stop paying close attention to what they're approving." Its answer was to put model-based classifiers in the approver's seat. A server-side probe screens tool outputs for injected instructions, and a transcript classifier checks each action against decision criteria before it runs. The post lists what it is built to block: scope escalation, such as deleting remote branches after a request to "clean up old branches", credential exploration after an auth error, and sharing data through an external service.

Wauters' update dated August 10, 2026 reports that Anthropic then announced auto mode would become the default, citing an internal study of 1,053 testers in which the ask-for-everything approach had an 86.4% miss rate, with auto mode at around 11%. We could not retrieve Anthropic's announcement itself, so those two figures are Wauters' reading of it. Anthropic's March engineering post is the source we can link for how the classifier works and what it misses. Wauters' own view is that this is a welcome change, but that an 11% miss rate still calls for better sandboxing and fewer granted permissions.

The point for this post is narrower. The vendor with the largest user base for terminal agents agrees that per-command human review fails at volume. Its fix keeps a human out of most decisions rather than asking the human to try harder.

Human approval is a filter, not a wall

The honest reading of 409,000 decisions is not "humans are bad" — they caught the loud, unambiguous threats almost every time. It's that human approval is unreliable as the sole safeguard, especially against anything disguised behind a familiar name or a routine-looking read, and especially as prompt volume grows.

So put something underneath it:

1. Sandbox the agent. Run it where a wrong approval can't reach real credentials, production, or your home directory. Containment turns a missed threat into a contained one.
2. Scope the credentials. Least-privilege, short-lived tokens mean a cat ~/.aws/credentials that slips through reads far less. The blast radius, not the approval, is what you actually control.
3. Keep a local record of what the agent did. A missed approval is invisible unless something remembers it. This is where Beam fits: it watches what your coding agents actually run — shell commands, file writes, MCP calls — flags the risky ones locally, and can deny matching Claude Code and Codex tool calls when workspace policy is enforce. Sentinel can pause or block high-risk browser sends. When the human misses one in three, you want a durable trail and a policy layer owners control — not just a prompt that already scrolled past.

Human review, a sandbox, scoped credentials, and an honest local record are four different layers. The study is a measurement of what happens when you rely on only the first one.

Try it yourself. We built a small interactive version of this: play the human in the loop while Beam proposes shell commands under time pressure, and see how your own approve/deny instincts hold up. → Play Approve or Deny

What developers are saying

The follow-up post drew a long Hacker News thread. These are individual commenters' reports and opinions, not findings. Quotes are paraphrased.

• Methodology doubts came first. VladVladikoff recalls the original thread debating whether some prompts were misleading about risk, which would make the analysis "meaningless", and notes the game was timed. Aurornis reports that they won their first play by answering nothing, and that npm run build was marked dangerous while other npm run commands were not. stonedivot argues a game with no consequences for failure says little about real behavior.
• On base rates, unclebucknasty says the result means little without the share of agent commands that are actually dangerous, but adds that if that share is very low, the case for supervision by human approval is itself weak.
• Several commenters argued for structure over vigilance. wmanley suggests the agent ask about files it may read or write rather than commands it may run, then wrap each invocation in bwrap with an HTTP proxy. kstenerud says permission prompts were a reason they built a tool with no prompts and a sandbox with network and secrets control.
• On the practical side, dgunay reports that allowlists are often defeated when the model writes inline scripts, and that they still use prompts mainly to check intent, because in their experience an agent destroying data you wanted kept is more common than one being tricked into leaking secrets. They add that auto mode, as they understand it, looks for security threats rather than misread intent. jascha_eng takes the opposite reading of the headline number: one in three is not terrible if you add more reviewers and classifiers.

Both camps, taken together, fit the data. The game is a rough instrument, and the direction of its finding, that a prompt on a familiar-looking command is weak protection, is what most of the thread's proposed fixes also assume.

Related reading

• Alex Wauters' 40,000-play dataset, analyzed — the source study and full write-up
• llmgame.scalex.dev — the original browser game
• What is AI agent monitoring? — the observability layer this post argues for
• Approve or Deny — our own playable version of the experiment
• Scale X: the original statistics post — Wauters' primary write-up
• How we stop dangerous Claude Code commands
• Monitor versus block for AI agents

Study figures are drawn from Alex Wauters' analysis (scalex.dev) and explainx.ai's write-up, current as of publication. Beam records locally, can deny matching Claude Code and Codex tool calls under workspace enforce, and can pause or block high-risk browser sends via Sentinel. Missing cached policy is fail-open.