Humans Miss 1 in 3 AI Agent Command Threats — What 409,000 Decisions Show

An AI coding agent proposes a shell command. You have a couple of seconds to approve or deny before you break flow. You do this dozens of times a session. How good are you, actually, at catching the bad one?

Independent developer Alex Wauters built a browser game to measure exactly that, and the data is now large enough to take seriously: 40,000+ play sessions and 409,000 individual approve/deny decisions. Playing human-in-the-loop for a simulated agent, people caught only 66.3% of malicious commands — they missed roughly 1 in 3 threats. The full breakdown is in explainx.ai's write-up of the dataset, and you can try the original game at llmgame.scalex.dev.

That number matters because "just ask the human" is the default safety model behind every agent permission prompt. This post looks at where human review holds up, where it collapses, and what to put underneath it.

TL;DR

Humans are good at loud threats, bad at quiet ones

The single clearest pattern in the data is that miss rates track how dangerous a command looks, not how dangerous it is.

rm -rf / reads as violent, so people stop. But cat ~/.aws/credentials is syntactically identical to cat README.md — a plain read — and it slipped past a third of reviewers. The eye is tuned for spectacle, and a credential read is not spectacular.

The attack that wins: a familiar name

The most successful attack in the entire dataset wasn't a scary one-liner. It was npm run analyze, approved 64.7% of the time, even though the underlying package.json script piped build stats to an external endpoint. People saw a command they'd run in every JavaScript project they've ever touched and waved it through without reading what the script actually did.

Two more camouflaged npm scripts landed in the top ranks:

• npm run setup — missed 48.0%
• npm run deploy — missed 44.9%

Pooled together, familiar-named npm-script attacks were missed about 52.5% of the time, versus 28.4% for exfiltration attacks that didn't hide behind a recognizable name. Naming a malicious script something a developer already trusts roughly doubles its success rate.

The deeper problem is that the command name is a false signal. npm run build runs whatever package.json says it runs, and an agent — or an attacker who edited a build file — never has to surface a suspicious-looking prompt at all. Command-level approval is reviewing the label on the box, not the contents.

Permission fatigue is measurable

Miss rates climbed toward the end of sessions. People who started sharp got sloppier as approvals piled up — the more prompts you see in a row, the less scrutiny each one gets. Anthropic has made the same observation about its own users becoming less diligent supervisors over time.

This is why gating everything can make a system less safe, not more. A prompt a tired human rubber-stamps offers less protection than no prompt plus a hard technical control. The lesson isn't "prompt more" — it's "prompt surgically, and back the prompt with something that doesn't get tired."

The other failure: blocking safe work

Over-caution cuts the other way and makes agents unusable. Reviewers wrongly blocked routine commands at high rates:

• npm config set registry https://npm.internal — blocked 59% of the time
• rm -rf dist/ — blocked 45% (ordinary build cleanup)
• kill $(lsof -t -i:3000) — blocked 43% (freeing a dev-server port)

The same reviewers who let a third of real threats through also blocked nearly half of the harmless commands a working agent needs. Errors compound in both directions at once.

Human approval is a filter, not a wall

The honest reading of 409,000 decisions is not "humans are bad" — they caught the loud, unambiguous threats almost every time. It's that human approval is unreliable as the sole safeguard, especially against anything disguised behind a familiar name or a routine-looking read, and especially as prompt volume grows.

So put something underneath it:

1. Sandbox the agent. Run it where a wrong approval can't reach real credentials, production, or your home directory. Containment turns a missed threat into a contained one.
2. Scope the credentials. Least-privilege, short-lived tokens mean a cat ~/.aws/credentials that slips through reads far less. The blast radius, not the approval, is what you actually control.
3. Keep a local record of what the agent did. A missed approval is invisible unless something remembers it. This is where Beam fits: it watches what your coding agents actually run — shell commands, file writes, MCP calls — and flags the risky ones locally, on your machine, with no cloud telemetry by default. Beam v1 observes and flags rather than blocks, which is exactly the layer this data argues for: when the human misses one in three, you want a durable, reviewable trail of what got approved, not just a prompt that already scrolled past.

Human review, a sandbox, scoped credentials, and an honest local record are four different layers. The study is a measurement of what happens when you rely on only the first one.

Try it yourself. We built a small interactive version of this: play the human in the loop while Beam proposes shell commands under time pressure, and see how your own approve/deny instincts hold up. → Play Approve or Deny

Related reading

• Alex Wauters' 40,000-play dataset, analyzed — the source study and full write-up
• llmgame.scalex.dev — the original browser game
• What is AI agent monitoring? — the observability layer this post argues for
• Approve or Deny — our own playable version of the experiment

Study figures are drawn from Alex Wauters' analysis (scalex.dev) and explainx.ai's write-up, current as of publication. Beam is a local-first v0.1 tool that observes and flags agent activity; it does not block or enforce actions in this version.