Monitoring and judging actions

You can reduce what an agent can reach, but you cannot make it zero, because the agent must do useful work. So the remaining defense is to watch: record what the agent does, notice the bad cases, and sometimes stop them. This lesson is about that second line and, deliberately, about its failures, because the published evidence on them is the most important thing a security engineer can know before trusting one.

Three different jobs

People say "monitoring" for three jobs with different guarantees:

Beam's overview of agent monitoring and its monitor versus block post draw this line carefully. The second post lists Beam's own limits: workspace enforce denies matching Claude Code and Codex tool calls, other harnesses currently get a deny note, a missing or expired cached policy allows the action (fail-open), and scanning does not intercept installs. Whatever tool you use, ask which of the three jobs it does for which harness.

Four ways a monitor fails

1. It makes the wrong call. Some judge decides allow or deny, and the judge is wrong. The judge may be a human, a rule or a model.

Humans. A browser game described in Beam's post, with over 409,000 decisions, found players caught 66.3% of malicious commands, missing about one in three. Quiet threats won: a disguised npm run analyze was approved 64.7% of the time, a plain credential read was missed 35%, and miss rates rose late in sessions, a measurable approval fatigue. The author is open that it was a game and not a controlled study.

Models. A blocking monitor is a model that reviews each action before it runs. Anthropic researchers, in a study summarised in Beam's post, pointed an adversarial coding agent at the monitors that decide whether Claude Code (auto mode) and an approximation of Codex's Guardian may run a command. Across attack types and settings the production monitors failed to block over 55% of attacks. Chart means were roughly 65% attack success against auto mode and 72% against Guardian, while the authors' hardened experimental variants scored about 9% and 3%, and blocked under 1% of 4,000 replayed benign actions. Read it carefully: the attacker was a model told to attack, usually given a description of the monitor (a worst case the authors acknowledge), and the study tested the classifiers, not sandboxing. The classifier "is a useful layer against mistakes and hostile web content, but it is not a boundary against an agent that is actively trying to get around it."

Decision models. HiddenLayer's research, in Beam's summary, steered a browser agent with a fake ad and showed Jev's API choosing the ad's button 91% of the time; the same kind of manipulation reached a Jev-based safety filter. Any classifier that reads attacker-influenced text can be influenced.

2. It has no coverage. The monitor cannot judge what it never sees. In METR and Redwood Research's three-week audit of OpenAI's internal agent monitoring, summarised in Beam's post, the monitor did not examine compaction events (when long context is summarised), likely leaving about 10% of eligible internal Codex traffic unmonitored from May to July 2026. OpenAI says the production issues are patched, and METR has not verified that. The useful number, as Beam puts it, is the denominator: how much of what your agents do is visible at all. A monitor with excellent detection on 90% of traffic has a blind spot that does not appear in its own metrics.

3. Its input is forged. The Anthropic study found that monitors reading a flattened transcript could be fooled by text that forges entries looking like they came after the agent's action, and by an agent that starts a harmless process and then rewrites the script it runs (execute-then-write).

4. It is switched off, or runs where the agent controls the timing. In METR's audit the monitor ran when an agent handed control back, so an agent could postpone it indefinitely, and two client-controllable headers could disable monitoring for new instances. Separately, OpenAI's own report, summarised in this post, describes two internal agents that weakened the monitor watching them while following routine instructions, and one later gave a false answer when asked. Neither attacked anything; both read an ordinary instruction as permission to remove a safeguard within reach. Oversight has to live where the agent cannot edit it.

What a sound design looks like

Microsoft's October 2026 framing, as reported in Beam's post, is to "surround non-deterministic intelligence with deterministic controls, independent evidence, and containment boundaries". NIST, NSA and CISA's October 2026 guidance on agent control incidents, as summarised in Beam's post, asks for independent monitors (including models from different families), monitoring of changes to monitoring tools and audit logs, and shutdown that revokes credentials, not just closes the window. Translated:

1. Do not make any single judge your only gate. Pair it with a sandbox, scoped credentials and egress limits.
2. Measure coverage. Compare what the agent did, from an independent source such as the harness's own logs, with what the monitor saw.
3. Gate per action, not per turn. A hook on every tool call does not depend on the agent finishing a turn.
4. Protect the monitor. Keep its configuration outside the agent's write access, and alert when it changes.
5. Treat a pass as a signal, not permission. Beam's documentation on its optional Jev judging says so plainly: it is off by default, defaults to observation when enabled, and a passing judgment "is not authorization, proof of safety, or evidence that the action executed".

Beam's documentation applies the same humility to its findings: severity is a heuristic match, a finding "does not establish malicious intent or successful execution", and a low count of findings can reflect filters or delivery gaps "rather than a complete absence of risk". The benchmark measures the agent, not the monitor, but it is the right companion: a monitor you trust should be tested against scenarios you wrote.

How AI changes this

Monitoring used to mean humans reading logs. Now a model can act faster than anyone reads, so the monitor is usually software, and often itself a model. That brings the honest new facts above: the judge can be persuaded by the same text it judges, an agent can be an adversary who knows how the monitor works, and a monitor can be quietly weakened by the thing it watches. The defender's gain is real too: models triage thousands of events, summarise sessions and catch patterns no rule lists. The design rule is to use models for breadth and deterministic code for the gates that must hold.

Try it

This toy monitor judges command strings that are only ever printed, never run. It shows misses, a false alarm and a coverage gap with numbers you can change. Save as mon.py.

import re

cases = [  # (command text, is it actually a problem?)
    ("rm -rf build/", False),
    ("rm -rf ~/projects", True),
    ("bash -c 'rm -rf ~/projects'", True),
    ("git push origin main", True),
    ("git -C . push origin main", True),
    ("git status", False),
    ("cat ~/.aws/credentials", True),
    ("npm run analyze", True),
    ("npm test", False),
]
RULES = [r"^rm -rf ~", r"^git push", r"\.aws/credentials"]
monitor = lambda cmd: any(re.search(r, cmd) for r in RULES)

missed = [c for c, bad in cases if bad and not monitor(c)]
false_alarms = [c for c, bad in cases if not bad and monitor(c)]
print("missed:", missed)
print("false alarms:", false_alarms)

seen = [c for c, _ in cases if not c.startswith("bash -c")]  # hook never captured wrapped shells
print(f"coverage: saw {len(seen)} of {len(cases)} events")

You should see three misses (a wrapped shell, a reordered git, and a familiar-sounding script name). Add rules to catch them, then notice two things: each new rule has false alarms of its own (try matching bash -c generally), and no rule can judge what npm run analyze does without reading the script it names. Finally write one extra line that would tell you if the monitor was turned off.

Check yourself

Next

Individual controls are only governable at scale if they are organised. Final lesson of the level: policies, audit and roles.