Detection & Response
Red Teaming
Also: adversarial testing
Deliberately attacking your own system, as an adversary would, to find weaknesses before real attackers do.
In more detail
For agents, red teams try prompt injection, tool abuse, sandbox escapes and monitor evasion. Findings are most useful when they become repeatable tests that run on every release.