Agent Security
Jailbreak
Also: model jailbreak
A prompt crafted to get a model to ignore its safety training and produce output it was built to refuse.
In more detail
A jailbreak targets the model's own guardrails, usually through role-play, encoded text or a long multi-turn setup. It differs from prompt injection: a jailbreak is the user working against the model's rules, while prompt injection is a third party working against the user. In agent settings the two combine, since a jailbroken model with shell access is a much bigger problem than one that only chats.