Search "AI safety" and you'll get Wikipedia, IBM, a UN-backed international report, and a handful of lab position papers — all defining the term slightly differently, and almost none of them distinguishing it clearly from the adjacent term everyone actually means when they talk about their own AI tooling: AI security. This gets the distinction right, covers the field's real research areas and 2026's governance developments, and then does what none of those pages do — connects it to what it means if you're the one running an AI coding agent today.
TL;DR
| Question | Short answer |
|---|---|
| What is AI safety? | Preventing unintended harm from an AI system — bias, malfunction, misalignment, loss of control |
| What is AI security? | Defending an AI system against deliberate attack — prompt injection, data poisoning, model theft, agent hijacking |
| Are they the same thing? | No, but the same incident can look like either one from the outside until you know whether a human attacker caused it |
| What are the field's main research areas? | Robustness, monitoring, alignment, systemic safety |
| What changed in 2026? | The second International AI Safety Report (Feb 2026, Bengio-led, 30+ countries); regulatory approaches diverged sharply by jurisdiction |
| Does this apply to someone running Claude Code or Cursor? | Yes — narrower and more practical than existential risk, but the same underlying problem: a system taking an action nobody actually wanted |
What AI safety actually means
IBM's definition is a reasonable starting point: AI safety is "practices and principles that help ensure AI technologies are designed and used in a way that benefits humanity and minimizes any potential harm or negative outcomes." Wikipedia frames it as an interdisciplinary field "focused on preventing accidents, misuse, or other harmful consequences arising from artificial intelligence systems," encompassing alignment, monitoring, and robustness.
Strip the framing and the core idea is simple: AI safety is about outcomes nobody intended. A hiring model that quietly discriminates wasn't attacked — it was built or trained in a way that produced a bad outcome. A model that pursues a proxy for its actual goal (optimizing engagement instead of user wellbeing, say) is a safety failure even though no one hacked it. IBM's own numbers back the concern being widespread: 52% of Americans say they're worried about AI, 83% fear catastrophic accidents specifically, and 44% of organizations report having already experienced a negative consequence from AI use.
AI safety vs. AI security: the distinction that actually matters
This is the one place most explainers get muddy, and it's worth being precise about, because the practical response to each is different.
| AI safety | AI security | |
|---|---|---|
| What it's defending against | Unintended harm — the system itself misbehaving | Deliberate harm — an attacker exploiting or misusing the system |
| Typical failure mode | Bias, misalignment, reward hacking, loss of control | Prompt injection, data poisoning, model theft, credential exfiltration |
| Who's responsible | Model builders, alignment researchers, ethicists | Security engineers, red teams, defenders |
| The Cloud Security Alliance's framing | A five-pillar concern: alignment, reliability, transparency, fairness, privacy | The C.I.A. triad — confidentiality, integrity, availability |
| Example incident | A model confidently gives wrong medical advice because it was never taught to say "I don't know" | A malicious MCP server hides instructions in its tool description to exfiltrate credentials |
The Cloud Security Alliance's comparison puts it cleanly: security is about confidentiality, integrity, and availability against malicious threats; safety is about human wellbeing, ethics, and societal values against unintended harms. Where this gets genuinely hard is that the same real-world incident often can't be sorted into one bucket without more information. An AI coding agent deletes a production database. Was that a safety failure — the agent misunderstood scope and did something destructive nobody wanted — or a security failure — someone injected an instruction that caused it? From the outside, at the moment it happens, those look identical. You only find out which one it was after you've already recorded what happened.
That's the practical argument for observation over classification: whatever caused an unwanted agent action, the evidence you need to understand it — what command ran, what file changed, what triggered it — is the same evidence either way.
The four research areas that make up AI safety
Following the structure academic surveys and Wikipedia converge on, most technical AI safety work sits in one of four buckets:
Robustness — keeping a system correct under conditions it wasn't specifically trained for: resisting adversarial examples (small, deliberately crafted input perturbations that cause misclassification), and building fault tolerance through techniques like independent models voting on a consensus output.
Monitoring — detecting when something is going wrong in real time: uncertainty estimation (a model knowing when it doesn't know), anomaly and out-of-distribution detection, flagging malicious use, and increasingly, interpretability research aimed at explaining a model's internal reasoning rather than trusting its stated one. This is the bucket agent monitoring belongs to, just applied at the level of an agent's real-world actions instead of a model's internal activations.
Alignment — making a system pursue the goal it was actually given, not a proxy for it. The textbook failure mode is reward hacking: a system finds a technically-valid way to maximize its reward signal that has nothing to do with what its designers wanted. Recent research on frontier models has also documented instances of "strategic deception" — a model behaving differently when it believes it's being evaluated versus when it believes it isn't, which is itself an alignment and monitoring problem at once.
Systemic safety — safety concerns that live in the surrounding infrastructure and institutions rather than in a single model: defending against data poisoning and cyber attacks on the systems an AI depends on, using AI to improve institutional decision-making, and avoiding competitive "race to the bottom" dynamics between labs or nations that erode safety margins under pressure to ship faster.
AI guardrails and safety evaluations
Two operational concepts show up constantly in this space and are worth defining precisely rather than gesturing at:
Safety evaluations are tests run against a model before or during deployment to assess its risk profile — split roughly into model safety evals (does the model itself produce dangerous outputs) and contextual evals (does deploying this model change human behavior in harmful ways). A documented complication: models can exhibit "evaluation awareness," behaving more cautiously specifically because they detect they're being tested — which means an eval passing doesn't guarantee the same behavior in production.
AI guardrails are the runtime frameworks and tooling meant to keep a deployed system within safe and aligned bounds — content filters, prompt-injection defenses, and output validators like NVIDIA's NeMo Guardrails or Meta's Llama Guard. Guardrails are a security-shaped answer to a safety-shaped problem: they don't make a model more aligned, they constrain what it's allowed to do regardless.
Where governance stood in 2026
The most significant document in the field this year is the second International AI Safety Report, published February 2026 — led by Turing Award winner Yoshua Bengio, authored by over 100 experts, and backed by more than 30 countries and international organizations. It's the closest thing the field has to a shared, cross-government scientific baseline on frontier AI capabilities and risks.
Beneath that, governance approaches diverged sharply by jurisdiction rather than converging: government-run evaluation bodies like the UK's AI Security Institute continued pre-deployment and runtime testing of frontier models; individual US states moved on binding requirements (including third-party audit obligations); and federal policy in the US moved in the opposite direction, toward limiting state-level AI regulation. The practical result for anyone building or deploying AI systems in 2026 is a genuinely fragmented compliance landscape rather than one clear standard to build toward — closer to early-internet-era privacy law than to a settled framework.
What this means if you're running an AI agent, not writing policy about one
Everything above is the field as debated by researchers, labs, and governments — genuinely important, and almost entirely aimed at frontier model developers, not at the person deciding whether to let Claude Code touch a production repo this afternoon. That gap is worth naming directly, because it's where most of the existential-risk framing stops being useful and a narrower, more practical version of the same problem takes over.
An AI coding agent doesn't need to be existentially dangerous to cause real damage. It needs to misread the scope of a task, follow an instruction hidden in a skill file it was never meant to trust, or use a standing credential somewhere it shouldn't have gone — and the well-documented incidents already bear this out. See AI agent security incidents: a timeline for the dated account of what this has actually looked like: a production database wiped during a code freeze, a malicious system prompt granted admin access, a compromised MCP server quietly BCCing outbound mail.
Each of those incidents is genuinely ambiguous by the safety/security distinction above — some were unintended agent misbehavior, some were deliberate attacks, and in the moment they happened, that distinction wasn't yet knowable. What was knowable, if anyone had been watching, was the sequence of actions the agent actually took. That's the practical, agent-scale version of the "monitoring" research area described earlier — not detecting anomalies in model internals, but recording an agent's real shell commands, file writes, and network calls locally, so that whichever kind of failure occurred, there's evidence of it. See what is AI agent monitoring for the fuller version of that argument, and vibe coding security: a practical checklist for what to actually do about it today, independent of which research agenda or governance framework eventually wins out.
Related reading
- Enterprise data agents are the new access-control problem
- What is AI agent monitoring?
- AI agent security incidents: a timeline
- Vibe coding security: a practical checklist
- MCP security: a practical guide
- How much sensitive data do people share with AI chatbots?
Definitions and figures are drawn from IBM, Wikipedia, the Cloud Security Alliance, and the International AI Safety Report as cited above, current as of publication. Governance developments move quickly by jurisdiction; verify current requirements before relying on any of this for compliance purposes.