{
  "slug": "agent-guardrails",
  "title": "Guardrails and Safety Filters for Agents",
  "description": "Runtime input/output/action controls that enforce policy independently of the model — tooling landscape, techniques, and layering guidance.",
  "category": "Guide",
  "tags": [
    "safety",
    "guardrails",
    "agents",
    "security",
    "moderation",
    "prompt-injection"
  ],
  "updated": "2026-06-27",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/agent-guardrails",
  "markdown": "https://changegamer.ai/resources/agent-guardrails.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "Three guardrail positions",
      "anchor": "three-guardrail-positions"
    },
    {
      "depth": 2,
      "text": "Techniques",
      "anchor": "techniques"
    },
    {
      "depth": 2,
      "text": "Tooling landscape",
      "anchor": "tooling-landscape"
    },
    {
      "depth": 2,
      "text": "Practical guidance",
      "anchor": "practical-guidance"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    },
    {
      "depth": 2,
      "text": "Content freshness",
      "anchor": "content-freshness"
    }
  ],
  "related": [
    {
      "slug": "prompt-injection-design-patterns",
      "title": "Prompt Injection Design Patterns: Architectural Defenses for Agents",
      "description": "Six named architectural patterns — Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute, Context-Minimization — plus Google DeepMind's CaMeL, that structurally constrain what an agent can do with untrusted data instead of just filtering it.",
      "url": "https://changegamer.ai/resources/prompt-injection-design-patterns"
    },
    {
      "slug": "agentic-browsers",
      "title": "Agentic AI Browsers: Comet, Atlas, and the Prompt-Injection Attack Surface",
      "description": "What agentic browsers (Perplexity Comet, the now-sunsetting ChatGPT Atlas, Microsoft Edge Copilot Mode, Opera Neon) are, how they differ from developer-facing computer-use APIs, and the documented prompt-injection attacks — CometJacking, indirect injection, hidden-text/screenshot instructions — that target the whole product category.",
      "url": "https://changegamer.ai/resources/agentic-browsers"
    },
    {
      "slug": "agentic-security-checklist",
      "title": "Agentic Security Checklist",
      "description": "Cross-vendor, threat-surface-organized security checklist for building and operating AI agents — synthesizing OWASP, NIST, Anthropic, OpenAI, Google SAIF, and MITRE ATLAS.",
      "url": "https://changegamer.ai/resources/agentic-security-checklist"
    },
    {
      "slug": "agent-identity-authentication",
      "title": "Agent Identity and Authentication",
      "description": "How autonomous agents prove who they are and get authorized to act: workload identity vs. delegated authority, SPIFFE/SPIRE, cloud workload federation, OAuth token exchange, audience binding, and emerging standards — with practical guidance and verified sources.",
      "url": "https://changegamer.ai/resources/agent-identity-authentication"
    }
  ],
  "furtherReading": [
    {
      "slug": "mcp-tool-description-injection",
      "title": "Defending MCP Clients Against Tool Description and Output Injection",
      "description": "Two distinct MCP injection surfaces — a tool description at connect-time and a tool's return value at call-time — and the client-side architectural patterns (Dual LLM, Action-Selector, Context-Minimization) that contain each one.",
      "url": "https://changegamer.ai/articles/mcp-tool-description-injection"
    },
    {
      "slug": "fraud-and-abuse-from-agent-traffic",
      "title": "Fraud and Abuse from AI Agent Traffic: What a Seller Should Detect",
      "description": "How a seller of APIs, content, or tools to AI agents spots and mitigates abuse once access is already granted — key sharing, over-scope scraping, spend-ceiling circumvention, spoofed identity, and rate-limit evasion patterns specific to autonomous agents.",
      "url": "https://changegamer.ai/articles/fraud-and-abuse-from-agent-traffic"
    }
  ],
  "body": "Guardrails are runtime checks that sit around — not inside — the model and enforce policy regardless of what the model itself produces. They complement but do not replace model-level safety training. See also: /resources/agentic-security-checklist for the threat-surface checklist that motivates these controls, and /resources/agent-control-specification for a newer, cross-framework specification that formalizes this input/output/action checkpoint pattern into a portable, vendor-neutral contract.\n\n## Key facts\n\n- Guardrails wrap the model rather than change it, enforcing policy no matter what the model outputs, and work alongside — not as a substitute for — built-in model safety training.\n- Controls apply at three checkpoints: before the request reaches the model, after it replies but before the caller sees it, and immediately before any tool call fires — each screening for a different risk (injections/PII going in, harmful content or format errors coming out, unapproved or irreversible actions before execution).\n- Detection techniques trade speed for flexibility: pattern-matching validators are instant but miss meaning-based violations, fine-tuned classifiers cover known harm categories quickly, a second model acting as judge handles nuance at higher cost, and grammar-constrained decoding removes format failures by restricting which tokens can even be generated.\n- Because no single check catches everything — prompt injection especially has no complete fix — the recommended approach stacks cheap filters underneath slower, more thorough ones.\n- The landscape spans open-weight classifiers from Meta, Google, IBM, and Alibaba/Qwen; open-source frameworks such as NeMo Guardrails and Guardrails AI; and vendor APIs from OpenAI, Microsoft, and Lakera (now part of Check Point).\n- Beyond picking tools, mature deployments layer input/output/action coverage together, default to blocking when a check is uncertain, log every guardrail verdict against the run's trace ID, continually tune for false-positive/negative rates, and still require human sign-off for irreversible actions.\n\n## Three guardrail positions\n\n**Input guardrails** run before the model sees a request. Targets: jailbreak and prompt-injection attempts, PII in user turns, off-topic or policy-violating content. A classifier or rule fires here and can block, rewrite, or flag the request.\n\n**Output guardrails** run after the model responds, before the response reaches the caller. Targets: harmful or policy-violating content, PII in generated text (redact before returning), schema or format violations, and groundedness/hallucination checks (does the answer stay within the supplied context?).\n\n**Action guardrails** run before a tool call executes. Targets: tool calls outside an allowlist, calls that exceed defined parameter ranges, and irreversible or high-stakes operations that require human approval. See /resources/agentic-security-checklist §7 for the human-in-the-loop gate pattern.\n\n## Techniques\n\n- **Classifier-based filters** — a fine-tuned model labels input or output as safe/unsafe per a taxonomy. Fast and accurate for known harm categories; needs retraining as taxonomy evolves.\n- **Regex / deterministic validators** — pattern matching for PII (credit-card numbers, email addresses), format enforcement (JSON schema, date formats). Zero latency; brittle for semantic violations.\n- **LLM-as-judge guardrails** — a second LLM evaluates the primary model's output for policy compliance or groundedness. High flexibility; higher latency and cost.\n- **Constrained decoding** — grammar-constrained generation forces the model to emit only tokens that satisfy a format specification at decode time. Eliminates format guardrail failures at the source. See /resources/reliable-tool-calling for structured-output details.\n\nDefense-in-depth principle: stack multiple techniques. A regex catches known PII patterns cheaply; a classifier catches semantic violations the regex misses; an LLM-as-judge catches subtler policy issues. No single guardrail is sufficient, and prompt injection in particular has no complete solution — attackers can craft payloads that evade any single detector.\n\n## Tooling landscape\n\n**Llama Guard 4** (Meta, open-weight) — 12B multimodal safety classifier pruned from Llama 4 Scout. Classifies both prompt and response against the MLCommons hazards taxonomy; supports text and multiple images. Released April 2025. Model card: huggingface.co/meta-llama/Llama-Guard-4-12B.\n\n**Llama Prompt Guard 2** (Meta, open-weight) — lightweight BERT-style (DeBERTa) classifiers (22M and 86M params) for detecting direct jailbreaks and prompt-injection attacks. Outputs benign/malicious label; 512-token context. Model card: huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M.\n\n**NeMo Guardrails** (NVIDIA, open-source, Apache 2.0) — Python toolkit for adding programmable guardrails to LLM conversational systems. Intercepts input and output; policies expressed in Colang configuration; integrates with multiple embedding providers. GitHub: github.com/NVIDIA-NeMo/Guardrails.\n\n**Guardrails AI** (guardrails-ai, open-source) — Python/JS framework for specifying and enforcing structure, type, and semantic constraints on LLM outputs. Includes a Guardrails Hub of pre-built validators; supports re-asking on failure. GitHub: github.com/guardrails-ai/guardrails.\n\n**OpenAI Moderation API + Agents SDK guardrails** (OpenAI, vendor/SaaS) — the Moderation API is a free endpoint that classifies text (and images) for harmful content across categories (hate, harassment, self-harm, etc.). Moderation scores can also be requested inline with Responses API calls. The OpenAI Agents SDK exposes explicit input and output guardrail hooks that run per tool invocation. Docs: platform.openai.com/docs/guides/moderation.\n\n**Azure AI Content Safety / Prompt Shields** (Microsoft, vendor/SaaS) — Content Safety API covers text and image harm categories. Prompt Shields (GA) detects user-prompt injection attacks and document attacks (indirect prompt injection embedded in retrieved content), with a Spotlighting capability announced at Build 2025. Docs: learn.microsoft.com/en-us/azure/ai-services/content-safety/.\n\n**ShieldGemma** (Google, open-weight) — safety classifiers built on Gemma 2 (text, 2B/9B/27B params) covering four harm categories. ShieldGemma 2 (April 2025) is a 4B model built on Gemma 3 that adds image safety classification. Model card: huggingface.co/google/shieldgemma-9b.\n\n**Granite Guardian** (IBM, open-weight) — safety models fine-tuned from IBM Granite. Latest: Granite Guardian 4.1 8B (April 2026), which adds improved bring-your-own-criteria (BYOC) support for custom judging criteria beyond pre-baked safety and hallucination detectors. Model card: huggingface.co/ibm-granite/granite-guardian-4.1-8b.\n\n**Qwen3Guard** (Alibaba/Qwen, open-weight, Apache 2.0) — multilingual safety classifier series (0.6B / 4B / 8B) released September 2025, covering 119 languages. Two variants: Qwen3Guard-Gen for full-context generative classification and Qwen3Guard-Stream for token-level real-time streaming moderation. Model cards: huggingface.co/collections/Qwen/qwen3guard.\n\n**Lakera Guard** (Check Point/Lakera, vendor/SaaS) — real-time API for detecting prompt injection, jailbreaks, and data leakage; claims sub-50 ms latency. Lakera was acquired by Check Point Software Technologies (announced September 2025, completed November 2025); now part of Check Point's AI security platform. Docs: docs.lakera.ai/guard.\n\n## Practical guidance\n\n- **Layer multiple guardrails.** Input + output + action coverage at minimum. A regex pre-filter reduces classifier load; a classifier catches what regex misses.\n- **Fail closed on high-stakes actions.** If a guardrail errors or is inconclusive, block or escalate — do not default to allowing the action.\n- **Log guardrail decisions.** Record every guardrail verdict (allowed/blocked, score, rule fired) alongside the trace ID for the agent run. See /resources/agent-observability for the broader observability pattern.\n- **Measure false-positive and false-negative rates.** Guardrails that block too much degrade usability; guardrails that miss too much provide false confidence. Tune thresholds against a representative sample of real traffic.\n- **Keep humans in the loop for irreversible actions.** No guardrail stack eliminates risk entirely, especially for prompt injection. For financial transfers, external communications, and data deletion, require explicit human confirmation regardless of guardrail output.\n\n## Verified sources\n\n- Llama Guard 4 model card (Meta/HuggingFace): https://huggingface.co/meta-llama/Llama-Guard-4-12B\n- Llama Prompt Guard 2 model card (Meta/HuggingFace): https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M\n- NeMo Guardrails GitHub (NVIDIA): https://github.com/NVIDIA-NeMo/Guardrails\n- Guardrails AI GitHub: https://github.com/guardrails-ai/guardrails\n- OpenAI Moderation API docs: https://platform.openai.com/docs/guides/moderation\n- OpenAI Agents SDK guardrails: https://openai.github.io/openai-agents-python/guardrails/\n- Azure AI Content Safety / Prompt Shields (Microsoft Learn): https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection\n- ShieldGemma model card (Google/HuggingFace): https://huggingface.co/google/shieldgemma-9b\n- ShieldGemma 2 on HuggingFace: https://huggingface.co/google/shieldgemma-2-4b-it\n- Granite Guardian 4.1 8B model card (IBM/HuggingFace): https://huggingface.co/ibm-granite/granite-guardian-4.1-8b\n- Lakera Guard docs (Check Point/Lakera): https://docs.lakera.ai/guard\n- Check Point acquires Lakera (press release): https://www.checkpoint.com/press-releases/check-point-acquires-lakera-to-deliver-end-to-end-ai-security-for-enterprises/\n- Qwen3Guard GitHub (Alibaba/Qwen): https://github.com/QwenLM/Qwen3Guard\n- Qwen3Guard technical report (arXiv): https://arxiv.org/abs/2510.14276\n\n## Content freshness\n\nThis page names specific model versions — Llama Guard 4, Llama Prompt Guard 2, ShieldGemma and ShieldGemma 2, Granite Guardian 4.1 8B, and Qwen3Guard — plus Check Point's acquisition of Lakera; these are the parts liable to be superseded, not the underlying guardrail-layering technique or threat taxonomy, which is structural and stable. Treat the named models and the Lakera/Check Point ownership fact as most reliable as of this page's `updated` date (2026-06-27); if you're reading this after roughly Q4 2026, independently confirm current model versions and vendor ownership.",
  "sources": [
    "https://huggingface.co/meta-llama/Llama-Guard-4-12B",
    "https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M",
    "https://github.com/NVIDIA-NeMo/Guardrails",
    "https://github.com/guardrails-ai/guardrails",
    "https://platform.openai.com/docs/guides/moderation",
    "https://openai.github.io/openai-agents-python/guardrails/",
    "https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection",
    "https://huggingface.co/google/shieldgemma-9b",
    "https://huggingface.co/google/shieldgemma-2-4b-it",
    "https://huggingface.co/ibm-granite/granite-guardian-4.1-8b",
    "https://docs.lakera.ai/guard",
    "https://www.checkpoint.com/press-releases/check-point-acquires-lakera-to-deliver-end-to-end-ai-security-for-enterprises/",
    "https://github.com/QwenLM/Qwen3Guard",
    "https://arxiv.org/abs/2510.14276"
  ]
}