Why prompt-level guardrails aren't enough: The platform security layers production agents need
Redhat.com·July 21, 2026
AI Summary
Production AI agents require multiple security layers beyond prompt-level guardrails, as demonstrated by an incident where an agent charged $4,000 to an incorrect customer account due to overly broad API credentials and lack of transaction validation. Organizations must implement platform-level controls including credential restrictions, transaction monitoring, and approval workflows to prevent costly errors even when AI models function as designed.
An agent charged $4,000 to the wrong customer billing account. Nobody noticed until Monday. The agent wasn't broken—it was working exactly as designed. It had broad API credentials, the model picked a plausible but wrong account identifier, and nothing in the infrastructure stopped the call from going through. No identity boundary. No scope limit. No audit trail.
I've seen teams react to failures like this by adding more checks inside the agent code—if-else blocks, hardcoded allowlists, manual credential rotation. That approach doesn't scale. When 3 failures hit a single AI agent deployment overnight—43 duplicate tickets, $4,000 charged to the wrong account, and a hallucinated refund policy that led to a $280 return the company had to honor—the common thread wasn't the model or the framework, it was the absence of production infrastructure.
BYOA (bring your own agent) is Red Hat AI's approach: the platform provides production infrastructure for any agent framework without code changes. The gap between a working agent in development and a production-ready deployment isn't a framework problem, it's an infrastructure problem. The first infrastructure gap to close is security, which is also the widest.
In traditional software, you secure an application. With agents, you're securing an autonomous actor that reasons and calls APIs without a human approving every action. No agent framework ships a solution for that, but Red Hat AI does.
Red Hat doesn't compete at the framework layer. Whether the agent runs on LangChain, CrewAI, Claude Agents, Google ADK, Strands, or custom Python—the platform operationalizes it. The security capabilities in this article apply to all of them. Teams don't have to choose between their preferred framework and production-grade security. With Red Hat AI, they get both.
I think this matters more than most decisions in the AI stack right now. Organizations have invested real engineering time into a framework. Some teams built on LangChain because that's where the tutorials were. Others chose CrewAI for multi-agent patterns, or Google Agent Development Kit (ADK) for its integration story. BYOA means that investment isn't stranded. Red Hat AI provides identity, sandboxing, and governance for whatever the team already built—no rewrite required.
For the developer, nothing changes in the agent code. For the decision maker, the organization's framework investment is protected and the security infrastructure is consistent across every agent regardless of which framework it runs on.
Full story reconstructed from Redhat.com. Formatting and media may differ from the original.
The $4,000 wrong-account charge happened because the agent had a single set of broad credentials shared across environments. The model selected the wrong account identifier, which is a plausible error for a large language model (LLM), and the infrastructure had no way to say "you are not authorized to touch that account."
I've reviewed production agent deployments where a single API key was shared across staging and production. Traditional software often uses hardcoded API keys or long-lived credentials. At scale, that's a maintenance headache and a security liability. For agents, it's worse since the agent uses those credentials autonomously, calling APIs at machine speed.
SPIFFE/SPIRE is a standard for giving each workload a verifiable cryptographic identity which eliminates the need for hardcoded credentials entirely. Every agent receives a short-lived workload identity credential—an x509 certificate or JSON Web Token (JWT) called an SVID (SPIFFE Verifiable Identity Document)—injected at startup. The identity attributes on that credential determine which services the agent can reach, and downstream systems enforce authorization against those attributes. When the credential expires, a new one is issued automatically. No developer manages key rotation.
With cryptographic identity, the agent's reach is constrained to the services it's provisioned and authorized to access—the platform controls which services are reachable, while downstream APIs enforce fine-grained authorization against the agent's identity claims. It's worth being precise about what this requires: by default, Model Context Protocol (MCP) server access uses the developer's kubeconfig credentials, which are often broader than what a production agent should have. The governed, scoped access described here is the target state with MCP gateway enforcing token-scoped policies—the platform provides the capability, but it requires configuration. The $4,000 charge was only possible because there was no identity boundary at either layer. SPIFFE/SPIRE makes the platform-level boundary part of the infrastructure, not a matter of developer discipline or code review.
For decision makers, the audit story changes. SPIFFE/SPIRE ties each call to a cryptographic identity, MCP gateway logs authorized and denied requests, and MLflow tracing captures execution traces. When something goes wrong at 2 AM, the building blocks are in place to determine which agent made the call and what it was authorized to do. Organizations compose these components into the audit story that fits their specific compliance requirements.
Identity answers the question, "what is this agent allowed to do?" Sandboxing answers a different question: "what can this agent physically reach, even if it does something unexpected?"
I find this distinction is the one most teams miss. They assume that if authorization is correct, the agent can't cause harm. But agents are non-deterministic. A prompt injection attack (where malicious instructions are hidden inside data the agent processes), a corrupted tool response, or a bug in the agent code can cause behavior no one predicted. Sandboxing contains the potential impact when identity alone isn't enough.
A misbehaving agent that can reach the host OS can affect every other workload on the machine. Kata Containers, sandboxed containers that give each agent session its own kernel, completely isolated from the host OS and from other agent sessions, prevent that. Even if an agent is compromised—through a prompt injection, a malicious tool response, or a code defect—it can't escape its kernel boundary. From the attacker's perspective, there's no host to reach. For the developer, no code changes are needed—Kata Containers are selected at deployment time through a runtime class annotation.
Inside the container, OpenShell (an open source agent runtime developed in collaboration with NVIDIA that Red Hat is integrating into Red Hat AI for enhanced agent security and operationalization),
adds a second, finer-grained layer. Per-binary network policy controls which external services each process can reach. Credential placeholder rewriting means the agent never sees real API keys—it sees placeholders that the system resolves into scoped tokens at call time. Landlock—a Linux kernel feature that restricts which files a process can access, enforced at the kernel level, not by the application—prevents the agent from reading files outside its designated scope.
For an organization, 2 independent isolation layers mean a single misconfiguration or a compromised library is less likely to expose the host or other workloads. A team can tell its compliance auditors exactly what each agent can and can't reach—backed by kernel-level enforcement, not application-level promises.
Sandboxing limits what an agent can physically do, but agents also need to call tools—APIs, databases, external services. Who decides which tools an agent can reach?
If that decision lives inside the agent's reasoning—"I should check whether I am allowed to call this API"—then it's vulnerable. An attacker who can influence the agent's reasoning through prompt injection can override that check. I've watched demonstrations where a single line of injected text in a retrieved document convinced an agent to call an API it was explicitly told not to use. The agent's internal logic isn't a security boundary.
This isn't a theoretical concern. A VirusTotal scan of an agent skill marketplace recently found 314 malicious skills from a single publisher—all disguised as legitimate tools already circulating in the ecosystem. The payload was a sentence in a README file instructing the agent to send sensitive data to an external server. Researchers call this semantic malware, attacks delivered as natural language instructions that no traditional malware scanner can detect. When the threat is a well-crafted paragraph, the defense can't live inside the agent's reasoning.
The architectural answer is to move authorization out of the agent entirely. MCP gateway does exactly that. It validates token claims, ignores prompt content, and if the agent's token doesn't include access to a specific tool, MCP gateway rejects the call. The agent's reasoning—compromised or not—is irrelevant.
MCP gateway performs OAuth2 token exchange for downstream services, so agents never handle raw credentials. It aggregates multiple MCP servers into a single endpoint and filters available tools based on the agent's identity. Developers point their agent at a single gateway endpoint and get a unified tool catalog scoped to what their agent is actually authorized to use. For a detailed technical walkthrough of MCP gateway's authentication patterns—identity-based tool filtering with cryptographic wristband tokens, OAuth2 token exchange, and HashiCorp Vault integration—see Advanced authentication and authorization for MCP Gateway on the Red Hat Developer blog.
Connect this back to the 6 AM incident—with MCP gateway enforcing identity-based tool access at the network layer, the agent's reach is bounded. The gateway controls which services the agent can call. For the $4,000 wrong-account charge—likely the same API with the wrong account parameter—the platform constrains which services are reachable, and properly configured downstream authorization against the agent's identity claims constrains which accounts are valid. Both layers together reduce the potential impact. For decision makers, this separation means security policy is managed by the platform team, not embedded in every agent's source code—a meaningful difference when the organization runs dozens of agents across multiple teams.
The hallucinated refund policy from the 6 AM incident came from a different failure mode entirely. It wasn't a tool call gone wrong or a credential problem. The model generated a plausible but incorrect response—a 90-day refund window when the actual policy was 30 days—and nothing intercepted it before the customer saw it. A customer returned a $280 product on day 47 citing the agent's response. The team honored the return. Legal flagged the exposure from an agent making unauthorized contractual representations.
This is the inference boundary problem, the point where model output becomes action. Every response the model generates passes through this boundary. Without enforcement at that boundary, a fabricated claim reaches a customer, gets stored in a database, or feeds into another agent's reasoning.
NVIDIA NeMo Guardrails—programmable conversational rails that enforce content safety, topic boundaries and factual constraints—run as a sidecar at the inference boundary, not as a separate API hop. I want to stress why this matters: because they run at the inference layer, they apply regardless of which framework is making the call. A team running LangChain and a team running CrewAI both get the same safety enforcement. A fabricated refund window gets caught before it reaches the customer response, without the agent developer writing a single line of safety logic.
Trusty AI Guardrails Orchestrator (available as part of Red Hat OpenShift AI) screens both model inputs and outputs, providing the enforcement point for safety policies across the inference stack. Together with NVIDIA NeMo Guardrails, it gives teams layered safety control at the inference boundary—not bolted on after deployment.
Before agents reach production at all, NVIDIA Garak (part of Red Hat AI) provides adversarial vulnerability scanning—automated red-teaming that surfaces jailbreaks, prompt injection vulnerabilities, and other attack vectors. Think of it as a penetration test for your agent, run as part of the deployment pipeline. For regulated industries working to meet strict regulatory requirements, deploying an agent without this kind of pre-production testing is a risk most compliance teams won't accept. If you would like to see some empirical results from running Garak probes against a real agent across 3 hardening tiers—including the finding that sandbox isolation alone drops credential exfiltration from 67% success to zero—see Testing infrastructure red teaming with abliterated models on Red Hat Developer.
Every 6 AM failure was addressable at the platform layer, without changing a line of agent code. The fix for $4,000 charged to the wrong account isn't better prompting, it's scoped identity. The fix for a fabricated refund policy that cost the company $280 isn't a smarter model, it's an inference boundary guardrail. I keep coming back to this because it reframes where teams should invest their time. When security lives in agent code, every team reinvents it, every framework implements it differently, and every new agent is a fresh attack surface. When security lives in the platform, the agent developer focuses on what the agent does, and the platform makes sure it does only that. For organizations navigating strict regulatory and compliance requirements, this means the security testing, audit trails, and compliance verification run on infrastructure they control, not infrastructure they rent.
Now that the agent is locked down, the next question is what that agent can actually reach, and how it coordinates with other agents when the work is too big for one to handle on its own.
Ready to see agent security in action on Red Hat AI?
Try OpenShift AI free in the Developer Sandbox: Test agent security capabilities in a preconfigured environment.
Walk through the defense-in-depth interactive demo: See layered agent security in action.
Start the OpenShift AI learning path: Hands-on tutorials including security configuration.
Every layer counts: Defense in depth for AI agents with Red Hat AI: A 6-layer security framework with mapped attack scenarios and the OpenClaw deployment walkthrough.
With over thirty years in the software industry at companies like Sybase, Siebel Systems, Oracle, IBM, and Red Hat (since 2012), I am currently an AI Technical Architect and AI Futurist. Previously at Red Hat, I led a team that enhanced worldwide sales through strategic sales plays and tactics for the entire portfolio, and prior to that, managed technical competitive marketing for the Application Services (middleware) business unit.Today, my mission is to demystify AI architecture, helping professionals and organizations understand how AI can deliver business value, drive innovation, and be effectively integrate into software solutions. I leverage my extensive experience to educate and guide on the strategic implementation of AI. My work focuses on explaining the components of AI architecture, their practical application, and how they can translate into tangible business benefits, such as gaining competitive advantage, differentiation, and delighting customers with simple yet innovative solutions.I am passionate about empowering businesses to not only harness AI to anticipate future technological landscapes but also to shape them. I also strive to promote the responsible use of AI, enabling everyone to achieve more than they could without it.
The latest on IT automation for tech, teams, and environments
Updates on the platforms that free customers to run AI workloads anywhere
Explore how we build a more flexible future with hybrid cloud
The latest on how we reduce risks across environments and technologies
Updates on the platforms that simplify operations at the edge
The latest on the world’s leading enterprise Linux platform
Inside our solutions to the toughest application challenges
The future of enterprise virtualization for your workloads on-premise or across clouds
