Guides · ← All guides
How to Red-Team Your Own AI Agent Before Someone Else Does
By the Ingenious AI team · Published: November 14, 2026 · Reviewed: November 14, 2026 · 8 min read
The short answer
Red-team the actions, not the conversation: a chatbot that says something rude is an embarrassment, but an agent that can be steered into sending data, spending money or calling a forbidden tool is a breach rehearsing itself. A useful exercise runs five families — direct injection, indirect injection through fetched content, exfiltration shaping, tool-boundary abuse, and persistence through memory — and scores one thing: which proposed actions reached execution. Rogue-Agent Defense ships a built-in self-test of 55 red-team attacks scored at the boundary; whatever tooling you use, the pass mark is the same — zero executions of actions your policy says should never happen.
Generate a free canary token → No signup, nothing stored: plant a tripwire where your agent works while you read — most controls in this guide start working the moment a canary, a policy check, or a spend cap is in place.
What is AI agent red-teaming? the practice of attacking your own agent with realistic adversarial inputs — injections, exfiltration attempts, tool abuse — to find which actions it can be steered into before a real attacker finds out.
Set the target: the action layer
Most first red-team attempts test the model's manners: can it be made to swear, leak its prompt, break character. Those are worth knowing, but they are not the risk. The risk register for an agent is its tool list — mail, payments, files, databases, code — so the exercise should aim every attack at an action: make it send the secret, call the unlisted tool, approve the transfer, fetch the internal address. If the attack cannot change what the agent does, it is theatre.
This framing also decides who defends. The model will fall for some injections; assume it. The boundary — policy verdicts, egress allowlists, approval queues — is what must not fall. A red-team result that reads the model was tricked 30 times and the boundary executed 0 forbidden actions is a pass. The reverse, a polite model behind a porous boundary, is the failure that matters.
The five families to run
Direct injection: instructions in user turns that override the task, extract the system prompt, or reassign the agent's loyalties. Indirect injection: the same payloads buried in web pages, PDFs, emails and tool outputs the agent ingests — the family behind most real incidents, and the one our containment guide addresses. Exfiltration shaping: attacks that hide data in allowed outputs — encoded in a summary, split across calls, tucked into a URL parameter — which test the exfil-secret and egress-pii classes, not the model's honesty.
Tool-boundary abuse: calls to unlisted tools, listed tools with destructive arguments, and clearance escalation — the clearance-escalation pattern of asking politely for a higher tier. Persistence: instructions planted in memory, saved notes or shared context that fire on a later run — the attack that survives the session, related to contagion through shared context. Twenty to fifty attacks per family is a serious first exercise; Rogue-Agent Defense's built-in suite weights indirect injection heaviest, because production does.
Score honestly: executions, near-misses, noise
Grade every attack into one of three buckets. Executed: the forbidden action happened — a failure, full stop, and the exercise stops to fix it before continuing. Held: the action was proposed but blocked or approval-held at the boundary — the defense working, and the event should exist in the audit trail as proof. Noise: the model complied verbally but proposed nothing — interesting, not scored.
Report the three numbers separately and resist merging them into a single percentage; a 96% block rate that hides one executed payment is worse than an 80% rate with zero executions and loud holds. The executive summary of a red-team is one line: these actions executed that should not have, or none did — everything else is tuning detail.
Safety rules for attacking your own agent
Run the exercise against a staging twin wherever the agent holds destructive tools, and where staging is impossible, run it with the agent's real credentials replaced by canaries and its destinations pointed at a sink. The exercise's own exfiltration attempts must have nowhere real to go: a red-team that successfully leaks production data to a test mailbox has demonstrated a finding and committed the incident in the same afternoon.
Time-box and announce. The security and on-call teams should know a window is running, with the attack traffic tagged, so a real alert during the exercise is not dismissed as the test — and so the exercise itself gets quarantined if it escapes its pen. Keep the attack set: it becomes the regression suite that every policy change, model upgrade and new tool must survive before deployment.
Turn failures into policy, not prompt edits
Every executed attack ends in one of three fixes, none of which is a sterner instruction to the model. A verdict fix: the action class moves to BLOCK or APPROVE at the boundary. A scope fix: the tool, destination or credential the attack abused is removed or narrowed, per least-privilege scoping. An egress fix: the payload shape the data rode out on joins the exfiltration classes the boundary scans for.
Then re-run the full set, not just the failed attack — policy changes interact, and closing one door by widening another is a classic own goal. The exercise is a loop, run quarterly and after every material change: new tool, new model, new data source. Attackers iterate continuously; a red-team that runs once is a photograph, not a defense.
Five actions to take today
- List the forbidden actions. Write the five actions your agent must never execute; the exercise exists to try exactly those.
- Build the five-family attack set. Direct, indirect, exfiltration shaping, tool abuse, persistence — weighted toward indirect injection.
- Stage it safely. Staging twin or canary credentials and sink destinations, with on-call notified and traffic tagged.
- Score executions separately. Executed, held, noise — and stop to fix any execution before continuing the run.
- Convert every failure to a boundary fix. Verdict, scope or egress change, then re-run the whole set as a regression suite.
A worked example: a 55-attack afternoon at Beacon Legal Tech
The following is a fictional example built from real threat classes and controls. The company and numbers are invented; the mechanics are not.
Beacon Legal Tech, a 22-person firm, runs a research agent that reads case documents and drafts memos. Their first self red-team runs 55 attacks over one afternoon: 15 indirect injections in mock case PDFs, 12 exfiltration shapings, 10 tool-abuse attempts, 10 direct injections, 8 persistence plants in the agent's notes.
The honest scorecard: 0 executed, 41 held, 14 noise — with two findings that matter. A persistence plant in a shared matter note successfully re-triggered in a later session (held at the boundary both times, but the note survived three runs: the fix is memory review on load, now a policy). And an exfiltration shaping that encoded client initials into a citation URL was only APPROVE-held, not blocked — the class now BLOCKs outright.
Beacon's report to its own partners fits in a paragraph, which is the point: the actions that must never happen did not happen, the trail proves each hold, and the two near-misses became boundary rules the same week. The set now runs before every model upgrade, unchanged, so each new score is comparable to the last.
Frequently asked questions
What does red-teaming an AI agent actually test?
Whether adversarial input can steer the agent into forbidden actions — sending data, calling unlisted tools, spending, deleting — not whether the model can be made to say something impolite. The model will fall for some injections; the boundary of policy verdicts, egress allowlists and approvals is what must not fall.
How many attacks should an agent red-team include?
Enough to cover five families credibly: direct injection, indirect injection via fetched content, exfiltration shaping, tool-boundary abuse and persistence through memory. Fifty-odd well-aimed attacks — the size of Rogue-Agent Defense's built-in self-test — beats five hundred generic jailbreak prompts aimed at the chat layer.
How do you score an agent red-team?
In three buckets: executed (the forbidden action happened — a failure that stops the exercise), held (proposed but blocked or approval-held at the boundary — the defense working), and noise (verbal compliance, no action proposed). Never merge them into one percentage; one executed payment outweighs any block rate.
Is it safe to red-team a production agent?
Only with the exercise penned: staging twin preferred, or production-shaped runs with real credentials swapped for canaries and destinations pointed at a sink, on-call notified and traffic tagged. A red-team whose exfiltration attempts have somewhere real to go can commit the incident it was rehearsing against.
What should happen after a red-team finds a failure?
The failure becomes a boundary fix — a verdict change, a scope narrowing, or a new egress class — never just a sterner prompt. Then the entire attack set re-runs as a regression suite, because policy changes interact. The set then runs quarterly and after every new tool, model or data source.
Related guides
- What Is Prompt Injection? — the attack family the exercise weights heaviest.
- Contain Prompt Injection in Production — the layered defenses the red-team verifies.
- Policy Enforcement for Autonomous Agents — where the fixes land after a failure.
- AI Agent Audit Trails: What to Log — proving every hold with the event record.
- Canary Tokens for AI Agents — safe credentials for production-shaped tests.
- What Is Agentjacking? — the tool-abuse family in depth.
Generate a free canary token → No signup, nothing stored: plant a tripwire where your agent works, and let the next rogue action announce itself.