Guides · ← All guides
The On-Call Runbook for a Rogue AI Agent at 3 AM
By the Ingenious AI team · Published: November 14, 2026 · Reviewed: November 14, 2026 · 7 min read
The short answer
A 3 AM runbook has one design constraint: the reader is half-asleep, alone, and holding a phone — so the first page is five actions in order, each completable in under two minutes, none requiring a judgement call harder than reading a number. Verify it is real (canary or boundary event, not a dashboard ghost), quarantine the agent, glance at the egress for the one question that changes everything (did data leave), rotate or freeze the top credentials if it did, and write the note. Everything else — the scoping, the notifications, the postmortem — is morning work, and the runbook's second job is protecting the night engineer from attempting it at 3 AM.
Generate a free canary token → No signup, nothing stored: plant a tripwire where your agent works while you read — most controls in this guide start working the moment a canary, a policy check, or a spend cap is in place.
What is a rogue agent on-call runbook? the pre-written, ordered procedure an on-call engineer follows when an agent alert fires out of hours — what to check, what to trigger, who to wake and what to record — designed to be executable without improvisation.
Page one: the first five minutes
Minute zero to one: open the alert and identify the signal class. A canary touch or an exfil-secret event is real by construction — act, do not investigate. A velocity or cost alert gets one check: is the run still climbing. Minute one to two: quarantine the named agent — the control from the quarantine guide, one action, pre-authorised. If the alert names several agents or the class is spreading, that decision is already made for you: kill switch, fleet scope, questions in the morning.
Minutes two to five: read three numbers from the trail — first off-policy call time, destinations outside the allowlist (count), records in executed outbound actions (count). Those three numbers decide the rest of the night. Zero and zero: the boundary held, the incident is a morning review with a scary title. Anything else: you are in the wake-people section, and the numbers are what you wake them with.
The decision tree fits on one screen
Resist the branching flowchart with fourteen nodes; the night tree has three forks. Fork one — did anything execute outside the allowlist? No: quarantine stands, note it, back to bed, morning review. Yes: fork two. Fork two — did records or secrets actually leave? No (attempts blocked, holds caught): freeze the agent's top credentials as a precaution, wake the owner at a civilised hour by message, morning review leads. Yes: fork three. Fork three — is the data personal or secret-class? Either answer wakes the incident lead now; personal data also starts the notification clock the lead owns.
Every fork is answerable from the trail in minutes, which is a property of the logging, not the engineer. If your night engineer cannot answer fork two from the tools on their phone, the runbook's real finding is a tooling gap — file it as the incident's first action item, in daylight.
Who to wake, and what to say
The runbook names roles, not heroes: the incident lead (any confirmed egress of personal or secret data), the agent's owner (any quarantine expected to last past morning), and nobody else until the lead says so. Each wake-up carries the three numbers and the state: agent quarantined at 03:07, first bad call 02:58, two unlisted destinations attempted, zero records out — a message that lets the woken person think immediately instead of interviewing the engineer.
Write the thresholds down in advance, in the runbook, signed by the people who will be woken. The engineer at 3 AM should never be deciding whether an event is worth a director's sleep; that decision was made in a meeting, in daylight, by the director. Pre-authorisation is what makes the night fast — the same principle as the kill switch pull, applied to telephones.
The note you leave for morning
Six lines, written before going back to bed, in the incident channel where the morning team will look: what fired and when; what you triggered and when; the three numbers; what you froze or rotated; who was woken and what they decided; what is still unknown. The note is not a report — it is a baton. Morning's scoping starts from the preserved trail and your six lines, and between them nothing needs the night engineer's memory, which by 9 AM is unreliable evidence anyway.
Attach nothing and summarise nothing else. The temptation to be thorough at 3:40 AM produces documents the morning team must fact-check; the six-line discipline produces a handover they can build on. Thorough happens in the 24-hour sequence, staffed and awake.
Rehearse the night before the night
Runbooks rot: the button moves, the rota changes, the trail query gets renamed. Rehearse quarterly with a game night — a simulated canary page at a civilised hour, the on-call engineer executing page one against a staging agent, timed. Every friction found (a login that needs a laptop, a number that needs three screens) is a runbook bug fixed in daylight.
Rotate the role and debrief the real nights without blame: the runbook's quality is measured by how boring the real 3 AMs are. The best outcome in this entire field is an incident that reads as: alert, quarantine, numbers, note, sleep — and a morning team that finds a frozen agent, a preserved trail and nothing to reconstruct.
Five actions to take today
- Write page one. Five actions, in order, each under two minutes, none needing a laptop or a judgement call.
- Draw the three-fork tree. Executed outside allowlist? Data actually left? Personal or secret class? — answerable from the trail by phone.
- Name the wake thresholds. Roles and the exact conditions that wake each, signed by the people who will be woken.
- Template the six-line note. Signal, action, three numbers, freezes, who was woken, what is unknown — pinned in the incident channel.
- Schedule the quarterly game night. Timed, staged, friction logged as runbook bugs with owners.
A worked example: a quiet 3 AM at Perrin & Cole
The following is a fictional example built from real threat classes and controls. The company and numbers are invented; the mechanics are not.
Perrin & Cole, a 30-person accountancy, gets its page at 03:12: a canary touch, exfil-secret class, on the client-onboarding agent. The on-call engineer — six weeks into the rota, runbook on her phone — does not investigate a canary; page one says act. Quarantine lands at 03:14.
The three numbers, read by 03:19: first off-policy call 03:09; unlisted destinations attempted, 1; records in executed outbound actions, 0. Fork one: attempts yes, executions no — the boundary blocked the call carrying the canary. Fork two: nothing left. The runbook's answer is freeze nothing further, message the agent's owner for a civilised hour, write the note.
The note, timed 03:24, is six lines and complete. She is back asleep by 03:40. The morning review scopes a clean story — an injected instruction in a client's uploaded document, blocked at its first action — and the incident closes inside the week with one policy addition. Perrin & Cole's partners never learn the incident happened at 3 AM, which the engineering lead offers, correctly, as the highest praise a runbook can earn.
Frequently asked questions
What should an on-call engineer do first when an agent alert fires at night?
Identify the signal class, then act before investigating: a canary touch or exfil-secret event is real by construction. Quarantine the named agent inside two minutes — or pull the fleet kill switch if several agents are named — then read three numbers from the trail: first off-policy call, unlisted destinations attempted, records actually out.
When should the on-call engineer wake other people?
On the runbook's pre-agreed thresholds, not on personal judgement: the incident lead wakes for any confirmed egress of personal or secret-class data; the agent's owner wakes (or is messaged for morning) for any quarantine lasting past the night. Thresholds are signed in daylight by the people who will be woken.
What is the most important thing to record during a night incident?
The six-line handover note: what fired and when, what you triggered, the three numbers, what was frozen or rotated, who was woken and their decision, and what remains unknown. It is a baton for the morning team, written before sleep — memory by 9 AM is unreliable evidence.
How is a night runbook different from an incident response plan?
Scope and audience. The full plan covers 24 hours — scoping, notification, review — executed by a staffed team. The runbook covers the night: contain, measure three numbers, decide wake-ups, leave the note. Its second job is protecting the night engineer from attempting daylight work at 3 AM.
How do you keep an on-call runbook from going stale?
Quarterly game nights: a simulated page at a civilised hour, page one executed against a staging agent, timed, with every friction — a login needing a laptop, a number buried three screens deep — filed as a runbook bug with an owner. Real nights are debriefed the same way, without blame.
Related guides
- AI Agent Incident Response: The First 24 Hours — the daylight plan the runbook hands over to.
- How to Quarantine an AI Agent — page one's central action.
- How to Kill Switch an AI Agent — the fork for multi-agent shapes.
- Human Approval Queues for AI Agents — the queue the night engineer may inherit.
- AI Agent Audit Trails: What to Log — where the three numbers live.
- AI Agent Incident Postmortem Template — the morning after, structured.
Generate a free canary token → No signup, nothing stored: plant a tripwire where your agent works, and let the next rogue action announce itself.