Agent Incident Postmortem

intent

use this skill when an ai agent or llm feature caused an incident (hallucinated facts shipped to users, runaway tool use, prompt injection, cost blowouts, wrong autonomous actions) and you need a blameless postmortem. ai incidents differ from outages: the system stayed up but did something wrong, confidently, and maybe only once. this skill adapts blameless postmortem practice to nondeterministic systems, where reproducibility requires traces and statistical measurement, not just step replication. output is a structured postmortem document with timeline, user/business impact, trace reconstruction, root-cause analysis across ai failure layers, and corrective actions including a permanent regression test case.

inputs

gather these before starting (ask if not already provided):

• incident description: what the agent did vs. what it should have done. be specific: "hallucinated a support ticket number" or "called delete_user on wrong id" not "was wrong".
• execution trace: full request including system prompt, context fed to the agent (retrieval results, user input, injected data), all tool calls and their results, model output. if no trace exists, document that absence as a finding.
• blast radius: count of affected users/requests, time window, whether ongoing, and any cost or regulatory impact (data exposure, soc-2 violation, etc.).
• detection method: how it was found (user report, alert, manual review, luck) and time gap between incident start and detection.
• model/prompt versions: exact model name, version, temperature, system prompt version, retrieval index timestamp if applicable. pinning this matters for reproducibility.
• tool call permissions and guardrails: which tools the agent can call, what validation existed on inputs/outputs, what was missing.

external connections (if applicable):

• llm api (openai, anthropic, etc.): api key in env var LLM_API_KEY. scope: model inference only. note rate limits and billing account in blast radius.
• tracing/logging system (datadog, honeycomb, cloudwatch): connection string in env var TRACE_BACKEND_URL. scope: read full execution traces. required to reconstruct exact context.
• regression test suite: write access to your test harness (pytest, jest, custom). permission to add new permanent test cases.
• slack/incident channel: optional, for async postmortem discussion and action item tracking.

procedure

1. collect the incident snapshot: obtain the execution trace, timestamp, model/prompt versions, and user report. document any missing pieces (if trace is unavailable, note this as a finding now).

   • input: incident report (verbal or written), trace logs or api response logs
   • output: incident summary document with all metadata above, gaps listed

2. reconstruct the execution trace: walk through exactly what context the agent received, what it decided, what tools it called, what results came back, what it output. cite the trace verbatim.

   • input: execution logs, api calls, tool results
   • output: timestamped trace document formatted as: [timestamp] context received → [timestamp] tool_call(args) → [timestamp] tool_result → [timestamp] model_output. flag where behavior diverged from intended path.

3. map the root-cause layers: walk each layer in order (input/context, model behavior, guardrails, action layer, detection). for each layer, ask: "could this layer have prevented the incident?" the root cause is the earliest layer that could have stopped it.

   • input: trace from step 2
   • output: root-cause table with finding for each layer
     • input/context: was context wrong, stale, contradictory, poisoned (injection)? did retrieval feed bad ground truth? were tool results corrupted or incomplete?
     • model behavior: given that context, was the output a known failure mode (fabrication when data missing, over-compliance with injected text, reward hacking)? is this a capability gap or a behavioral failure?
     • guardrails: what check should have caught this output and didn't exist or didn't fire? (schema validation, hallucination detector, action allow-list, cost limiter)
     • action layer: why could wrong output reach a user or execute without appropriate gate for risk level? was there manual review, approval, rate limiting, rollback capability?
     • detection: why learn about it this way, this late? what signal would have caught it in minutes instead of hours/days?

4. measure reproducibility and frequency: attempt to replay the exact trace (same context, model version, temperature, tool results). then run N times (minimum 5, recommended 20+) to measure how often the failure occurs.

   • input: pinned trace context (step 2), access to llm api and tools
   • output: reproducibility report: either "reproduced on [N]/[M] runs, [percentage]% failure rate" or "not reproducible under these conditions; frequency unknown". include which parameters (temp, model version, etc.) affect frequency if known.
   • note: if failure is 1-in-50 and you run 10k requests/day, that's 200 incidents/day. frequency matters for urgency.

5. write the postmortem document: assemble into structured format below. ensure blamelessness toward humans (no "prompt engineer error" or "operator fault") while being ruthlessly useful about system failure. cite the trace, cite the layer analysis, cite reproduction results.

   • input: all outputs from steps 1-4
   • output: postmortem document in format specified under "output contract"

6. write corrective actions: for each, specify which layer it addresses and whether it prevents, detects, or mitigates. always include at least one permanent regression case and one detection improvement.

   • input: root-cause findings, guardrails audit, detection gaps
   • output: action table with action, layer, owner, due date. verify no action is "improve the prompt" alone.

7. add permanent regression test: write a test case that exercises the exact trace from step 2 and asserts the model does not produce the bad output. add to test suite. this case should run in ci/cd.

   • input: trace, guardrails, expected correct output
   • output: test code in your regression suite. commit and document test id in postmortem.

decision points

if trace is fully available: reconstruct it step 3 (input/context through detection layers) and proceed to reproducibility testing.

if trace is partial or unavailable: document what's missing (were logs not captured? trace backend down?). treat absence of data as a finding in "detection" layer. do not speculate about missing steps; flag as unknown and recommend adding traces retroactively.

if failure is reproducible 100% of the time: root cause is often in input/context or model behavior. proceed to guardrails and action layer to find why it wasn't caught. set reproduction frequency to 100%.

if failure is intermittent (5-95% frequency): root cause usually in model behavior (nondeterminism at this temp/model) or guardrails (probabilistic check that doesn't always fire). note that rerunning at different temp will change frequency; always pin temperature in reproduction. mark frequency as "measured" not "estimated".

if failure is not reproducible (0% in local replay): context or tool results must differ in production. investigate stale retrieval index, live tool side effects, or production model serving a different version. call out that frequency is unknown and recommend production instrumentation (trace sampling, live replay). do not close this action as "rare".

if root cause spans multiple layers: identify the earliest one (the one that, if fixed alone, would prevent recurrence). flag other layers as secondary (e.g., "model behavior is a known failure mode; guardrails should catch it anyway").

if detection gap is large (hours/days between start and discovery): prioritize detection improvement as first corrective action. new alerts, sampling, or user feedback loops belong here.

if blast radius is ongoing: mark status as "monitoring" not "resolved" until traffic returns to baseline and no new incidents observed.

output contract

produce a markdown document named postmortem-[incident-slug]-[date].md. format must be:

# AI Incident Postmortem: [incident title] ,  [date]

**Severity:** [critical/high/medium/low] · **Status:** [resolved/monitoring] · **Owner:** [name]

## Summary
[3 sentences: what agent did, impact, root cause layer]

## Impact
- Users/requests affected: [number, timeframe]
- Window: [start datetime → end datetime, duration]
- Cost: [if applicable, in dollars or usage units]
- Trust/regulatory: [data exposure, soc-2 gap, user trust damage, etc., or "none"]

## Timeline
- [timestamp] incident start: [first bad output]
- [timestamp] detection: [how found, by whom]
- [timestamp] mitigation: [what was done, by whom]
- [timestamp] resolution: [fully fixed, monitoring in place]
- **Detection gap: [duration between start and detection]**

## Trace Reconstruction
[full execution trace as timeline, verbatim from logs]
[tool calls and results]
[where behavior diverged]

## Root Cause by Layer

| Layer | Finding |
|---|---|
| Input/context | [citation to trace] |
| Model behavior | [citation to trace] |
| Guardrails | [citation to trace] |
| Action layer | [citation to trace] |
| Detection | [citation to trace] |

**Earliest preventative layer:** [which layer could have stopped this]

## Reproduction
- Reproduced: [yes/no]
- Frequency: [N/M runs, percentage] OR [not reproducible, frequency unknown]
- Pinned parameters: [model, temp, prompt version, retrieval index, tool versions]

## Corrective Actions

| Action | Layer | Owner | Due |
|---|---|---|---|
| Add regression test case [test id] | eval | [name] | [date] |
| [action] | [layer] | [name] | [date] |
| [detection/alert improvement] | detection | [name] | [date] |

## What Went Well / What Got Lucky
- What went well: [human or system response that contained damage]
- What got lucky: [what almost went worse, or what found it early]

## Quality Checklist
- [ ] Blameless toward humans, useful about system (no "prompt engineer error", no "model hallucinated" as conclusion)
- [ ] Root cause is earliest preventative layer, not just layer that misbehaved
- [ ] Trace (or absence) is in document; findings cite it
- [ ] Failure frequency measured or explicitly marked unknown
- [ ] Corrective actions include permanent regression case + at least one detection improvement
- [ ] No action is "improve the prompt" alone; guardrails or gates also included
- [ ] Injection question asked if any untrusted text in window
- [ ] Model upgrade is tracked as migration, not as fix

outcome signal

the postmortem is done when:

• document is written, shared, and linked in incident tracker
• all root-cause findings cite the trace or trace absence
• at least one corrective action is assigned and has a due date
• permanent regression test case is written, committed, and passing (asserts the bad behavior does not happen)
• owner confirms status is either "resolved" (all critical actions complete) or "monitoring" (detection in place, fixes in progress)
• team has reviewed and no "approved but disagree with root cause" comments remain

measure success by: regression test runs in ci/cd, detection alert is live and would catch this class of incident in the future, and the same failure does not recur within 30 days of resolution.