Agent Reliability Playbook - Part 4: Close the Loop with Postmortems and QA Backlogs

Telemetry, alerts, and drills only matter if they change the roadmap. The final part of the series shows how to codify postmortems, feed issues into a reliability backlog, and run experiments to keep quality trending upward.

Scenario: Monthly Reliability Review

Every month, Lumenly's platform team meets with CX and finance to review:

• All Sev1/Sev2 incidents and their fixes.
• Quality audit scores from sampled transcripts.
• Experiment results that reduced hallucinations or costs.

We'll automate data collection so the meeting is about decisions, not scavenger hunts.

---

Step 1: Standardize Postmortems

Create a Markdown or Notion template with consistent sections.

# Incident PM-{{id}}
- Date:
- Severity:
- Owners:

## Summary

## Timeline

## Root Cause

## Customer Impact

## Action Items
- [ ] owner :: due

Store templates in docs/postmortems/PM-<id>.md. Require every Sev1/Sev2 to use the format and link to traces, NDJSON, and PagerDuty IDs.

---

Step 2: Build a Reliability Backlog

Aggregate unresolved action items into a single backlog.

Node script to sync action items to Linear/Jira

// reliability/backlog-sync.js
import fetch from "node-fetch";
import fs from "node:fs";

const files = fs.readdirSync("docs/postmortems");
for (const file of files) {
  const content = fs.readFileSync(`docs/postmortems/${file}`, "utf-8");
  const actions = [...content.matchAll(/- \[ \] (.+?) :: (.+)/g)];
  for (const [, summary, owner] of actions) {
    await createTicket(summary.trim(), owner.trim());
  }
}

async function createTicket(summary, owner) {
  return fetch(process.env.LINEAR_URL, {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${process.env.LINEAR_TOKEN}` },
    body: JSON.stringify({
      summary,
      teamId: process.env.LINEAR_TEAM,
      assigneeId: await mapOwner(owner)
    })
  });
}

Run this script weekly so no action item gets lost.

---

Step 3: Score Quality via Transcript Audits

Sample transcripts from Part 1's NDJSON logs and grade them.

Python audit job

# reliability/audit.py
import json, random
from pathlib import Path

LOG_DIR = Path("logs/missions")
SAMPLE_SIZE = 25

rubric = {
    "policy_adherence": 0,
    "tone": 0,
    "accuracy": 0
}

def sample_transcripts():
    files = list(LOG_DIR.glob("*.ndjson"))
    return random.sample(files, min(SAMPLE_SIZE, len(files)))

def score(file):
    events = [json.loads(line) for line in file.read_text().splitlines()]
    # Replace with human review UI or LLM judge
    rubric["policy_adherence"] = int(any("policy violation" in evt.get("notes", "") for evt in events))
    return rubric

Push scores to a warehouse (Snowflake/BigQuery) and chart rolling averages in Looker or Grafana.

---

Step 4: Track Experiments and Regression Gates

When you tweak prompts or tools, log the experiment metadata so you can roll back fast.

# reliability/experiments.py
import uuid
from datetime import datetime

def log_experiment(name, hypothesis, metrics):
    return {
        "id": str(uuid.uuid4()),
        "name": name,
        "hypothesis": hypothesis,
        "metrics": metrics,
        "timestamp": datetime.utcnow().isoformat()
    }

Store experiment logs alongside drill metrics. Update CI so a change cannot merge unless:

• Drill suite (Part 3) passes.
• No open Sev1 actions exist.
• Experiment doc includes rollback plan.

---

Step 5: Run the Reliability Review

Automate agenda creation:

1. Query incidents from incident:timeline where ts falls in the past month.
2. Pull open backlog tickets labeled reliability.
3. Summarize audit scores and experiment outcomes.
4. Email the packet or publish to Confluence.

Example Python summary:

# reliability/review.py
import json
from datetime import datetime, timedelta
from pathlib import Path

LOOKBACK = 30
INCIDENT_LOG = Path("logs/incidents")

def gather_incidents():
    cutoff = datetime.utcnow() - timedelta(days=LOOKBACK)
    incidents = []
    for file in INCIDENT_LOG.glob("*.ndjson"):
        with file.open() as fh:
            records = [json.loads(line) for line in fh]
            if records and records[-1]["ts"] >= cutoff.isoformat():
                incidents.append(records[-1])
    return incidents

Send the packet to execs plus the ops team. Track decisions and add new work to the backlog script above.

---

Step 6: Implementation Checklist

• Adopt the standard postmortem template with required links.
• Sync unresolved action items into a tracked backlog.
• Automate transcript sampling and scoring.
• Log experiments with metrics plus rollback plans.
• Generate a monthly reliability packet for stakeholders.

You now have a living reliability program: signals (Part 1), actions (Part 2), drills (Part 3), and governance (Part 4). Keep iterating on the backlog and drill matrix as your agent grows.

---

Continue Building

• Part 1: Instrument, Replay, and Fix Faster
• Part 2: Detect, Triage, and Page Faster
• Part 3: Run Reliability Drills Before Production