Two agents that do
on-call work

The demo, end to end

A bad release breaks a liveness probe. Nothing here is simulated: the alert is a real Grafana rule over real Prometheus data, the cluster is a real kind cluster, and the pull request is a real pull request.

1. A commit changes the demo service's health endpoint. ArgoCD auto-syncs it. The liveness probe now points at a path that 404s.
2. Pods start failing their probe and restarting. kubelet · CrashLoopBackOff
3. Grafana fires ReplicasUnavailable. The rule is Terraform-managed. Its webhook posts to the orchestrator, bearer-authenticated.
4. The orchestrator admits the alert and starts a healing session. Deduplicated per fingerprint, rate-limited, flap-delayed. One session per incident, not one per notification.
5. SRE-Oncall investigates. Grafana MCP for the alert rule and PromQL · Kubernetes MCP for pod status, events and the previous container's logs · ArgoCD MCP for "what synced, and when"
6. It correlates the failure to the sync. "This started four minutes after sync abc123, which changed the probe path" — the single most valuable sentence in an incident.
7. It opens a revert pull request — without asking. Creating a branch and a PR changes nothing that runs. The PR is the artefact a human reads, so producing one must not itself wait on approval.
8. The PR body carries the whole case. CAUSE with evidence · CHANGE · WHY · EXPECT · RISK. The reviewer never has to read back through the session.
9. A human reviews and merges. This is the gate. The agent cannot merge — merge_pull_request is gated and the prompt forbids it outright.
10. ArgoCD syncs the revert. The probe recovers. Verified by re-querying the alert's own expression, not by a pod turning Ready — a crashlooping pod is Ready in bursts.
11. The alert resolves and the agent posts the summary in the same Slack thread. Then drafts a postmortem into Notion.

Who calls what

The healing loop as it actually runs, one lane per phase. Every hop names the MCP server that carries it — that is the part worth being precise about, because it is where the agent's reach begins and ends.

The agent never merges. It reaches production only by writing a document a person reads. Opening that document is ungated on purpose — stopping to ask permission to open a PR leaves the incident burning while you wait for permission to do the one thing that helps.

Slack → run an automation

Two Slack apps, two Socket Mode connections. A mention fires in any channel a bot is in, so each bot is scoped to its own — otherwise both would answer in the same room.

Two agents, disjoint tools

The split is not organisational tidiness. The two jobs pull the harness configuration in opposite directions, and giving one agent both sets of tools makes each one worse.

Sub-agents are off on both, deliberately. A fresh sub-agent starts cold and re-reads every tool schema already paid for. The binding constraint here is a tokens-per-minute ceiling, not reasoning depth — investigations have died at 535k and 618k input tokens. The prompt says not to delegate; the config enforces it.

Which agent holds which MCP servers

Because the sets are disjoint, the automation agent cannot touch production even if a requirement asks it to — its blast radius is the workflows it writes. The split also stops each agent paying tool-schema tokens for the other's servers on every single request.

Every tool, and where the gate is

Gating is declared per MCP attachment and enforced by the harness. The interesting part is not that gates exist — it is where they are not, because each of those is a decision that had to be argued for.

Why the gates are structural

A gate is not an instruction the model chooses to follow. It is declared in the agent manifest as requireApprovalForTools, and the harness refuses to execute the call until a human decides.

A paused turn and a finished turn both report status "done" — the pending gates are in requiredActions. Treating that as the end of the session is the subtle bug that would silently strand every approval, so the surface detects it specifically.

Said plainly: what the gates do not cover

"Never call alerting_manage_rules with update" and "never kubectl a fix into place" are instructions in the prompt, not access controls. The harness gates by tool name; it has no argument-aware authorization. A model error or a prompt injection could in principle act outside those rules. That is a real limitation of the design, not a detail to leave out of the diagram.

What runs where

Everything that touches a cluster or holds a vendor credential runs on the local host. The only thing deployed publicly is this page, which is static and can invoke nothing.

Two Slack bots, not one. SRE-Oncall owns the incident channel and hears about alerts; Automation-Agent lives in its own channel and routes to the automation agent. Each is a separate Slack app with its own identity and its own Socket Mode connection, and each is scoped to its own channel — a mention fires in any channel a bot is in, so without scoping two bots in one room would both answer.

The loops

Safety model

What is real, and what is not

A demo that overstates itself is worse than a smaller one that does not. This is the honest split.