How the agents evaluated themselves

Summary

Each Shepherd run ends with a post-mortem written by an agent from that run's own artifacts. Read on their own, the five post-mortems support six conclusions:

1. Only the control finished in one run. It merged 5 of 5 tasks and exited 0. Both treatment campaigns stopped once and needed a second run for the last task.
2. Both stops were agent errors, and the post-mortems say so plainly. In run-03, the coding agent finished a remediation cycle that changed nothing. In run-04, the local shepherd decided from help text that GitHub CLI couldn't request a Copilot review, although the same run had already done so three times.
3. The agents rate review convergence as strong in every report, including the failed runs. Self-reported Copilot Code Review findings were 1, 1 and 4 per campaign, and every report says all findings were resolved before merge.
4. The self-evaluations don't agree with each other on method. Given the same kinds of artifacts, the reports call token counts unavailable, measured, or redacted, and they mix UTC and Eastern time.
5. All five ask for the same fix: a machine-readable outcome record per stage, so the next post-mortem doesn't have to reconstruct metrics from transcripts.
6. The post-mortems can't see the experiment. All five describe their run as a "control" because lesson propagation was off. None mentions guardrails, and none records how PR #10 in run-04 was merged.

The five post-mortems

Each campaign directory holds one post-mortem per run. The treatment campaigns each ran twice, so there are five reports for three campaigns.

Arm labels come from the repository names (cargotracker-control-01, tricket-out and tricked-out). The post-mortems themselves call every run a control run.

Task outcomes as reported

By their own accounts, the agents merged 14 of 15 tasks. The fifteenth, run-04's issue #5, stopped before review, and no post-mortem says what happened to it afterward.

Only one post-mortem reports problems fixed before review: the control's stage 30 caught out-of-scope Maven Wrapper files and a formatting failure on issue #5. The other four reports describe stage 30 only as validation that passed, so they either saw no such problems or didn't report them. The post-mortems alone can't tell which.

Self-reported numbers

These are the numbers as each post-mortem states them, summed across a campaign's runs where it had two. A blank cell means no post-mortem for that campaign reported the figure.

Two of the treatment figures cover only part of a campaign, because the other run's post-mortem reported tokens as unavailable. Token totals can't be compared across campaigns from these reports.

The control's wall clock includes a 10h 49m 16s interval between stage 30 and stage 40 of issue #4. Its post-mortem says the cause is "not observable locally" and declines to guess.

How the agents explained their failures

Both terminal failures were attributed to agent behavior, not to the code under change, and both post-mortems call the shepherd's fail-closed stop correct.

The run-04 post-mortem is the most self-critical of the five: its own section headed "What failed" lists the wrong capability conclusion, the ignored contrary evidence and the misleading error message. It also estimates that without the error the run would likely have merged four tasks instead of three, which is the report's speculation.

How consistent the self-evaluations are

The same agent, reading the same kinds of artifacts, reached different conclusions about what the artifacts contain.

Small arithmetic disagreements also appear inside single reports. The control post-mortem gives its session total as 2h 16m 34s in one table and 2h 16m 29s in another; run-04's second report gives 41m 01s and 41m 02s.

The level of detail varies too. The control report lists each pre-review remediation; the treatment reports mostly summarize stage 30 as passed validation. That makes the reports a weak basis for comparing stage-30 activity between arms.

What the post-mortems leave out

• The experiment's variable. No report mentions guardrails. Each frames its run as a lesson-propagation control and suggests comparing against a future "campaign" lesson mode.
• PR #10 in run-04. The first run left it draft and unreviewed; the second run targeted only issue #6. No post-mortem records #10 being reviewed or merged.
• How run-04 resumed. Run-03's recovery report documents the maintainer's manual evidence work. Run-04's second report doesn't say what changed between runs.
• Remote agent cost. Every report notes that coding-agent and review-agent credits are absent from the local artifacts.
• The control's 10h 49m gap. Reported as an observability limitation with no cause.

What the agents recommend

The recommendations are remarkably uniform: the agents mostly ask for better records of their own work.

The first run-03 report's top-priority action was to resume issue #6 after adding the missing evidence to PR #11. The recovery report shows that this is what happened.

Method

• Source: only the five *-post-mortem.md files in post-mortems.zip. No evaluator output, transcripts or repository data were used.
• Combining runs: where a campaign has two post-mortems, counts and durations are summed as each report states them. Nothing was re-derived.
• Classification: outcome categories in the chart follow each report's own description of the task. The "first review clean" counts include only tasks a report says were reviewed.
• Trust: each post-mortem was written by an agent from its own run's artifacts. This report checks the post-mortems against each other, not against those artifacts.