Two views of one experiment
Summary
The two reports agree on outcomes, timing and requests, but disagree on what went wrong. Each one caught errors in the other.
- The post-mortems mention only 9 of the evaluator's 29 product defects. They report what code review found and almost nothing the stage-30 gate or CI caught in the treatment runs. The agents summarize gates they passed as passed, not as problems fixed.
- The post-mortems correct the evaluator in three places. Run-04's stop was an agent error, not a harness problem. Two of run-04's "behavioral" review findings were formatting. And the evaluator undercounted code review rounds.
- The evaluator corrects the post-mortems in three places. Token counts the agents called unavailable or redacted were recoverable. Run-04's PR #10 merged, which no post-mortem records. And the duplicated dialog code, repeated tooling misuse and test-integrity results never appear in the agents' reports.
- Both agree on the headline facts: session time, wall clock, premium requests, the run-03 failure, and that every review finding was resolved before merge.
- For the talk, neither report is reliable alone. The agents' self-evaluation misses most of what went wrong, and an evaluator built with an agent misclassified events the agents described correctly. Cross-checking the two found errors in both.
The two reports
| Results report | Self-evaluation report | |
|---|---|---|
| File | cargotracker-guardrails-results.html | cargotracker-agent-self-evaluation.html |
| Source | Evaluator 0.4.4 output, plus transcript and git checks during review | The five post-mortems only |
| Written by | An evaluator Copilot built, reviewed by hand against raw data | A post-mortem agent at the end of each run, reading that run's artifacts |
| Knows the arms | Yes: control versus guardrail treatment | No: every run calls itself a lesson-propagation control |
How much each report sees
The post-mortems mention 9 of 29 product defects: half of the control's, and under a fifth of treatment run-03's.
All four of run-04's mentions are code review findings. In run-03, the post-mortems mention the review finding on #4 and the missing evidence on #6, but none of the nine problems the stage-30 gate and CI caught on #2 to #5: four scope violations, two formatting failures and three test-inventory failures. Only the control's report names any pre-review problem, the scope violation and formatting failure on issue #5.
The gap is largest exactly where the treatment added guardrails. The agents treat a gate failure that was fixed before review as part of passing validation, so the guardrails' main effect, catching problems earlier, is invisible in their self-evaluation.
Where they agree
| Measure | Both reports |
|---|---|
| Recorded session time | 2h 16m, 4h 13m, 4h 14m |
| Wall clock, including gaps | 13h 58m, 5h 31m, 5h 23m; the control's 10h 49m gap during task #4 |
| Premium requests | 10, 11, 10 |
| Run-03 AIU | 1,051.1 |
| Code review findings | 1, 1 and 4, all resolved before merge |
| Run-03, issue #6 | Failed in stage 30; the coding agent's remediation cycle pushed nothing |
| Control, issue #5 | Out-of-scope Maven Wrapper files and a formatting failure, fixed before review |
Where they disagree
On six points the reports conflict: the post-mortems are better supported on three, the evaluator on two, and one needs both.
| Point | Results report | Post-mortems | More likely right |
|---|---|---|---|
| Why run-04 stopped after task #5 | A harness failure, repaired by hand | The shepherd wrongly concluded gh couldn't request a Copilot review, contradicting three successful requests in the same run | Post-mortems: a specific, evidenced agent error |
| Run-04, issue #3 review findings | Two behavioral defects | Two low-severity findings about unrelated file-wide formatting | Post-mortems: the evaluator labels every review finding behavioral |
| Code review rounds | 3, 6, 2 | 6, 6, 7 | Post-mortems: every merged PR was reviewed at least once, so the control can't have had only 3 |
| Token counts | Measured for all three campaigns | Unavailable or redacted in three of five reports | Results: the token metrics are readable even though span attributes are redacted |
| Run-04, PR #10 | Merged | Left draft; never recorded as reviewed or merged | Results: the evaluator found its merge commit |
| How run-03's #6 was recovered | Harness repair committed between directories | The maintainer ran the acceptance tests by hand and added the evidence to the PR | Both: the post-mortem adds that a human did the agent's acceptance work |
What only one report sees
Only the results report:
- The guardrail arms and every comparison between them.
- Most stage-30 and CI detections: formatting failures in every run, test-inventory failures, scope violations in the treatment runs, and the unit-test failure in run-04.
- The 16-line duplicated dialog code, present in all three runs.
- Repeated tooling misuse: 21 events, mostly the same invalid
ghflag combinations. The post-mortems mention one. - Test integrity: no tests deleted or disabled in any PR, and the four legitimate edits to gate files.
- Zero new deprecation warnings, and model and tool call counts for all runs.
Only the post-mortems:
- What each review finding was about: null-date handling, missing-cargo handling, strict-date parsing that took two fixes, and the formatting findings.
- The capability false negative that stopped run-04, and the post-mortem's own judgment that the error message misled the user.
- A review that reported zero findings while calling one observation "previously missed".
- That a maintainer, not an agent, produced the evidence that let run-03's last task merge.
- Test counts at each gate, and a consistent set of process recommendations, led by per-stage outcome records.
Corrections to the results report
The comparison surfaced four errors in the results report, and the results report now includes these corrections. None changes its cost or formatting claims, but two change the behavioral-defect claim.
| Item | Before | After |
|---|---|---|
| Run-04 behavioral defects | 5 (4 by review, 1 by CI) | 3 (2 by review, 1 by CI); the other 2 review findings are formatting |
| Summary claim 1, treatment | 5 of 6 behavioral defects caught by review | 3 of 4 caught by review, 1 earlier by a CI unit test |
| Code review rounds | 3, 6, 2 | 6, 6, 7, from the post-mortems' per-task tables |
| Failed task attempts | 0, 1, 0 | 0, 1, 1; run-04's task #5 stopped on the agent's capability error |
The harness caveat changed too. Run-03 needed a maintainer to do the acceptance testing; run-04 needed a restart after an agent error. Neither was a guardrail change, but neither was purely a harness repair either.
What this means for the talk
- Agent self-evaluation supports H1. The post-mortems describe gates as passed and miss most of what those gates caught. An agent that can't say what went wrong in its own run doesn't "know" the codebase or the process the way a reviewer does.
- A capability hallucination is a strong H2 example. Run-04's shepherd decided a working tool was unsupported, against evidence from earlier in the same run, and the whole campaign stopped.
- Guardrail effects are invisible to the agents. The treatment's earlier detections show up only in the external evaluation. If you judged the arms by the agents' own reports, the guardrails would appear to do nothing.
- Cross-checking is the method worth showing. The evaluator, built with an agent, also erred: it labeled formatting findings as behavioral and undercounted review rounds. Only comparing the two reports, and checking disagreements against the raw data, produced numbers you can stand behind.
Method
- Sources: the two HTML reports, and for disagreements, the post-mortem text and evaluator data they were built from.
- Matching defects: a post-mortem mentions a defect when it describes the same problem on the same task. Control: issue #4's null-date finding and issue #5's scope and formatting problems. Run-03: issue #4's missing-cargo finding and issue #6's missing evidence. Run-04: the four review findings on issues #3 and #4.
- Not yet verified: the reclassification of run-04's two issue #3 findings rests on the post-mortem's description. A look at review comments on PR #8 in that repository would confirm it.