Cargotracker guardrails experiment: results for Devoxx slides

Summary

With one control run and two treatment runs, the results support a case study, not a statistical comparison. Five claims hold up against the data:

1. Behavioral defects still reached code review in both arms. Control: 1 of 1 caught by Copilot Code Review. Treatment: 3 of 4 caught by review, 1 caught earlier by a CI unit test.
2. Guardrails add their own failure class. The treatment's test-inventory check failed 4 times across the two runs. These are obligations the guardrail creates, so the control has no equivalent.
3. Agents never produce formatted code. Spotless runs in both arms and failed in every run (3, 2 and 4 times). Two treatment violations came from fix commits made during review.
4. Some failure modes repeat exactly, in every run, in both arms. The same 16-line duplicated dialog code, out-of-scope Maven Wrapper files, and the same invalid gh flag combinations appear in all three runs.
5. The treatment costs more. Mean AIU is 25% higher (1,101 vs 881) and recorded session time is 86% higher (4h 14m vs 2h 16m).

Every open item is now resolved, and none changed these claims.

What was compared

Both arms share the project layout, Spotless formatting and a Maven build that runs tests; the treatment adds deeper CI guardrails on top, plus a stricter test section in its plan and 20 production files already reformatted and fixed for its gates (see Caveats).

CI checks in both arms: formatting (Spotless) and a Maven build. Tests run by default (skipTests=false), the code lives in demo/, and the compiler targets Java 17 via maven.compiler.release.

Added in the treatment only: a source-gates check, plus 43 more CI steps. The ones that can fail a build:

• Test inventory, build contract, and the Java 17 / Java EE 7 contract
• Unit and Arquillian test-count assertions
• SpotBugs
• A dependency security gate
• OpenTelemetry acceptance with negative controls
• A bounded performance comparison

Held constant in all three runs: Shepherd 1.0.4, Copilot CLI 1.0.91, model gpt-5.6-sol at reasoning level medium, lesson propagation off, and the same three skill name hashes. Skill content lengths match exactly (27,427, 25,715 and 4,423 characters); content hashes were not captured.

Where each defect was first caught

In every run, at least 67% of defects were caught before code review. The treatment's extra gates moved one behavioral defect earlier.

Behavioral defects are the comparison that matters. Copilot Code Review found the control's single behavioral defect, treatment run-03's single one, and two of treatment run-04's three. The third, in run-04 task #4, failed a CI unit test first. That CI step exists only in the treatment.

Two of run-04's formatting failures happened after review: the fix commits for review comments were not formatted.

Defect classes across arms

Only behavioral defects, formatting and out-of-scope changes can be compared between arms; test-inventory failures exist only because the treatment checks for them.

* Includes one scope violation the evaluator left unclassified and I classified by hand from its excerpt (see Method). † Includes two review findings on run-04's issue #3 that its post-mortem describes as unrelated formatting; the evaluator had labeled them behavioral.

The out-of-scope changes are the same in every run: the agent adds demo/mvnw, demo/mvnw.cmd and the two .mvn/wrapper files. That happens in both arms, so it comes from the project setup, not from the guardrails.

The two treatment runs differ widely: run-04 had three behavioral defects to run-03's one. With one control run, that spread is larger than any difference between the arms.

Cost to converge

The treatment used 25% more AIU and 28% more input tokens, and its sessions ran 86% longer. The likely cause is waiting on a CI pipeline with 43 more steps; the data does not break session time down by wait type.

Elapsed wall-clock time is not comparable. The control sat idle for 10h 49m between stage 30 and stage 40 of task #4 while its Dev Box slept, and both treatment runs were split into two directories with a gap of 1h 05m to 1h 27m.

The tokens cover local Shepherd sessions only. The remote coding agent and the review agent run on GitHub, and their costs are not in the artifacts.

Symptom scorecard

Five of the eight measured symptoms appeared, and only token cost differed clearly between arms.

Scope creep was on the literature-only list, but the runs measured it: out-of-scope Maven Wrapper files appeared in all three runs, and Shepherd's stage-30 gate caught every case.

Caveats to disclose on stage

The sample is one control run and two treatment runs, so every difference above is anecdotal.

• The arms differ by deep guardrails only. Formatting and a test-running build exist in both arms by design, because both repos derive from a project that already had these basics. The experiment measures what the extra 43 CI steps add on top of them, not guardrails versus none.
• The runs overlapped. Treatment run-03 and run-04 ran concurrently for about 2h 09m. Control run-01 and the resumed part of run-04 both started at 00:42:08 UTC on Oct 3 and overlapped for 47m. All three used the same GitHub account.
• The control's idle gap broke its prompt cache. The control's uncached input was 298,285 tokens; each treatment run had under 1,000. Most likely the cache expired while the control's Dev Box slept for 10h 49m, which inflates the control's cost slightly and makes the treatment's cost increase look smaller.
• The treatment also got a stricter plan and different starting code. Compared against control-run-01's start (325b7e1), the treatment's plan requires running BookingServiceTest under Open Liberty with all five tests passing; the control's plan only requires the tests to compile. Shepherd's stage-30 gate checks these criteria, so the treatment gate asked for more evidence. The treatment plan also says to keep the Java 7 source level, contradicting its own POM (release 17). Twenty production files also differ at the start, all from four treatment commits: Spotless reformatting, SpotBugs and compiler fixes (serialVersionUID fields, a per-call SimpleDateFormat instead of a shared static one, an unused field and getter removed), an access log for observability, and removal of the GlassFish descriptor. No feature behavior changed. The files include ChangeDestinationDialog.java and BookingBackingBean.java, which the feature touches. Both treatment runs used the identical plan. By design, the stricter plan is part of the guardrail package, so results describe the package as a whole rather than the CI checks alone.
• The evaluator could not classify every event. It left 7, 10 and 23 events unclassified in the three runs. I reviewed them by hand: two were scope violations, now counted; the rest are check-status polls, skipped jobs, and agent tool or script errors.
• Both treatment runs stopped once and needed manual help to finish. Run-03's last task merged only after the maintainer ran its acceptance tests by hand and added the evidence to the PR. Run-04 stopped when the shepherd wrongly concluded GitHub CLI couldn't request a Copilot review. Harness repair commits went onto the base branch between the two directories of each run. They didn't change the guardrails, but they add about 130,000 lines to the treatment's start-to-final diff, so campaign-level diff and test-integrity numbers for the treatment are left out.

Resolved checks

All items are resolved; none changed the summary claims.

• The mid-campaign base-branch changes were harness repairs, and the agents didn't tamper with tests. Each treatment run has one repair commit (run-03 f586a46, run-04 c94b49f), named after the interrupted campaign directory. Neither touches a workflow, CI script, POM, config or source file; together they account for about 130,000 lines in the evaluator's start-to-final totals, so the treatment's campaign diff size and its "test execution reduced" flag are not agent results. Checked per PR, all ten treatment PRs delete no test files, disable nothing, add no skip flags, and add 76 assertions while removing none. Four PRs edited the guardrail files, but only to register new test classes and raise the expected counts (active tests 8→9→10; unit tests 27→34). None removed a check or lowered a count.
• The 20 production files are the treatment's own gate fixes. All changes come from four treatment commits: Spotless reformatting, SpotBugs and compiler fixes, an observability access log, and making Open Liberty the only runtime. Run-04 started from identical production code.
• Control task #5 had no convergence failure. Every SHEPHERD FAILED line in that transcript is part of a script's text, either the skill's instructions or a command's own failure branch (line 1115). None is command output. The command that contains the matched line printed its normal result, and the session's four nonzero exits (lines 1296, 1643, 1838 and 2535) belong to other commands. The evaluator matched the string inside the command rather than in its output, so the control had zero failed attempts.
• Run-04's unit-test detection counts as a guardrail detection. In the treatment, unit tests run in CI only in treatment-only steps (Run focused unit tests, Assert unit test count); the control's tests run inside its own Build with Maven step, which the treatment doesn't have.
• Why the runs paused. The control's 10h 49m gap during task #4 happened because the Dev Box running it went to sleep. Run-03 stopped when the coding agent pushed no remediation; run-04 stopped when the shepherd wrongly concluded GitHub CLI couldn't request a Copilot review. Both were restarted by hand; see Caveats.
• Control task #2 had one stage-30 request, not two; the control total is 6. Shepherd posted two change requests on PR #7. At 00:58:55 UTC it requested a fix for the Maven Wrapper scope violation plus a wrong test reload path (one request, two items). At 01:05:13 UTC it requested a formatting fix after CI failed. The evaluator's "missing requirement" defect at 01:05:07 is that formatting request, mislabeled and double-counted. Corrected: one formatting defect and one scope violation with two items.
• Run-04 had one missing-requirement defect, not two; its total is 12. Task #3 had three requests: the Maven Wrapper scope violation, a formatting fix and a test-inventory fix. The evaluator's "missing requirement" at 20:01:48 UTC is the inventory request counted a second time. Task #5's request is genuine: the plan requires the Update action to refresh the Administration view after a successful close, and CCA's first version didn't.

Method

All three runs were scored by shepherd-task-evaluator 0.4.4 at commit f76ebf2, from a clean worktree, on October 4.

• Sources: each run's findings.json, defects.csv, summary.csv and unclassified.md. Treatment runs use the combined output, which merges the two campaign directories into one record.
• Manual adjudication: two unclassified stage-30 change requests had headings the evaluator didn't recognize ("Scope violation: four disallowed Maven wrapper files" in the control, "Focused diff gate failure: unrelated Maven Wrapper files" in run-04). I counted both as scope violations. I also removed two evaluator defects that were other requests counted a second time: control #2's "missing requirement" (its formatting request) and run-04 #3's (its test-inventory request). Comparing with the runs' post-mortems led to three more corrections: run-04 issue #3's two review findings are counted as formatting rather than behavioral (based on the post-mortem's description, not yet checked against PR #8's comments), review rounds come from the post-mortems' per-task tables, and run-04's task #5 stop counts as a failed attempt. Every other count is the evaluator's own.
• Tokens: the last cumulative OTEL export per metric series. Input tokens include cache reads and writes.
• Time to fix: one treatment defect (run-04, #5) has a fix timestamped 57 seconds before its detection. It is flagged as an anomaly and left out of timing claims.
• Raw transcripts: I checked treatment run-03 against its raw transcripts in an earlier review. For the control and run-04, I checked the evaluator's outputs plus the transcript passages and git history cited under Resolved checks.