Campaign Evidence Matrix

Campaign: 1-trick-out-01-remove-before-merge

This document records what the campaign actually demonstrates about the ten boring reasons from the talk abstract. It begins with hypotheses and Not exercised classifications. Implementation issues must replace those initial values only when durable campaign evidence exists.

The ignorance-reduction plan and generated implementation issues define the mandatory update timing and completion gates. In particular, the current issue's evidence must be recorded and merged before the next serial issue begins.

Field definitions

Update rules

1. Preserve the reason numbering and ordering.
2. Treat the failure modes and mechanisms below as initial hypotheses, not evidence.
3. Update every applicable summary row after implementation, validation, and review-feedback resolution for an issue.
4. Identify implementation work with exact issue and PR numbers.
5. Link or name durable artifacts precisely: commit SHA, GitHub Actions run and job, check name, repository-relative artifact path, log, trace, profile, screenshot, or post-mortem section.
6. Distinguish a mechanism being installed from being executed, detecting a problem, preventing a problem, or materially improving the work.
7. Record silent or negative results. If the work did not exercise a reason, leave or set its confidence to Not exercised and explain the non-event.
8. Do not invent evidence or upgrade confidence because a tool merely passed.
9. Do not delete earlier evidence. Update the summary and append a dated issue-specific entry to the evidence log.
10. Put reusable implementation guidance in campaign-lessons.md, not here.

Summary matrix

Issue-specific evidence log

Append one subsection for every implementation issue, even when it produces no meaningful evidence. Keep entries in serial issue order.

Issue #2: Establish the Open Liberty-only baseline

• PR: #9
• Implementation commit: bc6c3e387ccf5b3cab5e14b5ab4d91d512ef426b (pre-merge; do not treat as a future merge commit)
• Completed: 2026-09-30 UTC
• Reasons expected to be exercised: 2, 3, 5, 6
• Reasons actually exercised: 2, 3, 5, 6
• Implementation result: Removed Payara/Cargo/GlassFish runtime paths and guidance; retained Open Liberty as the sole active-by-default profile to preserve the resolved Maven validation tiers. Preserved Java 17, Java EE 7, javax.*, WAR packaging, and cargo-tracker.war.
• Observed events:
  • Spotless passed; the clean package passed all 28 tests (including four managed Open Liberty tests).
  • The production-WAR lifecycle deployed the WAR, started Liberty with the 90-second bound, observed server/application readiness, and received HTTP 200 application/json containing seeded ABC123; the EXIT cleanup invoked liberty:stop successfully.
  • The canonical profile-excluded compile and 24-test unit tiers passed; the explicit Liberty integration tier passed four tests; canonical package produced the WAR without a Liberty runtime directory.
  • A temporary unsupported-server fixture was found by the same search that returned no unsupported-server guidance in the demo.
  • Main Build push run #36754101720, attempt 2, passed for implementation commit bc6c3e387ccf5b3cab5e14b5ab4d91d512ef426b: formatting job/check 110022167006 and build job/check 110022310098 both succeeded. The run has no artifacts.
  • The evidence-matrix update commit bb9a8f033ac14935a235cd0710aed95efd1014d0 also passed Main Build push run #36754242738, attempt 2: formatting job/check 110022156492 and build job/check 110022342235 both succeeded. This run has no artifacts.
• Durable artifacts:
  • Local deployable WAR identity: demo/target/cargo-tracker.war, 8,232,335 bytes, SHA-256 93b8fc97ab3b8afcd44b1a062cbada335af79f5a764865a2e1fe2ed47e0c5419. This was produced and deployed locally; it is not represented as a hosted workflow artifact.
  • Supporting configuration/source: demo/pom.xml, demo/src/main/liberty/config/server.xml, demo/src/test/resources/arquillian.xml, and demo/src/main/java/org/eclipse/cargotracker/interfaces/booking/socket/RealtimeCargoTrackingService.java.
  • Hosted CI evidence: implementation push run #29 / ID 36754101720 on commit bc6c3e387ccf5b3cab5e14b5ab4d91d512ef426b, with successful formatting job/check ID 110022167006 and build job/check ID 110022310098; current evidence update push run #31 / ID 36754242738 on commit bb9a8f033ac14935a235cd0710aed95efd1014d0, with successful formatting job/check ID 110022156492 and build job/check ID 110022342235. Both runs have no artifacts.
• Evidence assessment: Strong for hosted formatting/build and local unit/integration/runtime lifecycle. The hosted workflow verifies formatting and Maven build; the production HTTP readiness assertion and guaranteed Liberty shutdown were verified locally, not by hosted acceptance CI.
• Candidate reusable lessons: Keep the openliberty profile excludable while canonical fast tiers use -P!openliberty; flattening the plugin into the main build would change those commands.

Issue #3: Make CI authoritative and establish the Maven/dependency foundation

• PR: #10
• Implementation commit: c98096d70caad040bd8c3613a630c7c064cc13e2
• Merged commit: ff40701729cbfac46e2bcb6cd17fcd14a5dfeff0
• Completed: 2026-09-30 UTC
• Reasons expected to be exercised: 5, 6
• Reasons actually exercised: 5, 6
• Implementation result: Established serial formatting-first CI, Maven Enforcer governance, reproducible dependency inventories, advisory delta gating, negative controls, and immutable build/dependency artifacts.
• Observed events:
  • Successful Main Build PR run 36792676155, attempt 1, validated synthetic merge SHA 37099baee55ca89517a83a3a555b436ffe1f4387 for implementation HEAD c98096d70caad040bd8c3613a630c7c064cc13e2.
  • PR formatting job/check 110148953240 and build job/check 110149060803 passed in serial order.
  • PR #10 merged as primary merge SHA ff40701729cbfac46e2bcb6cd17fcd14a5dfeff0.
  • Successful Main Build push run 36793581535, attempt 1, validated that exact SHA on edburns/dd-3016202-cargotracker-devoxx-be-2026-experiment.
  • formatting job/check 110151794413 and build job/check 110151881020 passed in serial order.
  • Required negative controls rejected Java range, Maven range, unversioned plugin, duplicate dependency, banned dependency, unauthorized repository, dependency convergence, malformed formatting, corrupted WAR checksum, and known-vulnerable Log4j advisory fixtures.
  • Current and baseline inventories each contained 108 resolved coordinates; the full-coordinate delta was empty and no matching new HIGH/CRITICAL advisories were returned.
• Durable artifacts:
  • PR build-contract artifact ID 11133305135, URL, digest sha256:d3219a3cdb329839b8aadc1229132c4402b4b24917be45d4db8c8f0e4b53aec1.
  • PR dependency-reports artifact ID 11132736102, URL, digest sha256:70ef51f9e344ea1c1d7e77dd7f2ae07bb0a06bdfb4e01ca2fcb00388f70ee690.
  • build-contract artifact ID 11132333582, URL, digest sha256:62a97f70663473867dc7f4646b79e0632da775ddfe82dfa1a01df8682070fc72; supporting files enforcer-negative-controls.txt, war-inventory.txt, war.sha256, and artifact-metadata.json.
  • dependency-reports artifact ID 11132333587, URL, digest sha256:7b1f547c05280d8c5b270b38e00c95192112514bab64de41d780fe1e8bd8e7aa; supporting files effective-pom.xml, dependency-tree.txt, resolved-plugins.txt, vulnerability-report.json, vulnerability-report.txt, and artifact-metadata.json.
• Evidence assessment: Strong for both the final PR synthetic-merge validation and the authoritative exact-SHA experiment-branch push run with immutable artifacts.
• Candidate reusable lessons: Keep formatting first, retain the historical Spotless ratchet, and record synthetic-merge SHA separately from implementation HEAD.

Issue #4: Enforce the Java 17 and Java EE 7 compatibility contract

• PR: #12
• Implementation commit: a3bee8d24ec54e6b3587ccf0cec443969d0609bd (validated implementation HEAD)
• Completed: 2026-10-01 UTC
• Reasons expected to be exercised: 1, 3, 5
• Reasons actually exercised: 1, 3, 5
• Implementation result: Added focused POM, Liberty, and production-source assertions; isolated negative fixtures; repository agent instructions; and a single deployed Open Liberty acceptance lifecycle with bounded evidence.
• Observed events:
  • Local compatibility validation passed for compiler release 17, javax:javaee-api:7.0 in provided scope, WAR packaging/final name, javaee-7.0, WAR location, and /cargo-tracker.
  • Negative fixtures rejected a Jakarta import, Jakarta dependency, release 21, JAR packaging, renamed WAR, Spring dependency, and removed javaee-7.0, each naming the violated boundary.
  • The packaged WAR deployed and returned seeded ABC123 from /cargo-tracker/rest/cargo; the acceptance script captured deploy/start/readiness/stop evidence and stopped Liberty cleanly.
  • Hosted Main Build run #36801557821 passed on validated implementation HEAD a3bee8d24ec54e6b3587ccf0cec443969d0609bd; formatting job/check 110176804872 and build job/check 110176909417 both succeeded.
  • Direct Jakarta and Spring dependencies were rejected from the project-level dependency list, and compatibility metadata passed required-field, file-size, and SHA-256 verification.
• Durable artifacts:
  • demo/scripts/ci/verify-compatibility-contract.sh
  • demo/scripts/ci/run-openliberty-acceptance.sh
  • demo/ci-artifacts/compatibility-contract/compatibility-report.txt, negative-controls.txt, liberty-deploy.log, liberty-start.log, liberty-stop.log, readiness.json from local validation
  • demo/ci-artifacts/compatibility-contract/liberty-messages-excerpt.txt and compatibility-specific artifact-metadata.json
  • .github/workflows/main.yml compatibility-contract upload with 90-day retention
• Hosted artifact: compatibility-contract artifact 11135847970, digest sha256:1de927b096f1ceaef7c1a3aae9c9f1bb2cafcbcc9ee53b5b3ae4ef1324f2018f.
• Evidence assessment: Strong for the validated implementation HEAD hosted contract, negative controls, artifact evidence, and deployed runtime lifecycle.
• Candidate reusable lessons: Keep repository assertions narrow and let the packaged WAR, rather than a managed test-only deployment, define the Open Liberty acceptance boundary.

Issue #5: Strengthen formatting, compiler, type, and static-analysis gates

• PR: #13
• Implementation commit: 64e2cbd
• Completed: 2026-10-01 UTC
• Reasons expected to be exercised: 1, 4, 6
• Reasons actually exercised: 1, 4, 6
• Implementation result: Preserved the Spotless ratchet, enabled -Xlint:all -Werror for main and test compilation, fixed the 14 compiler warnings, and added the focused SpotBugs 4.10.4 gate with a 90-day source-gates artifact.
• Observed events:
  • Local formatting passed after applying the historical ratchet; the malformed formatting fixture failed with Spotless's spotless:apply remediation.
  • Main and test compilation passed independently with Java 17 and -Werror; the invented API fixture failed with a file/line cannot find symbol diagnostic. Hosted source-gates job/check 110185685393 passed on tested SHA b19c8c9f2a67822ff5c3abc4a7de888dee353e79.
  • SpotBugs passed with zero selected production findings after fixing five shared SimpleDateFormat hazards, the permanently null booking result, and the unwritten route field. The null-dereference fixture failed as priority 1 NP_ALWAYS_NULL with a source line.
  • Source-gate commands disabled the active Open Liberty profile and did not run deployment, readiness, or acceptance lifecycle steps. The successful hosted formatting, source-gates, and build jobs were independently attributable: formatting 110185574947, source-gates 110185685393, and build 110186078979, all on tested SHA b19c8c9f2a67822ff5c3abc4a7de888dee353e79.
• Durable artifacts:
  • demo/pom.xml, demo/config/spotbugs-exclude.xml, and demo/scripts/ci/verify-source-gates.sh
  • demo/ci-artifacts/source-gates/ generated by the source-gates job, including formatting-negative.log, compiler.log, test-compiler.log, compiler-negative.log, spotbugs.xml, spotbugs.tsv, analyzer-negative.log, durations.tsv, and artifact-metadata.json
  • Hosted source-gates artifact 11136618790, digest sha256:fdfd048be3d64bde3462a70ab6a289cb4abf49f8ada206abae821265e636942e; successful Main Build run #36804048610 tested SHA b19c8c9f2a67822ff5c3abc4a7de888dee353e79
  • .github/workflows/main.yml formatting job/check 110185574947, source-gates job/check 110185685393, and build job/check 110186078979
  • Evidence assessment: Strong for hosted formatting, compiler, test-compiler, SpotBugs, and controlled-failure evidence on tested SHA b19c8c9f2a67822ff5c3abc4a7de888dee353e79; the local transcript demonstrates the same mechanisms independently.
• Candidate reusable lessons: Keep the compiler and analyzer source gates profile-excluded so they cannot repeat the Open Liberty acceptance lifecycle; keep future SpotBugs suppressions class-and-pattern-specific.

Issue #7: Add CI observability and diagnostic artifacts

• PR: #16
• Implementation commit: 87d6768d9eed2a29efee6357107f383a00c0ab5e
• Completed: 2026-10-01 UTC
• Reasons expected to be exercised: 8
• Reasons actually exercised: 8
• Implementation result: Added pinned local OpenTelemetry Java-agent and Collector instrumentation to the existing bounded Liberty acceptance lifecycle, with correlated trace/metric exports, artifact redaction gates, and immutable diagnostic artifacts.
• Observed events:
  • Main Build run 36839380274 succeeded on final reviewed SHA 87d6768d9eed2a29efee6357107f383a00c0ab5e; the build job 110295058827 completed observability acceptance and all negative controls.
  • Both fixed-ID requests were correlated across request-transcript.jsonl, server spans in traces.json, and observability-access.log; metrics.json contained nonempty JVM/runtime metrics.
  • Negative controls explicitly rejected collector unavailability, incompatible instrumentation, missing telemetry, broken correlation, an unsafe query-bearing exemplar with seeded cargo data, secret-like artifact content, and an invalid request without a diagnostic signal. The final artifact scan passed before metadata and artifact upload.
  • The workflow uploaded otel-telemetry and liberty-logs as separate immutable artifacts with schema-1 metadata and inventories.
• Durable artifacts:
  • Successful Main Build run 36839380274, final reviewed SHA 87d6768d9eed2a29efee6357107f383a00c0ab5e, build job 110295058827.
  • otel-telemetry artifact 11151111829, digest sha256:e5e180578fb5f31f8fe53a9a96f371bb1ddd86c697f68b3d3016842878dfbf21; supporting files traces.json, metrics.json, request-transcript.jsonl, artifact-metadata.json, and redaction-check.txt.
  • liberty-logs artifact 11151376069, digest sha256:928e88ef86f80ae017256cf6d9e4b1662238a668e0722c9d9799d745bae9c3a6; supporting files observability-access.log, messages.log, and artifact-metadata.json.
  • The uploaded compatibility-contract artifact from the same run contains demo/ci-artifacts/compatibility-contract/observability-negative-controls.txt.
  • Implementation sources: demo/observability/otel-collector-config.yaml, demo/scripts/ci/verify-observability.py, demo/scripts/ci/redact-artifacts.sh, demo/scripts/ci/run-observability-check.sh, and demo/scripts/ci/run-observability-negative-controls.sh.
• Evidence assessment: Strong: exact-SHA hosted CI passed the full relevant lifecycle and diagnostics, checked final redaction, and published artifacts with recorded IDs and digests. This demonstrates the configured diagnostic path and its exercised failure controls, not external telemetry services or production-wide observability.
• Candidate reusable lessons: Keep telemetry local and artifact-bound; correlate requests with fixed IDs across transcript, spans, and filtered runtime logs, and enforce redaction at both Collector and final-upload boundaries.

Issue #8: Add bounded JVM performance and jaz evidence

• PR: #17
• Implementation HEAD: 3e141c26cfbd6e5469aeebd2b627b620b28ab725 (validated pre-merge PR head)
• Hosted PR merge-test SHA: 8ce8b6e0bfaaa8c40d5951ffd7e1c7d5d867bd14 (artifact metadata; distinct from the implementation HEAD)
• Completed: 2026-10-01 UTC
• Reasons expected to be exercised: 9
• Reasons actually exercised: 8 (additional JVM/process diagnostics) and 9
• Implementation result: Added the shared non-containerized workload and performed the five-cycle direct Java, bypassed jaz, and tuned jaz comparison with JVM/process/JFR/GC diagnostics and immutable evidence artifacts. Hosted measurements are diagnostic evidence, not a performance ranking.
• Observed events:
  • Successful Main Build run 36918453972, attempt 1, tested implementation HEAD 3e141c26cfbd6e5469aeebd2b627b620b28ab725. The PR merge-test SHA in artifact metadata is 8ce8b6e0bfaaa8c40d5951ffd7e1c7d5d867bd14; it is not the implementation HEAD.
  • The formatting job 110558201739, source-gates job 110558380614, and build job 110558962626 succeeded. The build ran all 15 repetitions, verified every retained JFR, generated checksummed metadata, passed final performance redaction, and uploaded all four performance artifacts.
  • performance-comparison/status.txt reports all 15 launcher-mode repetitions passed functional, diagnostic, resource, and cleanup gates; run-order.tsv records the complete required five-cycle order. mode-summary.tsv and paired-comparison.tsv preserve descriptive results; no winner is selected from hosted-runner variation.
  • Supporting per-mode evidence includes artifact-metadata.json, per-run GC logs, JFR summaries, selected JVM flags, and, for tuned jaz, the JAZ_DRY_RUN=1 output. The logs verify launcher modes, workload equivalence, JVM diagnostics, and cleanup.
  • Stage 40 Copilot review identified that the JFC transformation changed an event attribute instead of its child enabled setting. The correction unconditionally disables all five sensitive event types and requires a zero-count check on every recording before upload; see review thread.
  • A follow-up review found that resource sampling began only after readiness. The correction distinguishes Liberty status helpers from the server JVM and samples the unique server PID during startup while retaining strict identity checks; see review thread.
  • The next review found that artifact metadata was generated after the upload redaction boundary and that post-run gate failures retained a zero exit status. Upload now requires a final scan of the complete artifact directories, and summaries set exitStatus to 1 when post-run gates fail; see review thread.
  • A further review found that failed runs could retain unverified recordings and identified related measurement-integrity gaps. The workflow now rejects any retained JFR with sensitive events; summaries select the server-PID GC log; direct/bypass initial and maximum heap evidence is required and compared; compiler-count tuning is rejected; each repetition has a cleanup-reserving watchdog; and immutable inputs are rehashed before every launch. See review thread.
  • The subsequent zero-finding review surfaced two missed no-user-tuning cases. The harness now also rejects -Xint, -Xcomp, and tuning inside quoted server.env assignments before launch; see review.
  • The final implementation review expanded compiler-override rejection and found persistent Derby state outside the Liberty server snapshot. The validated harness rejects batch/tiered/background/threshold compiler controls, restores and hashes an empty database baseline before every launch, and preserves any pre-existing local database; see review thread.
  • The final remediation rejects active Liberty before touching Derby state and rejects String Deduplication from every JVM option source; see database preflight and tuning review threads.
• Durable artifacts:
  • Main Build run 36918453972, attempt 1, implementation HEAD 3e141c26cfbd6e5469aeebd2b627b620b28ab725; merge-test SHA 8ce8b6e0bfaaa8c40d5951ffd7e1c7d5d867bd14; jobs formatting 110558201739, source-gates 110558380614, and build 110558962626.
  • performance-java artifact 11191254069, digest sha256:b670c2f03802f8589ab78c900d268225e0ca1d715de0ea96de20644241a65232; supporting artifact-metadata.json, per-run GC logs, JFR summaries, and selected JVM flags.
  • performance-jaz-bypassed artifact 11190869300, digest sha256:9cdedb105579884724484721a7e5a4ad7de8a604ad9916e0ed99190fcd869023; supporting artifact-metadata.json, per-run GC logs, JFR summaries, and selected JVM flags.
  • performance-jaz-tuned artifact 11191029124, digest sha256:d5e8568c611aa17d308c3457727264183372d88eb63ea7cc9c25e8d68eb8d662; supporting artifact-metadata.json, per-run GC logs, JFR summaries, selected JVM flags, and tuned JAZ_DRY_RUN=1 output.
  • performance-comparison artifact 11191258963, digest sha256:a0a221a61cbae90e63a5c792e70cc723817920ba64d4c5fd8551651d959d1461; supporting artifact-metadata.json, status.txt, run-order.tsv, mode-summary.tsv, and paired-comparison.tsv.
  • Implementation and workload contract: demo/performance/README.md and demo/performance/.
• Evidence assessment: Strong for the bounded hosted comparison, functional equivalence, diagnostics, resource envelope, and cleanup across all 15 launches, with immutable artifacts and checksummed inventories. It supports launcher integration and descriptive measurement only; it does not establish a performance winner or AKS-limit behavior.
• Candidate reusable lessons: Treat hosted JVM measurements as diagnostic evidence; retain per-run workload and launch metadata, and do not rank modes based on noisy runner measurements.

Use this template:

### Issue #<number>: <title>

- **PR:** #<number>
- **Merged commit:** `<full SHA>`
- **Completed:** `<UTC timestamp>`
- **Reasons expected to be exercised:** <reason numbers>
- **Reasons actually exercised:** <reason numbers or `None`>
- **Implementation result:** <concise factual summary>
- **Observed events:**
  - <what happened, including failures, corrections, and meaningful non-events>
- **Durable artifacts:**
  - <exact run URL, job/check name, repository-relative path, log, trace,
    profile, screenshot, review thread, or post-mortem section>
- **Evidence assessment:** <why the resulting confidence is Strong, Moderate,
  Weak, Unsupported, or Not exercised>
- **Candidate reusable lessons:** <candidate guidance, or `None`; validated
  reusable guidance belongs in campaign-lessons.md>

No implementation issue is complete until its subsection has been appended and the corresponding summary rows have been updated.

Fixture baseline repair: extensible BookingServiceTest CI gate

• Implementation commit: 94950cfceefc85edaec68eb6b61a16e0e6779ecc
• Completed: 2026-10-02 UTC
• Reason exercised: 2
• Implementation result: Replaced the exact BookingServiceTest count of four with a contract that requires the historical four named methods, requires zero failures, errors, and skipped tests, and permits additional valid tests.
• Observed events:
  • The exact local Open Liberty integration tier passed four tests with zero failures, errors, or skipped tests.
  • The revised assertion passed the real four-test Surefire report.
  • A derived five-test report containing testChangeDeadline passed the same assertion, proving the gate no longer rejects the fixture's required fifth test merely because the suite grew.
  • Hosted Main Build passed formatting, source gates, the full build and acceptance pipeline, and artifact publication on the exact implementation SHA.
• Durable artifacts:
  • Main Build run #36956818505; formatting job/check 110681507992, source-gates job/check 110681594814, and build job/check 110681940309.
  • test-reports-liberty artifact 11206483198, digest sha256:19b0c9c08136763bbb8f947581e35680c6770717d0a82cf7a89311578ea4af53.
• Evidence assessment: Strong exact-SHA hosted evidence plus direct local proof that both the feature-free four-test suite and the planned five-test suite satisfy the repaired contract.
• Candidate reusable lesson: Prefer named required tests and zero-failure/error/skip invariants over a brittle exact suite cardinality when a planned feature intentionally adds tests.

Appendix I: Human observations

• While filling out the ignorance reduction plan, I was frequently guided toward narrowing the possible paths the agents could go. See "1.6 — Executable Java 17 and Java EE 7 compatibility contract". Without narrowing it down, the agents might want to upgrade the system forward: javax → jakarta for example. By explicitly disallowing that, we close down that possible wandering path.