Supporting data: findings

result. Tag: recomputed.

Finding: In the planned rep-1 contrast all 5 Anchor Engineering replicates reached adjusted mutation 1.000 against a median of 0.984 for Classic greedy TDD, exact Mann-Whitney U 25 and Holm p 0.016 over the two contrasts. Functions first, classes later and Noun-verb map did not differ, U 16 and Holm p 0.548. The CQP sensitivity check separated neither pair, Holm p 0.333.

Evidence: runs/rep-1/contrast.json contrasts and sensitivity, from harness/report/contrast.py.

result. Tag: recomputed.

Finding: In the planned rep-2 contrasts no survey leader differed from Classic greedy TDD on M, exact Mann-Whitney U 11 for Held-out TDD, 9 for Coarse-to-fine TDD and 17 for Most informative next test, each with Holm p 1.000 over the three contrasts. On adjusted mutation score, the fallback outcome, Held-out TDD scored lower than Classic greedy TDD, a median of 0.777 against 1.000, U 0 and Holm p 0.024. The other two leaders gave Holm p 1.000.

Evidence: runs/rep-2/contrast.json primary and secondary, from harness/report/contrast.py.

result. Tag: recomputed.

Finding: Over the seven replicated methods of rep-1 and rep-2 consistency score C runs from 0.38 for mutation-guided to 1.65 for noun-verb, and a permutation test of its spread across methods gives p 0.464. With 5 to 10 replicates each, no method is shown more consistent than another.

Evidence: replicate.consistency over runs/rep-1 and runs/rep-2 pooled at seed 20261024 with 10000 draws.

result. Tag: recomputed.

Finding: Three blind raters scored each final on the eight cqp.html principles. Their agreement over the 11 finals is accepted for Simple Constructs, Minimal Duplication and Modular Structure and tentative for Problem Alignment. The other four stay out of the headline. Every median of those four principles is 3, 4 or 5. Classic alone has median 3 on Simple Constructs. Modular Structure runs from 3 for annealed, classic and noun-verb to 5 for coarse-to-fine and held-out. Minimal Duplication is accepted on its alpha of 0.88, though its 95% interval runs from -0.03 to 1.00. Simple Constructs is accepted on its alpha of 1.00, which rests on that one varying item. Every rater gave classic 3 and every other final 4, so 34.7% of resamples hold no variation and its interval is not estimable. Alterability is not measured. The rating leaves it out and the single review that forecast it is superseded.

Evidence: rating.json of each final clone and runs/full-1/agreement.json, which agree.py recomputes from the three attempts per final.

result. Tag: recomputed.

Finding: Cache reads are 90.1 to 95.8 percent of each method's tokens. Weighted tokens count a cache read at 0.05, so they span 760686 to 2126219 while raw tokens span 4795045 to 19389975.

Evidence: costs.json of every clone summed per method.

result. Tag: unchecked.

Finding: LOG.md rows that mention a conflict number 9 in held-out, 7 in classic, 6 in tpp, 5 in coarse-to-fine, 4 each in annealed and noun-verb, 2 each in mapped-path, mutation-guided and property-based, 1 in multi-start and in each start clone and 0 in functions-first. Classic's 7 rows each log sed or a script used to keep trailing spaces on spec prompt lines against the no-script edit rule.

Evidence: Case-insensitive grep -c conflict over each clone's LOG.md. Rows outside classic were not read.

result. Tag: recomputed.

Finding: The first of five bench runs exceeds the median by 291 to 468 ms in 12 clones and the reference, and by 68 and 16 ms in tpp and multi-start-2. Those two are the only clones whose src calls no String.format or formatted. Their medians of 205 and 192 ms are the lowest. The other clones and the reference have medians of 500 to 611 ms.

Evidence: run_ms and median_ms in each bench. git grep of String.format and .formatted( over src on each method branch.

result. Tag: unchecked.

Finding: Warming String.format with 200000 calls before the timed runs cut the first-run excess from 288 to 465 ms to 37 to 149 ms in classic, mapped-path and the reference. tpp and multi-start-2 stayed between -25 and 38 ms.

Evidence: A one-off trace bench outside the harness with three repeats per clone. Its logs are not in this repo. Its tpp and mapped-path runs predate the reader fix and play 99610 inputs.

result. Tag: verified.

Finding: tpp and mapped-path parse the typed house with Integer.parseInt and no range check, so a non-number or a house off the board throws. The other 12 clones read through IO.readInteger with bounds. No example spec sends such input, both clones pass every hidden and unseen test and the README does not say what invalid input should do.

Evidence: src/kalah/Kalah.java:27 on method/tpp. src/kalah/Command.java:22 and Board.java:26 on method/mapped-path. The example specs send only 1 to 6 and q.

caveat. Tag: recomputed.

Finding: Output tokens per request rose from 148 in full-1 to 605 in rep-1, with an agent CLI change from 2.1.295 to 2.1.308 between the batches and identical prompts and launch scripts. So weighted-token costs of full-1 and rep-1 do not compare. rep-2 ran on CLI 2.1.309 at 495 output tokens per request.

Evidence: tokens.output over requests summed over every costs.json of each run. The full-1 and rep-1 CLI versions come from one agent transcript of each batch and the rep-2 one from its contrast.json runtime.

caveat. Tag: verified.

Finding: Each survey method ran once. No significance test applies and a gap between two survey methods is a single observation.

Evidence: args.json lists 11 methods. slugs.tsv adds only the three multi-start start clones.

caveat. Tag: recomputed.

Finding: Mutation denominators run from 111 to 149 because each method's code and tests differ, so mutation scores compare loosely. The reference's 154/165 comes from its own TestKalah tests from the hidden jar.

Evidence: mutation in each final clone's metrics.json and in runs/reference/metrics.json.

caveat. Tag: unchecked.

Finding: tpp and mapped-path were benched again in a later session than the other clones, where a classic control timed 591 ms against its collected 605 ms.

Evidence: Commit fc1a647 holds the two new bench objects. The control run's log is not in this repo.

measurement. Tag: verified.

Finding: Two agent calls stopped at the 15-cycle cap, tpp:L2-7#1 and mapped-path:L3-8#1.

Evidence: cap_hits in batch.json, listed in RUN.md and in the Run section of report.html.

measurement. Tag: verified.

Finding: Bench.java's readFromKeyboard drew a house with nextInt while readInteger drew with nextDouble. tpp and mapped-path read through readFromKeyboard and so played other games of 99610 inputs. Both readers now draw alike.

Evidence: harness/bench/Bench.java RandomIO and its guard benchReadersDrawAlike in docker_bench.sh. The pre-fix bench objects are in runs/full-1/tpp/metrics.json and runs/full-1/mapped-path/metrics.json at commit c4fc77c.

measurement. Tag: recomputed.

Finding: All 14 clones and the reference play 100614 inputs and print 40083576 characters, so every bench median times the same games.

Evidence: bench in each clone's metrics.json and in runs/reference/metrics.json.

data note. Tag: verified.

Finding: Classic's step log names the UNITS.md tick commit for steps 2, 6, 7, 8 and 9 while its MEASURE.md names the feature commit before it, so 5 of 9 step shas point at a 2-line UNITS.md change.

Evidence: steps.csv and MEASURE.md of classic against git log of method/classic.

data note. Tag: verified.

Finding: Commit times on the method branches carry UTC+13 while step logs and run times are UTC.

Evidence: Classic commit d3428d8 reads 2026-10-16 12:53:37 +1300 and its steps.csv row reads 2026-10-15T23:53:37Z.