12 ways to test-drive Kalah

The study

Classic greedy TDD writes the smallest failing test and then the smallest change that passes it. Do TDD variants that plan the class structure first, change the step size, try several starts or change how the tests check the code lead an AI coding agent to a better Kalah solution?

Run ended 2026-10-16. Model Claude Opus 5.5 (claude-opus-5-5). Runs: full-1 survey, tcp-1-list2 list reliability, rep-1 planned contrasts, rep-2 planned contrasts. 11 survey methods. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

The methods text on these pages is from methods doc 12f026f6. The runs this page names were built under these doc versions, by short sha256: full-1 64606950, tcp-1-list2 12f026f6, rep-1 12f026f6, rep-2 12f026f6, tcp-1 1f11dd37.

Read on with Experimental design and approach, Methodology: harness and protocol and Results: survey (full-1), or pick any of the 29 parts.

Awards

The first five awards rank the 11 full-1 survey builds, lower first, with ties sharing a place. The consistency badge covers seven replicated designs in rep-1 and rep-2. The full-1 survey ended 2026-10-16. A medal only styles a place and carries no score. M and M_io read curve.json, E sums the distinct PMD findings, adjusted surviving mutants, hidden failures and unseen failures, and alignment needs both the blind CQP rating agreeing with PMD and PMD reporting no issue. recomputed

The designs

One card per method in run order. Each gives the method's family and idea from the methods doc, its final clone's class diagram and its headline scores. A contrast arm with no survey run shows its representative replicate. The A02 reference comes last and apart.

Here each card gives a design's headline scores. Its page holds the full method card and class diagram.

Headline

The study has a survey and two contrast runs. A survey ran 11 methods once each in run full-1. Planned contrasts then ran 5 replicates each of four methods in run rep-1 and in run rep-2 of three new methods plus Classic greedy TDD, seven methods in all. Every build reached the functional ceiling of 15/15. In rep-1 Anchor Engineering scored a higher adjusted mutation score than Classic greedy TDD, all 5 replicates at 1.000 against a median of 0.984, with Holm p 0.016. The test did not detect a difference between Functions first, classes later and Noun-verb map (Holm p 0.548). In rep-2 the M tests did not detect a difference from Classic greedy TDD. Held-out TDD scored lower on adjusted mutation (Holm p 0.024) in this task and batch. recomputed

Key findings

All 11 methods score the functional mark 15/15, hidden 76/76, unseen 69/69 and acceptance 6/6. Only functions-first, mapped-path and noun-verb pass strict PMD compliance, as the reference does. The other 8 fail with 2 to 10 violations. The methods also differ in size, structure, mutation, cost, time, bench and rated code quality. recomputed

• The 3 methods with no violation used 1.0 to 2.1 million weighted tokens, the other 8 used 0.8 to 1.3 million. recomputed PMD violations against weighted tokens
• The 3 methods that pass strict PMD compliance have 6, 14 and 16 classes, and the other 8 have at most 5. recomputed PMD violations against classes
• Mutation scores run from 75% (held-out) to 95% (mapped-path). recomputed Mutation score per method
• Bench medians run from 205 ms (tpp) to 606 ms (held-out). recomputed Bench run times per method
• Medians over each method's replicates: Functions first, classes later 0.927, Noun-verb map 0.964, Anchor Engineering 1.000, Classic greedy TDD 0.984. recomputed Adjusted mutation per replicate, rep-1
• Medians over each method's replicates: Functions first, classes later 6, Noun-verb map 15, Anchor Engineering 6, Classic greedy TDD 2. recomputed Classes per replicate, rep-1
• Medians over each method's replicates: Functions first, classes later 1,339,770, Noun-verb map 1,334,836, Anchor Engineering 1,039,535, Classic greedy TDD 1,033,995. recomputed Cost to completion per replicate, rep-1
• Medians over each method's replicates: Functions first, classes later 15, Noun-verb map 16, Anchor Engineering 16, Classic greedy TDD 14. recomputed CQP per replicate, rep-1
• Medians over each method's replicates: Held-out TDD 0.777, Classic greedy TDD 1.000, Coarse-to-fine TDD 1.000, Most informative next test 1.000. recomputed Adjusted mutation per replicate, rep-2
• Medians over each method's replicates: Held-out TDD 3, Classic greedy TDD 1, Coarse-to-fine TDD 3, Most informative next test 3. recomputed Classes per replicate, rep-2
• Medians over each method's replicates: Held-out TDD 973,901, Classic greedy TDD 1,011,641, Coarse-to-fine TDD 926,033, Most informative next test 1,068,727. recomputed Cost to completion per replicate, rep-2
• Medians over each method's replicates: Held-out TDD 17, Classic greedy TDD 13, Coarse-to-fine TDD 16, Most informative next test 16. recomputed CQP per replicate, rep-2
• Medians over each method's replicates: Held-out TDD 607308, Classic greedy TDD 601555, Coarse-to-fine TDD 559186, Most informative next test 735111. recomputed M per replicate, rep-2
• Exploratory. Consistency scores C: Functions first, classes later 1.14, Noun-verb map 1.65, Anchor Engineering 1.06, Classic greedy TDD 0.87, Held-out TDD 0.96, Coarse-to-fine TDD 1.06, Most informative next test 0.38. The spread of C across methods gives permutation p 0.464. recomputed Consistency score per method
• In the planned rep-1 contrast all 5 Anchor Engineering replicates reached adjusted mutation 1.000 against a median of 0.984 for Classic greedy TDD, exact Mann-Whitney U 25 and Holm p 0.016 over the two contrasts. Functions first, classes later and Noun-verb map did not differ, U 16 and Holm p 0.548. The CQP sensitivity check separated neither pair, Holm p 0.333. recomputed Findings table
• In the planned rep-2 contrasts no survey leader differed from Classic greedy TDD on M, exact Mann-Whitney U 11 for Held-out TDD, 9 for Coarse-to-fine TDD and 17 for Most informative next test, each with Holm p 1.000 over the three contrasts. On adjusted mutation score, the fallback outcome, Held-out TDD scored lower than Classic greedy TDD, a median of 0.777 against 1.000, U 0 and Holm p 0.024. The other two leaders gave Holm p 1.000. recomputed Findings table
• Over the seven replicated methods of rep-1 and rep-2 consistency score C runs from 0.38 for mutation-guided to 1.65 for noun-verb, and a permutation test of its spread across methods gives p 0.464. With 5 to 10 replicates each, no method is shown more consistent than another. recomputed Findings table
• Output tokens per request rose from 148 in full-1 to 605 in rep-1, with an agent CLI change from 2.1.295 to 2.1.308 between the batches and identical prompts and launch scripts. So weighted-token costs of full-1 and rep-1 do not compare. rep-2 ran on CLI 2.1.309 at 495 output tokens per request. recomputed Findings table
• Cache reads are 90.1 to 95.8 percent of each method's tokens. Weighted tokens count a cache read at 0.05, so they span 760686 to 2126219 while raw tokens span 4795045 to 19389975. recomputed Findings table
• tpp and mapped-path parse the typed house with Integer.parseInt and no range check, so a non-number or a house off the board throws. The other 12 clones read through IO.readInteger with bounds. No example spec sends such input, both clones pass every hidden and unseen test and the README does not say what invalid input should do. verified Findings table

About the data

• Awards rank the 11 survey builds of run full-1, one build per design.
• Runs rep-1 and rep-2 made 40 repeat builds of 7 designs. Replicate medians and C come from them, and each design names its own runs and build counts.
• Run tcp-1 is not ranked here. Five arms with one test-list order each for random, coverage, cost, risk and usage at seed 20261020.
• Each run is its own batch, so costs from different runs do not compare.
• The full research report is the full study report.

Terms

Functional mark

Marks out of 15 from the hidden course tests, scored as the A02 rubric does.

E

Errors: distinct PMD findings plus adjusted surviving mutants, those the hidden tests kill, plus failed hidden and unseen tests. Lower is better.

Cost to completion

Weighted tokens the agents spent to finish the build.

M

The trapezoidal checkpoint-area estimate of weighted-token cost per hidden test. Lower is better.

M_io

A sensitivity check that recomputes M without output-token costs.

Alignment

The blind code-quality rating agrees with PMD on every principle and PMD finds no issue.

C

Consistency score over repeat builds of one design. Lower means steadier results. Exploratory.

PMD

A static analysis tool for Java. A finding is one rule broken in one place.

Weighted tokens

Model tokens weighted by price: input 1, cache write 1.25, cache read 0.05, output 5.

APFDc-H

How early the hidden tests pass: the area under the hidden pass curve over weighted tokens spent, divided by the cost to completion.

CQP

Code Quality Principles, the course's design principles. A CQP score sums the blind rating's medians over four of them, 4 to 20.

Adjusted mutation

Surviving mutants missed by the hidden suite are excluded. They may be equivalent or expose gaps in that suite. Higher is better.

Planned contrast

A comparison of two designs fixed before the builds ran, tested over their repeat builds.

U

The exact two-sided Mann-Whitney statistic of a planned contrast: how many pairs of builds, one from each design, have this design's value above the other's, a tie counting half.

Holm

A correction that raises each contrast's p value, the chance of a gap this large with no real difference, so testing several contrasts at once keeps the false alarm rate at its usual level.

The glossary defines every term the study pages use.

Contents

All 29 parts in reading order, grouped as the sidebar groups them. Part numbers match the header of each page.