Critical questions

Q1 How far does the analogy of classic TDD with greedy pathing or gradient descent hold, and do variants that change the map, the step size, the breadth of search or the check reach a better end point?

Status: answered

Classic TDD writes the smallest failing test and then makes the smallest change that passes it, which resembles greedy pathing or gradient descent toward the requirements.

The 10 variants and Classic greedy TDD reached the same functional score in these survey builds: 15/15 with 76/76 hidden and 69/69 unseen tests. Cost, PMD and mutation results differed. No recorded LOG.md checkpoint reduced the hidden-pass count.

The methods differed in how early tests passed per weighted token. At 500000 weighted tokens held-out passed 0.42 of the hidden tests, coarse-to-fine and mutation-guided 0.39, classic 0.29, annealed 0.26 and tpp 0.05. Noun-verb, functions-first, mapped-path, property-based and multi-start passed none.

At 750000 noun-verb, functions-first and mutation-guided passed all 76 tests and classic and held-out 0.68. Mapped-path, property-based and multi-start passed 0.03 or less. In rep-2, a different batch from the survey, the median share passed at 500000 weighted tokens over 5 replicates was 0.00 for held-out, 0.34 for coarse-to-fine, 0.09 for mutation-guided and 0.29 for classic, a descriptive figure with no test.

Multi-start's 3 start clones alone cost 869732 weighted tokens before its final clone began. Later commits deleted 33 of the 147 src lines classic added, a rewrite share of 22 percent. Functions-first rewrote 43 percent, annealed 35 and tpp 34.

The other 7 methods rewrote 11 percent or less, with none for held-out and multi-start. So the small steps of classic did not lead to the largest rewrite. Each survey method ran once on one task, so the survey figures are descriptive and a gap among them may be run noise.

Evidence: runs/full-1/tdd-methodologies.md Summary; runs/full-1/findings.json headline; runs/full-1/*/summary.json; runs/full-1/curve-shapes.tsv p_at columns; runs/full-1/churn.tsv; harness/report/curve.py p_at_cost; harness/report/churn.py; runs/rep-2/contrast.json descriptive p_at_cost. recomputed

Q2 Can each TDD methodology be written as a short numbered procedure that an AI coding agent carries out unattended from start to finish?

Status: answered

Yes for all 11.

Each procedure has 7 or 8 steps and every method reached a final clone with no failed collection step. The harness turned minute timeboxes into cycle caps and throwaway branches into uncommitted trials. Rule-only steps got no agent of their own.

Two calls hit the 15-cycle cap.

Evidence: runs/full-1/tdd-methodologies.md; harness/PLAN.md D4, D7 and A4; runs/full-1/RUN.md; runs/full-1/findings.json cap hits. verified

Q3 Does the hidden-test grader give the known-good A02 course solution full marks, so that a lower mark means a real defect?

Status: answered

The reference passes the grader.

The A02 reference at commit 2ed6c36 passes 76/76 hidden tests with 69/69 unseen and 7/7 seen. This checks compatibility with the reference. It does not establish that every grading failure represents a real defect. The guard harness/tests/docker_grader_reference.sh fails unless all 76 pass. The Reference row of COMPARISON.md leaves its grades as not measured because the reference sits apart from the comparison and is never ranked.

Evidence: harness/tests/docker_grader_reference.sh; runs/full-1/COMPARISON.md Reference section. verified

Q4 Do the methodologies differ in functional correctness when graded on hidden tests the agents never saw?

Status: answered

No.

All 11 methods score 15/15 with 76/76 hidden, 69/69 unseen and 6/6 acceptance tests, so the functional mark does not separate them. The batch holds 11 methods in 14 clones because Multi-start TDD adds 3 start clones. All 20 rep-1 replicates and all 20 rep-2 replicates also pass 76/76 hidden and 69/69 unseen tests.

Evidence: runs/full-1/*/summary.json; runs/full-1/*/grade.tsv; runs/full-1/COMPARISON.md Methods table; runs/full-1/args.json; runs/rep-1/*/summary.json; runs/rep-2/*/summary.json. recomputed

Q5 Were the hidden grading tests kept away from every implementing agent, so that the hidden-test score is a fair measure?

Status: answered

Yes as far as git access goes.

Each clone came from the starter tag with --no-local and then lost its origin remote, so main and its hidden suite were not reachable through git. Protection was prompt plus audit with no settings deny. The 10 high audit flags were 9 on Held-out TDD and 1 on Noun-verb map, and each was judged a false positive. rep-2 raised 50 high flags in 9 of its 20 builds.

46 are on the 5 Held-out TDD replicates, 37 for added code that uses reflection and 9 for _heldout paths. The other 4 are outside-path flags on rep-2-r02, rep-2-r10, rep-2-r13 and rep-2-r09. 49 were judged false positives.

The rep-2-r09 flag is a minor true positive. That build ran ls ../ from its clone, which listed only the names in the clones dir, _heldout among them, and read no file outside its clone.

Evidence: harness/PLAN.md A1, A7 and R9; runs/full-1/COMPARISON.md Audit flag dispositions; runs/full-1/findings.json dispositions; runs/rep-2/*/audit.json; runs/rep-2/findings.json dispositions. verified

Q6 How do the implementations compare in size, class (object) count and method count, and how close does each come to the structure of the A02 reference solution?

Status: answered

Source lines in src/ run from 114 to 409 against 380 for the reference.

Classes run from 1 to 16 against 10 and methods from 9 to 67 against 55. The structure-first methods sit nearest the reference. noun-verb has 14 classes, 60 methods and 378 lines. mapped-path has 16, 67 and 409. functions-first has 6, 45 and 270. The other 8 methods have 1 to 5 classes.

The Lines of code column runs from 181 to 1237 because it adds test code, so it does not compare with the src-only reference figure. The 20 rep-1 replicates keep the class pattern. Noun-verb has 15 to 19 classes, functions-first 5 to 7, anchor-engineering 5 to 8 and classic 1 to 3.

The 20 rep-2 replicates have 1 to 2 classes for classic, 3 to 5 for held-out, 3 to 7 for coarse-to-fine and 3 for mutation-guided.

Evidence: runs/full-1/*/metrics.json; runs/reference/metrics.json; runs/rep-1/*/metrics.json; runs/rep-2/*/metrics.json. recomputed

Q7 Do the implementations differ in runtime speed over a fixed benchmark of seeded games?

Status: answered

The median time for 1000 seeded games is 520 to 611 ms for every clone except tpp at 205 ms and the multi-start-2 start clone at 192 ms.

The reference takes 500 ms. Those two clones are the only sources without String.format. A one-off warm-up check points to String.format warm-up as the cause.

Its logs are not kept, so that mechanism is unchecked.

Evidence: runs/full-1/*/metrics.json bench; runs/reference/metrics.json bench; runs/full-1/findings.json bench entries. recomputed

Q8 What does each methodology cost to carry out in AI tokens and in wall-clock time?

Status: answered

Weighted tokens run from 760686 for mutation-guided to 2126219 for mapped-path, with classic at 843734.

Cache reads are 90.1 to 95.8 percent of raw tokens. The mean weighted-token cost to pass a hidden test runs from M 470362 and M_io 441270 for functions-first to M 1475658 and M_io 1386111 for mapped-path, with M 542369 and M_io 496442 for classic. M_io leaves output tokens out of every cost.

Wall time runs from 0:21:19 for multi-start to 0:54:34 for mapped-path, with classic at 0:22:31. All methods ran in parallel on one machine. In rep-1 the median cost to completion over 5 replicates was 1033995 weighted tokens for classic, 1039535 for anchor-engineering, 1334836 for noun-verb and 1339770 for functions-first.

In rep-2 the median cost to completion was 1011641 for classic, 973901 for held-out, 926033 for coarse-to-fine and 1068727 for mutation-guided. Their median M was 601555, 607308, 559186 and 735111 in the same order. Costs of full-1 and rep-1 do not compare, as the Limits explain.

The drift test of classic between rep-1 and rep-2, outside Holm, gives M p 0.310 and M_io p 0.421.

Evidence: runs/full-1/*/costs.json; runs/full-1/COMPARISON.md Weighted tokens and Wall time columns; runs/full-1/*/curve.json; runs/rep-1/contrast.json cost_to_completion; runs/rep-2/contrast.json descriptive and drift. recomputed

Q9 Did the agents that built each implementation know the code quality principles (cqp.html) and the PMD rules they were later graded on?

Status: answered

No.

The preamble named README.md and docs/SPEC.md as the task and docs/EXPERIMENT.md as the guide to ./run.sh. Those files list cqp.html and kalah_pmd.xml only as guidance. None of the 111 method or measure agents opened cqp.html.

Strict PMD compliance became a graded column only after the batch. Only 7 of 11 reviewers in the default-agent re-review opened cqp.html although all 11 scored its principles, so those review scores are superseded. The blind rating that replaces them puts cqp.html and UsingPMD.html in full, each line numbered, in every rater's prompt.

Evidence: runs/full-1/batch.js line 269; README.md line 106; docs/EXPERIMENT.md line 15; harness/workflow/batch.js reviewPrompt; harness/workflow/rate.js ratePrompt; harness/PLAN.md User requirements; full-1 agent transcripts. recomputed

Q10 Which implementing agents ran the PMD linter during development, and why?

Status: answered

Four did, counted as ./run.sh lint calls in the agent transcripts. mapped-path ran it 68 times, annealed 25 times, functions-first 7 times and noun-verb 5 times.

Each of those methods has a step that names lint or PMD warnings. The preamble told agents to run only the checks a step names, so the other 7 never ran it.

Evidence: runs/full-1/batch.js line 271; runs/full-1/tdd-methodologies.md method steps; docs/EXPERIMENT.md line 27; full-1 agent transcripts. recomputed

Q11 Which implementations meet strict PMD compliance (every priority of kalah_pmd.xml over src/), including the single-class monolithic ones?

Status: answered

Only functions-first, mapped-path and noun-verb have 0 violations with no suppression comments, as does the reference.

The other 8 fail with 2 for annealed up to 10 for classic. Each count is of distinct violations, since PMD reports a long line once per syntax node on it. Both single-class solutions fail, classic with 10 and annealed with 2.

The A02 rubric gives no PMD marks, so this is a separate pass or fail column. In rep-1 all 15 replicates of functions-first, noun-verb and anchor-engineering pass, and all 5 classic replicates fail with 4 to 10 distinct violations. None of the 20 rep-2 replicates passes.

Their distinct violations run from 3 to 12 in held-out, 4 to 7 in classic, 2 to 10 in coarse-to-fine and 2 to 6 in mutation-guided.

Evidence: runs/full-1/*/metrics.json PMD violations; runs/full-1/findings.json headline; A02 rubric; runs/rep-1/*/metrics.json; runs/rep-2/*/metrics.json. recomputed

Q12 Beyond functional correctness, how do the implementations compare in test strength (mutation score) and in reviewed code quality and alterability?

Status: partly answered

Mutation scores run from 75.2 percent for held-out at 112/149 to 94.7 percent for mapped-path at 107/113.

A seeded permutation chi-square over the 11 methods gives chi-square 63.0 and p below 0.001. Fisher exact tests with Holm correction separate 11 of the 55 pairs at 0.05. Each of those pairs includes held-out or noun-verb.

The hidden suite also passes 77 of the 164 surviving mutants. They may be equivalent or expose gaps in that suite. The adjusted score drops them and runs from 79.4 percent for held-out at 112/141 to 100 percent for classic, coarse-to-fine, mutation-guided and property-based. Holm then separates 22 of the 55 pairs.

The tests treat each mutant as independent within one run per method. The mutant counts differ by method, so the scores compare only loosely. The mutation figures are recomputed.

Code quality comes from a blind rating by three raters per final on the eight cqp.html principles. Ordinal alpha over the 11 finals reads, as principle, alpha and status: Clear Presentation 0.62 unchecked, Explanatory Language 0.41 unchecked, Consistent Code not measured unchecked, Used Content 0.58 unchecked, Simple Constructs 1.00 accepted, Minimal Duplication 0.88 accepted, Modular Structure 0.85 accepted and Problem Alignment 0.75 tentative. Consistent Code has no alpha because all 33 of its ratings are 4.

Simple Constructs rests on one varying item. Every rater gave classic 3 and every other final 4, so 34.7% of resamples hold no variation and its interval is not estimable. The headline shows the medians of the accepted and tentative principles.

Each of those medians is 3, 4 or 5. The Methods table shows the medians of all eight principles. The Column definitions tag the unchecked ones unchecked.

The alphas and medians are recomputed. Alterability and test quality are not measured. The rating leaves them out and the single review per method that scored them is superseded.

In rep-1 the median adjusted mutation over 5 replicates is 0.927 for functions-first, 0.964 for noun-verb, 0.984 for classic and 1.000 for anchor-engineering. A separate blind rating of the 20 rep-1 finals accepts Clear Presentation and Problem Alignment and takes Used Content and Modular Structure as tentative. Its median CQP over the four counted principles is 14 for classic, 15 for functions-first and 16 for noun-verb and anchor-engineering.

In rep-2 the median adjusted mutation over 5 replicates is 0.777 for held-out and 1.000 for classic, coarse-to-fine and mutation-guided. The blind rating of the 20 rep-2 finals accepts no principle and takes Minimal Duplication, Modular Structure and Problem Alignment as tentative.

Evidence: runs/full-1/*/metrics.json mutation; runs/full-1/*/survivors.tsv hidden grades of each surviving mutant; harness/grade/survivors.sh; harness/report/stats.py permutation_chi2 and fisher_exact; runs/full-1/*/rating.json medians; runs/full-1/agreement.json finals alpha; runs/rep-1/contrast.json; runs/rep-1/agreement.json finals alpha; runs/rep-2/contrast.json; runs/rep-2/agreement.json finals alpha. recomputed

Q13 Does test-driving pure functions first and grouping them into classes later give a better structure than planning the classes first from the nouns and verbs of the specification?

Status: partly answered

The functions-first method gave 6 classes, 45 methods and 270 source lines.

It passed strict PMD and cost 1273066 weighted tokens. The noun-verb method gave 14 classes, 60 methods and 378 source lines. It passed strict PMD and cost 961206 weighted tokens.

The mean cost to pass a hidden test is M 470362 and M_io 441270 for functions-first against M 545018 and M_io 415798 for noun-verb, so leaving output tokens out reverses the order. Functions-first scores 117/132 on mutation against 88/115 for noun-verb. Fisher exact gives p 0.017 for that pair and 0.59 after Holm over all 55 pairs.

The adjusted scores are 117/122 against 88/109 with p 0.0003 and 0.010 after Holm. These tests compare two programs. Each method ran once.

On the accepted and tentative rating principles both have median 4, except Modular Structure at 4 for functions-first and 3 for noun-verb. One median point on one program each shows neither structure better. The planned rep-1 contrast repeats the pair with 5 replicates each in one batch.

Median adjusted mutation is 0.927 for functions-first and 0.964 for noun-verb, with Mann-Whitney U 16 and Holm p 0.548. The CQP sensitivity check gives Holm p 0.333. So the replicates show neither structure better on test strength or rated quality.

Evidence: runs/full-1/functions-first/metrics.json; runs/full-1/functions-first/costs.json; runs/full-1/noun-verb/metrics.json; runs/full-1/noun-verb/costs.json; runs/full-1/functions-first/survivors.tsv; runs/full-1/noun-verb/survivors.tsv; runs/full-1/functions-first/rating.json; runs/full-1/noun-verb/rating.json; runs/full-1/COMPARISON.md Methods table; runs/full-1/*/curve.json; runs/rep-1/contrast.json. recomputed

Q14 Does each methodology approach full correctness gradually, as gradient descent would, or in one jump?

Status: answered

Both shapes occur.

A curve counts as one jump when one step holds half or more of its gain. That rule was fixed before the shapes were read. Of the 14 full-1 clones 7 rose in one jump and 7 gradually.

Noun-verb, functions-first, multi-start and its 3 start clones went from 0 to 76/76 hidden tests in one LOG.md row. Property-based gained 52 of its 76 in one row. The 7 gradual clones gained on 4 to 9 rows with a largest step of 20 to 32 tests.

Five of them gained their last 24 tests in their final row. Cumulative weighted tokens give all full-1 methods a shared x axis. On it the largest step ended at 0.49 of the cost to completion for functions-first, 0.52 for held-out and 0.73 to 1.0 for the other clones.

Of the 5 tcp-1 arms 4 rose gradually over their 12 loop calls. The risk arm was one jump. It held at 38 tests for six calls and then gained the other 38 in one call.

No step of any of the 19 curves lost a hidden test. Each method ran once, so the classes are descriptive and any link to method family is post hoc.

Evidence: runs/full-1/curve-shapes.tsv; runs/tcp-1/*/curve.json p; harness/report/curve.py shape. recomputed

Q15 Can any difference between two methodologies be attributed to the methodology itself?

Status: partly answered

Partly.

In the survey each method ran once with one model on one task, so no significance test is possible and every survey gap is a single observation. rep-1 fixed two contrasts before any build and ran 5 replicates of each method in one batch. Anchor Engineering scored a higher adjusted mutation score than Classic greedy TDD with U 25 and Holm p 0.016. The planned contrast detected this adjusted mutation difference for this task, model and batch. The test did not detect a difference between Functions first, classes later and Noun-verb map (Holm p 0.548). rep-2 fixed three contrasts before any build, each survey leader against Classic greedy TDD on M with 5 replicates each in one batch.

The tests did not detect a difference from Classic greedy TDD on M (Holm p 1.000 for all three). On the secondary adjusted mutation score Held-out TDD scored lower than Classic greedy TDD with Holm p 0.024. Costs of full-1 and rep-1 do not compare, as the Limits explain.

Evidence: runs/full-1/args.json; runs/full-1/*/costs.json; runs/full-1/findings.json; runs/rep-1/args.json; runs/rep-1/contrast.json; runs/rep-2/args.json; runs/rep-2/contrast.json. recomputed

The Held-out TDD contrast spans a later rebuild. See rep-2 planned contrasts and limits.

Q16 Did each implementing agent follow its methodology's steps and rules?

Status: partly answered

Partly.

The rules a machine can check were mostly followed. Most methodology steps cannot be checked by machine and stay unaudited. Red before green is unclear for 21 of 198 full-1 and 18 of 42 tcp-1 test edits and the recall of the Bash-write check is unchecked.

A mechanical audit read each implementing agent's tool calls and each commit from the starter tag to its method branch. It checks three rules and one measure. The counts below are recomputed.

All 426 full-1 src edits the audit found and 88 of the 92 tcp-1 src edits it found used the Write or Edit tools. No Bash call the audit recognises wrote src and kept the change. The other 4 tcp-1 edits were reverted fault injections.

Each was a Bash sed that broke src for one test run, and the same call undid it with git restore. The check does not detect git apply, patch or script files such as sh .scratch/x.sh. Of 198 full-1 test edits followed by a src edit 162 had a red run of the edited test class between them.

15 had no test run. 21 are unclear. In 10 of those only passing runs came between.

In 11 no red run named an edited test class. 7 of those followed a spec-file edit. A spec-file edit names no class, so its red runs stay unclear.

Of 42 such tcp-1 edits 20 had a red run of the edited test class, 4 had no test run and 18 are unclear. Noun-verb's tests and src were written by different agents, so red before green is unaudited for it. Of 109 full-1 LOG.md rows 3 claim a test count that differs from the net test methods added since the row before.

They are in held-out, multi-start and tpp. All 60 tcp-1 rows match and every row sha names a branch commit. No prompt caps tests per commit.

Of 124 full-1 code commits made after a passing run 17 added no test, 59 added one and 48 added two or more. All 35 such tcp-1 commits added one test. The main session read a seeded sample of 20 flagged and 20 compliant items, and these precision figures are verified.

All 20 compliant items were correct, with a Wilson 95% interval of 0.84 to 1.00. All 9 no-test-run flags were correct, 0.70 to 1.00. None of the 9 only-passing-run flags was a breach, 0/9.

In each the test passed on arrival and the src edit was a refactor or a fault injection. That case is now unclear. The 3 LOG.md flags were not sampled.

A hand read of classic's LOG.md verifies that it logs its sed calls on spec files and UNITS.md as conflicts with its no-script edit rule. The audit checks src writes only. The unaudited steps are step choice, change size, refactoring, design fidelity, schedules, score rules and conflict logging.

Each method ran once. The next two claims come from findings.json and are unchecked. Each method's LOG.md has 0 to 9 rows that mention a conflict.

Two calls hit the cycle cap. The same audit over the 20 rep-1 replicates flagged 1 of 850 src edits as a Bash write, in a functions-first replicate. Of 222 rep-1 test edits followed by a src edit 175 had a red run of the edited test class, 25 were flagged and 22 are unclear, and 19 of the flagged are in functions-first.

Noun-verb's red before green stays unaudited there too. 2 of 148 rep-1 LOG.md rows were flagged. No rep-1 flag was sampled by hand, so their precision is unchecked.

Over the 20 rep-2 replicates the audit flagged 16 of 330 src edits as Bash writes, 10 in mutation-guided, 5 in classic and 1 in held-out. Of 168 rep-2 test edits followed by a src edit 119 had a red run of the edited test class, 8 were flagged and 41 are unclear. 9 of 162 rep-2 LOG.md rows were flagged.

No rep-2 flag was sampled by hand, so their precision is unchecked.

Evidence: runs/full-1/adherence.tsv; runs/tcp-1/adherence.tsv; runs/full-1/*/adherence.json; runs/tcp-1/*/adherence.json; runs/rep-1/*/adherence.json; runs/rep-2/*/adherence.json; harness/report/adherence.py; runs/full-1/findings.json; runs/full-1/classic/LOG.md. recomputed

Q17 Does the TDD literature lack a defined method for building and ordering the test list that drives development?

Status: partly answered

Partly.

The sources read give a few defined methods for building a test list and none for ordering one. Pratama et al. 2026 derive one test per basis path of a use case.

Riquelme et al. 2019 generate tests from models. Neither was tested beyond one hypothetical case or small tool trials.

Guerra et al. 2014 and the worked example in Beck 2002 treat the list as an informal checklist that grows during the session. Ordering rests on heuristics.

Guerra et al. 2013 name step types such as the simplest scenario first and small increments and give no ranking procedure. Martin's Transformation Priority Premise of 2013 is the most specific ordering rule found.

Its author calls the transformations informal at best and their order not likely correct. Beck 2023 calls picking the next test a skill that only comes with experience and says the order can significantly affect the result. In a 2011 Stack Overflow thread the accepted answer says there is no procedure to write the tests and the list.

Khan and Arshad 2012 found 7 test-list sources and all 7 are grey literature (p. 33). None of the sources read compares one list or order against another.

These claims are verified against the saved sources and their critiques. The answer stays partial for five reasons. ACM DL and Scopus could not be searched on 25 Oct 2026 and OpenAlex stood in.

Forward snowballing from Guerra 2013 and 2014 was not run. 23 full texts could not be obtained. The abstracts or snippets of 5 of those works describe building or ordering rules in adjacent settings.

They are unchecked and one could narrow the ordering claim. Beck 2002 was read through publisher excerpts only, so its Test List pattern and Red Bar Patterns chapter are unread. One reviewer screened all records with rule assistance.

Evidence: references/pratama2026-test-first-protocol_critique.md; references/riquelme2019-mofqa_critique.md; references/guerra2014-preparing-tdd-session_critique.md; references/guerra2013-step-patterns_critique.md; references/beck2002-tdd-by-example_critique.md; references/beck2023-canon-tdd_critique.md; references/martin2013-tpp_critique.md; references/so2011-test-lists_critique.md; references/khan2012-test-lists_critique.md; Search protocol, screening and search log in tdd-experiment-runs/_logs/research-2026-10-25/R1; runs/tcp-1/tdd-methodologies.md Test-list order. verified

Q18 Can regression test case prioritisation criteria (coverage, cost, risk and usage) be carried over to ordering a TDD test list before any of its tests exist?

Status: partly answered

Partly.

They can be carried over, and in tcp-1 they did not clearly help. The usage ratings matched simulated random play. The cost and risk ratings showed no detectable link to the measured data, and 12 candidates are too few to rule out a moderate one.

A lister agent wrote 19 requirement clauses and 12 candidate tests with the clauses each checks and with risk, usage and cost estimates. The list passed the harness checks with 0 audit flags. Each of the 5 arms built all 12 candidates in its order in 12 loop calls with 0 off-order calls and its list files intact.

Only the cost order scored above random on both APFD-I and APFDc-H. Usage scored above random on APFDc-H only. No criterion reached completion for fewer weighted tokens than random.

Q20 gives the figures. Each rating check below is a Spearman rho with a seeded permutation p against rho = 0 and a seeded percentile bootstrap 95% interval, each over 10000 draws at seed 20261024. All are recomputed.

Usage against the share of moves that exercise each clause gave rho 0.974, p 0.0001 and an interval of 0.80 to 1.00 over n = 9 clauses, R4, R7 to R11 and R12 to R14. A p of 0.0001 is the smallest 10000 draws can give. The shares come from 10000 seeded games of random legal moves split over the 6-4 and 7-5 boards.

They measure rule use under random play and not under human play. R12 to R14 share one end-of-game value. Taken as one point they give rho 0.991 at n = 7.

The cost estimate against src lines added per candidate gave rho 0.03, p 0.92 and an interval of -0.57 to 0.59 at n = 12. The risk of each candidate's riskiest checked clause against its red-green cycles gave rho 0.19, p 0.65 and an interval of -0.38 to 0.68 at n = 12. Both measures are means over the 5 arms.

Src lines depend on build order, since an early candidate adds code that later ones reuse. The cycles are the agents' own LOG.md counts. Every row logs 0 or 1, and 6 rows, all in the cost arm, log a cycle although the actions say the test never went red on behaviour.

At n = 12 a two-sided test at the 0.05 level has 80% power only for a true rho of about 0.73 or above by the Fisher z approximation, so a weaker link would likely go unseen. The gap was how far the ratings depend on the lister. A second blind lister in run tcp-1-list2 rated the same 19 clauses and wrote the checks of the same 12 candidates.

Its usage ratings agreed with the first lister's with rho 0.962 and an interval of 0.882 to 0.997 at n = 19. Its risk ratings agreed with weighted kappa 0.530 at n = 19, a moderate agreement. Each candidate's checked clauses overlapped the first list's with a mean Jaccard of 0.71.

So the usage ratings hold across two listers and the risk ratings only in part.

Evidence: runs/tcp-1/tdd-methodologies.md Test-list order; runs/tcp-1/args.json; runs/tcp-1-list/ORDERS.md; runs/tcp-1-list/tcp-1-list-lister/CLAUSES.md; runs/tcp-1-list/tcp-1-list-lister/TESTLIST.md; runs/tcp-1/q18-ratings.json; harness/report/q18_ratings.py; harness/report/usage.py; runs/tcp-1/*/LOG.md; runs/tcp-1/*/curve.json; runs/tcp-1/COMPARISON.md Test-list order family; runs/tcp-1-list2/reliability.json. recomputed

Q19 Do the different prioritisation criteria produce materially different orders of the same test list?

Status: answered

Yes.

Normalised Kendall distance among the 5 launched orders runs from 0.227 for coverage and cost to 0.667 for random and usage. The list's own written order sat 0.061 from the cost-only order, under the 0.15 threshold fixed before any data. Neither of those two orders is a launched arm.

This is one list of 12 tests.

Evidence: runs/tcp-1-list/ORDERS.md Kendall matrix; runs/tcp-1/args.json orders. recomputed

Q20 Does the order of the test list change how early hidden tests start to pass and the token cost to reach completion, and does any criterion beat a random order?

Status: answered

The five orders gave different curves and costs.

Only the cost order beat random on both measures of how early hidden tests passed. Usage beat random on APFDc-H only. No order beat random on token cost.

All 5 arms reached 15/15 with 76/76 hidden and 69/69 unseen tests. APFD-I on the calls basis was 0.6316 for random, 0.5515 for coverage, 0.6634 for cost, 0.5274 for risk and 0.6129 for usage. APFDc-H was 0.5111, 0.4715, 0.5716, 0.4166 and 0.5309 in the same order.

Cost to completion was 1192505 weighted tokens for random and 1236588 to 1349860 for the other four. Weighted tokens per arm were 1156744 for random and 1204450 to 1319757 for the other four. Each order ran once at seed 20261020, so these figures are descriptive.

No significance test applies and a gap may be run noise.

Evidence: runs/tcp-1/*/curve.json; runs/tcp-1/*/costs.json; runs/tcp-1/*/summary.json; runs/tcp-1/COMPARISON.md Test-list order family; harness/report/curve.py. recomputed

Q21 Should the approach in the post "Principles of Anchor Engineering" become an additional methodology arm?

Status: partly answered

Partly.

The post was saved and critiqued on 25 Oct 2026 and judged against four criteria for a new arm. It fails two, passes one and one is unclear. These limits define what the constructed agent-only arm represents.

The post is a blog post with no peer review. It measures or compares nothing. It states five principles and puts human judgement in an Anchor Book kept in the repository.

The Book holds the design, anchors that state what must be true about the system, the check for each anchor and planned chunks of work. It fails the executable-steps criterion. It gives no numbered procedure, Book schema or anchor format and leaves the mechanics to later posts.

The one follow-up post of 10 Oct 2026 gives none either. Its judgement check-in and its conflict surfacing need a human. The harness tells agents never to stop to ask.

It passes the criterion of not relabelling noun-verb or mapped-path. An anchor is a property with its check and not a class or verb of their map. It fails the measurable-prediction criterion because it predicts none of the measured columns.

Its fit to the F and L stage kinds is unclear. An agent-only construction would fit. The human check-in matches no stage kind and replacing it would change the method.

Any arm built from the post would be the experimenters' own design. Its anchor list would partly restate the numbered clauses of the tcp list. An agent-only arm named Anchor Engineering was then written from the post and ran 5 replicates in rep-1.

All 5 reached adjusted mutation 1.000 and it scored higher than Classic greedy TDD with Holm p 0.016. The arm's steps remain the experimenters' own design, so the result speaks for that arm and not for the post's human-in-the-loop method.

Evidence: references/wilhelm2026-anchor-engineering_critique.md; references/wilhelm2026-anchor-engineering.html; references/wilhelm2026-anchoring-mental-model_critique.md; Criteria verdicts in tdd-experiment-runs/_logs/research-2026-10-25/R2/criteria.md; harness/PLAN.md Stage map per method; harness/PLAN.md Batch workflow outline; harness/workflow/stages.json; runs/rep-1/tdd-methodologies.md; runs/rep-1/contrast.json. recomputed

Q22 Does any methodology give more consistent results across its replicates on the other measures, so that it is more predictable and therefore more valuable than one with higher variance?

Status: partly answered

Seven methods were replicated for the planned contrasts of rep-1 and rep-2, so this answer covers only those seven and is exploratory.

C pools every replicate of both runs, 10 for classic and 5 for each other method. The global spread test did not detect a difference. Consistency score C, where lower means less observed spread, is 0.38 for mutation-guided, 0.87 for classic, 0.96 for held-out, 1.06 for anchor-engineering and coarse-to-fine, 1.14 for functions-first and 1.65 for noun-verb.

A permutation test of the spread of C across the seven methods gives p 0.464. Brown-Forsythe permutation tests of spread per metric give Holm p 0.339 for unadjusted mutation, 0.613 for classes and 0.518 for cost to completion. Per metric anchor-engineering, coarse-to-fine and mutation-guided sat at the adjusted mutation bound of 1.000 and noun-verb had the widest relative spread on both mutation scores.

Every replicate of functions-first, noun-verb and anchor-engineering sat at 0 PMD violations, while classic varied from 4 to 10 and no rep-2 replicate passed. The raw costs mix CLI builds 2.1.308 and 2.1.309. The drift test of classic between rep-1 and rep-2 gives M p 0.310.

Evidence: runs/rep-1/*/metrics.json; runs/rep-1/*/curve.json; runs/rep-1/contrast.json; runs/rep-2/*/metrics.json; runs/rep-2/*/curve.json; runs/rep-2/contrast.json; harness/report/replicate.py consistency; harness/report/stats.py brown_forsythe_permutation. recomputed