Each survey method ran once on one task, so no significance test is possible and every survey gap is a single observation.
Tag: verified.
This page sets out how the study scores its awards, which test answers each question and what the study cannot show. It defines no metric. M, E, C and every other measure are defined on Methodology: measures, and short labels such as CQP and PMD in the Glossary. The test results are on the results pages for the survey, the planned contrasts and consistency.
The medals are on the landing page. unchecked
Award scoring
Each award but best overall places the methods on one value, lower first. places gives competition places, ties sharing the best place. podium keeps the places up to TOP = 3 and award gives them the medals gold, silver and bronze. Best overall ranks every method on the 3 CRITERIA cost_effective, fewest_errors and alignment with stats.ranks, ties sharing their average rank, and sums the ranks. Cheapest and Most consistent design are bronze badges that never count toward it. Checked against awards.py at tdd-experiment c3dd988. verified
Figure 5.1
How awards are scored
- Method records. one per method. Lane: Criteria.
Next: CRITERIA; Cost to completion, C.
- CRITERIA. cost_effective, fewest_errors, alignment. Lane: Criteria.
Next: places (Criteria, places); stats.ranks.
- places (Criteria, places). per criterion, lower first. Lane: Criteria.
Next: podium.
- podium. top 3: gold, silver, bronze. Lane: Criteria.
- stats.ranks. ties share their average rank. Lane: Best overall.
Next: overall.
- overall. rank sum over CRITERIA. Lane: Best overall.
Next: places and podium.
- places and podium. top 3: gold, silver, bronze. Lane: Best overall.
- Cost to completion, C. never ranked in CRITERIA. Lane: Badges.
Next: places (Badges, badge-places).
- places (Badges, badge-places). lower first. Lane: Badges.
Next: Bronze badge only.
- Bronze badge only. limit stated with the award. Lane: Badges.
Each award but best overall places methods on one value. Only the 3 criteria feed the best overall rank sum. verified
Text version: How awards are scored
- Method records. one per method. Lane: Criteria.
Next: CRITERIA; Cost to completion, C.
- CRITERIA. cost_effective, fewest_errors, alignment. Lane: Criteria.
Next: places (Criteria, places); stats.ranks.
- places (Criteria, places). per criterion, lower first. Lane: Criteria.
Next: podium.
- podium. top 3: gold, silver, bronze. Lane: Criteria.
- stats.ranks. ties share their average rank. Lane: Best overall.
Next: overall.
- overall. rank sum over CRITERIA. Lane: Best overall.
Next: places and podium.
- places and podium. top 3: gold, silver, bronze. Lane: Best overall.
- Cost to completion, C. never ranked in CRITERIA. Lane: Badges.
Next: places (Badges, badge-places).
- places (Badges, badge-places). lower first. Lane: Badges.
Next: Bronze badge only.
- Bronze badge only. limit stated with the award. Lane: Badges.
Award rules
Each rule is quoted from awards.py. verified
Counts toward best overall marks the awards whose criterion joins the rank sum, by the criterion to award mapping in the Award column below. unchecked
| Award | Rule | Counts toward best overall |
|---|---|---|
| Best overall | Lowest plain sum of each method's ranks on M, E and alignment, ties sharing their average rank. Cheapest and consistency never count. | Is the rank sum |
| Most cost effective | Lowest M, the cost to completion times 1 minus APFDc-H, among methods with a functional mark of 15/15. M_io leaves output tokens out. | Yes |
| Fewest errors | Lowest E, the distinct PMD findings plus adjusted surviving mutants, hidden failures and unseen failures. | Yes |
| Best CQP/PMD alignment | The blind CQP rating agrees with PMD on every principle and PMD reports no issue. | Yes |
| Cheapest | Lowest cost to completion. A bronze badge that never counts toward best overall. | No |
| Most consistent design | Lowest consistency score C. A bronze badge that never counts toward best overall. | No |
The Most consistent design rule carries a consistency limit, which the landing page states with the award. unchecked
awards.py functions and constants
Criterion names and returns, docstrings and constants are quoted verbatim from awards.py at the snapshot sha. verified
The Award column is this page's mapping of each criterion to the award it scores. awards.py does not state it. unchecked
| Criterion | Award | Returns |
|---|---|---|
cost_effective | Most cost effective | record['m'] if record['full'] else None |
fewest_errors | Fewest errors | record['e'] |
alignment | Best CQP/PMD alignment | 0 if record['aligned'] else None |
| Function | Docstring |
|---|---|
error_count | E: distinct PMD findings, adjusted surviving mutants, hidden failures and unseen failures, None when any is unmeasured. |
agrees | Whether the raters read each principle clean exactly where PMD finds nothing in it. |
aligned | Both the blind rating agrees with PMD and PMD reports no issue. False unrated or unlinted. |
cost_effective | No docstring in awards.py. |
fewest_errors | No docstring in awards.py. |
alignment | No docstring in awards.py. |
places | Competition places, lower first, ties sharing the best place. |
podium | [(place, record, value)] of the records value scores, lower first, up to place top. |
overall | {name: rank sum} over CRITERIA, each ranked by stats.ranks so ties share their average rank. A method a criterion leaves out ranks last on it. |
consistency_limit | No docstring in awards.py. |
tokens | No docstring in awards.py. |
award | {title, rule, places, hover}, each place (place, medal, name, value text). medal fixes every place's medal. |
awards | The award list. spread is {scores: {name: C}, p, runs, n: {name: builds}, builds, drift: [{name, x, y, p}]} of the replicates pooled over the joined contrasts runs, None without one. |
| Constant | Value |
|---|---|
TOP | 3 |
MEDALS | gold, silver, bronze |
CLEAN_MEDIAN | 4 |
Statistical tests
Each row of the table names a question the study asks, the data it uses, the test as the report names it and the stats.py functions whose docstrings describe that test. The source sentence is quoted from the report section that reports the test. verified
Functions are matched to tests by docstring. The snapshot does not record which function computed each reported number. unchecked
Figure 5.2
Which test answers each question
- Do mutation scores differ between the survey methods?. Lane: Survey scores.
Next: permutation_chi2, fisher_pairs and holm.
- permutation_chi2, fisher_pairs and holm. Lane: Survey scores.
Next: Holm over every pair of methods.
- Holm over every pair of methods. Lane: Survey scores.
Next: Survey (full-1) (Survey scores, at0).
- Survey (full-1) (Survey scores, at0). Lane: Survey scores.
- How precise is each method's mutation score?. Lane: Score intervals.
Next: wilson.
- wilson. Lane: Score intervals.
Next: None, descriptive only.
- None, descriptive only. Lane: Score intervals.
Next: Survey (full-1) (Score intervals, at1).
- Survey (full-1) (Score intervals, at1). Lane: Score intervals.
- Does one replicated arm differ from another?. Lane: Planned contrasts.
Next: mann_whitney_exact and holm.
- mann_whitney_exact and holm. Lane: Planned contrasts.
Next: Holm over the planned pairs of each outcome.
- Holm over the planned pairs of each outcome. Lane: Planned contrasts.
Next: Planned contrasts.
- Planned contrasts. Lane: Planned contrasts.
- Does one design vary more between builds?. Lane: Consistency.
Next: brown_forsythe_permutation and holm.
- brown_forsythe_permutation and holm. Lane: Consistency.
Next: Holm over the metrics.
- Holm over the metrics. Lane: Consistency.
Next: Consistency across runs.
- Consistency across runs. Lane: Consistency.
- Do the ratings track lines of code and PMD?. Lane: Ratings and code.
Next: spearman_trial and holm.
- spearman_trial and holm. Lane: Ratings and code.
Next: Holm over the correlations.
- Holm over the correlations. Lane: Ratings and code.
Next: Ratings (Ratings and code, at4).
- Ratings (Ratings and code, at4). Lane: Ratings and code.
- Do two listers agree on the test list?. Lane: List reliability.
Next: weighted_kappa, spearman_trial and jaccard.
- weighted_kappa, spearman_trial and jaccard. Lane: List reliability.
Next: None.
- None. Lane: List reliability.
Next: Ratings (List reliability, at5).
- Ratings (List reliability, at5). Lane: List reliability.
Connections show the next step in each lane. Clean classes and Rater agreement name no stats.py function in the snapshot, so only the table lists them. unchecked
Text version: Which test answers each question
- Do mutation scores differ between the survey methods?. Lane: Survey scores.
Next: permutation_chi2, fisher_pairs and holm.
- permutation_chi2, fisher_pairs and holm. Lane: Survey scores.
Next: Holm over every pair of methods.
- Holm over every pair of methods. Lane: Survey scores.
Next: Survey (full-1) (Survey scores, at0).
- Survey (full-1) (Survey scores, at0). Lane: Survey scores.
- How precise is each method's mutation score?. Lane: Score intervals.
Next: wilson.
- wilson. Lane: Score intervals.
Next: None, descriptive only.
- None, descriptive only. Lane: Score intervals.
Next: Survey (full-1) (Score intervals, at1).
- Survey (full-1) (Score intervals, at1). Lane: Score intervals.
- Does one replicated arm differ from another?. Lane: Planned contrasts.
Next: mann_whitney_exact and holm.
- mann_whitney_exact and holm. Lane: Planned contrasts.
Next: Holm over the planned pairs of each outcome.
- Holm over the planned pairs of each outcome. Lane: Planned contrasts.
Next: Planned contrasts.
- Planned contrasts. Lane: Planned contrasts.
- Does one design vary more between builds?. Lane: Consistency.
Next: brown_forsythe_permutation and holm.
- brown_forsythe_permutation and holm. Lane: Consistency.
Next: Holm over the metrics.
- Holm over the metrics. Lane: Consistency.
Next: Consistency across runs.
- Consistency across runs. Lane: Consistency.
- Do the ratings track lines of code and PMD?. Lane: Ratings and code.
Next: spearman_trial and holm.
- spearman_trial and holm. Lane: Ratings and code.
Next: Holm over the correlations.
- Holm over the correlations. Lane: Ratings and code.
Next: Ratings (Ratings and code, at4).
- Ratings (Ratings and code, at4). Lane: Ratings and code.
- Do two listers agree on the test list?. Lane: List reliability.
Next: weighted_kappa, spearman_trial and jaccard.
- weighted_kappa, spearman_trial and jaccard. Lane: List reliability.
Next: None.
- None. Lane: List reliability.
Next: Ratings (List reliability, at5).
- Ratings (List reliability, at5). Lane: List reliability.
Where each test is used
| Question | Data | Test | Function | Correction | Reported in | Source sentence |
|---|---|---|---|---|---|---|
| Do mutation scores differ between the survey methods? | full-1 mutation scores of 11 methods | seeded permutation chi-square; Fisher exact tests with Holm correction | permutation_chi2, fisher_pairs, holm | Holm over every pair of methods | Results: survey (full-1) | Over the scores a seeded permutation chi-square gives 63.0 with p below 0.001, and Fisher exact tests with Holm correction separate 11 of the 55 pairs at 0.05. |
| How precise is each method's mutation score? | full-1 killed and compiled mutants per method | Wilson 95% interval | wilson | None, descriptive only | Results: survey (full-1) | Each dot is the share of a method's N compiled mutants its tests killed or timed out, labelled killed/N, and its whisker is the Wilson 95% interval, descriptive only since one suite judges all of a method's mutants. |
| Do clean methods have more classes? | full-1 classes per method, start clones left out | exact one-sided permutation test | not stated in the snapshot | None, one test | Results: survey (full-1) | An exact one-sided permutation test over all 165 ways to pick 3 of the 11 methods gives p = 0.006 for clean methods having this many classes or more. |
| Does one replicated arm differ from another? | rep-1 and rep-2 replicates, 5 per arm | exact two-sided Mann-Whitney U; Holm correction | mann_whitney_exact, holm | Holm over the planned pairs of each outcome | Results: planned contrasts | The planned contrasts of run rep-1, each an exact two-sided Mann-Whitney U over the replicates with Holm correction over the planned pairs of each outcome. |
| Does one design vary more between builds? | rep-1 and rep-2 replicates pooled per method | Brown-Forsythe permutation tests; Holm p | brown_forsythe_permutation, holm | Holm over the metrics | Results: consistency across runs | Brown-Forsythe permutation tests of spread per metric give Holm p 0.339 for unadjusted mutation, 0.613 for classes and 0.518 for cost to completion. |
| Do the three raters agree? | full-1, rep-1 and rep-2 blind ratings | Krippendorff's ordinal alpha; seeded bootstrap 95% interval | not stated in the snapshot | None | Supporting data: ratings | Alpha is Krippendorff's ordinal alpha over the three rater slots, with a seeded bootstrap 95% interval in brackets. |
| Do the ratings track lines of code and PMD? | finals medians against LOC and PMD violations | Spearman correlations; Holm-adjusted q | spearman_trial, holm | Holm over the correlations | Supporting data: ratings | LOC and PMD rho are Spearman correlations of the finals medians with lines of code and with distinct PMD violations, with Holm-adjusted q. |
| Do two listers agree on the test list? | tcp-1-list clauses and candidates rated again | linearly weighted kappa; Spearman rho with a seeded bootstrap 95% interval; mean Jaccard overlap | weighted_kappa, spearman_trial, jaccard | None | Supporting data: ratings | Risk agreement is a linearly weighted kappa, usage agreement a Spearman rho with a seeded bootstrap 95% interval, and checks agreement the mean Jaccard overlap of the clauses each candidate test checks. |
stats.py functions
Docstrings are quoted verbatim from stats.py at the snapshot sha. verified
| Function | Docstring |
|---|---|
wilson | (lo, hi) Wilson score interval of k successes in n trials, which stays inside 0 to 1 at k = 0 and k = n. |
fisher_exact | Two-sided Fisher exact p of the 2x2 table [[a, b], [c, d]]: the summed probability of every table with the same margins that is no more likely than this one. Integer weights keep ties exact. |
holm | Holm step-down adjusted p values in the order of ps, each capped at 1 and none below a smaller p's. |
fisher_pairs | {(i, j): (p, Holm p)} of the Fisher exact test of each pair of rows i < j of a k x 2 table, Holm-adjusted over every pair. |
chi2_rows | Pearson chi-square of a k x 2 table from each row's successes and size, 0 when a column is empty. An empty row adds nothing. |
permutation_chi2 | (chi-square, p) of the k x 2 table of (successes, failures) rows. p is the share of seeded label shuffles, counting the observed one, whose chi-square reaches the observed. |
deviations | Each value's absolute deviation from its group median. |
brown_forsythe | Brown-Forsythe F: one-way ANOVA F of each value's absolute deviation from its group median. |
anova_f | One-way ANOVA F. inf when the values vary between groups only, 0 when they do not vary at all. |
brown_forsythe_permutation | (F, p) of brown_forsythe. p is the share of seeded shuffles of the pooled deviations from each value's own group median over groups of the same sizes, counting the observed one, whose F reaches the observed. Shuffling raw values would mix the group medians into the spread. |
ranks | 1-based ranks with ties given their average rank. |
mann_whitney_exact | (U, p) of the exact two-sided Mann-Whitney test. U counts the pairs with x above y, ties as half. p is the share of every split of the pooled average ranks into a group of len(xs) whose rank sum lies at least as far from its mean as the observed, so ties stay exact. The C(n, len(xs)) splits suit small samples only. |
weighted_kappa | Cohen's linearly weighted kappa of two raters' numeric ratings of the same items: 1 minus the mean |x - y| over its chance mean across every pairing of the two raters' ratings. None when chance disagreement is 0. |
jaccard | Shared items over all items of two collections. Two empty ones agree fully and give 1. |
spearman | Spearman rho as Pearson r of average ranks, None when either side has no spread. |
spearman_test | No docstring in stats.py. |
spearman_trial | (rho, p, lo, hi, skipped): p is the share of seeded shuffles of ys, counting the observed one, whose |rho| reaches the observed, against the null rho = 0. lo to hi is the seeded percentile bootstrap 95% interval over resampled pairs, skipping the resamples with no spread, which skipped counts. lo and hi are None when no resample has spread. All None when either side has no spread. |
Threats to validity
The report states its limits once, kept here word for word. The gaps after them come from the run records and the code docstrings in the snapshot. verified
Limits
What this study cannot show, stated once for the whole page.
rep-1 and rep-2 each replicated only 4 methods, 5 times each, on one task with one model, with Classic greedy TDD in both. Their planned contrasts detect only large effects and the consistency comparison is exploratory.
Tag: verified.
full-1, rep-1 and rep-2 are three batches. The cost gap between full-1 and rep-1 is no harness change, since the prompts and launch scripts are identical. It coincides with an agent CLI change from 2.1.295 in full-1 to 2.1.308 in rep-1, checked on one agent transcript of each batch. rep-2 ran on CLI 2.1.309, as its contrast.json runtime records. Every agent in the costs.json files records CLI 2.1.295 in full-1 and 2.1.309 in rep-2, so full-1 and rep-2 differ by the same kind of CLI change.
Tag: verified.
Output tokens per request were 148 in full-1, 605 in rep-1 and 495 in rep-2, pooled over every clone of each run. The survey builds of classic, functions-first and noun-verb each cost less than all 5 of their rep-1 replicates, with median ratios of 1.13 to 1.45. With output tokens left out those ratios fall to 0.91 to 1.09 and only noun-verb's survey build still costs less than all 5 replicates. Cost comparisons of full-1 against rep-1 or rep-2 are not valid on full weights. The survey builds on the replicate figures are captioned as a different batch and no claim pools them. Consistency score C pools the raw costs of rep-1 and rep-2 across CLI builds 2.1.308 and 2.1.309. The drift test of Classic greedy TDD between those runs gives M p 0.310.
Tag: recomputed.
The rep-2 rating accepts none of the eight principles. Minimal Duplication, Modular Structure and Problem Alignment are tentative and Simple Constructs reaches alpha 0.52. So CQP accepted, which sums the three principles accepted on the full-1 finals, has no validity check on the rep-2 finals, and the rep-2 CQP sensitivity check is flagged.
Tag: recomputed.
One rep-2 high audit flag is a minor true positive. Most informative next test replicate rep-2-r09 ran ls ../ from its clone, which listed only the names in the clones dir, _heldout among them. It read no file outside its clone. The other 49 rep-2 high flags were judged false positives.
Tag: verified.
All 5 Held-out TDD replicates of rep-2 were rebuilt on 2026-10-30 at harness commit cc8c8cb, the held-out splitter merge, with the recorded harness delta of sha256 8fde5d73. The rest of the batch ended on 2026-10-29 at harness commit 809589d. So the Held-out TDD against Classic greedy TDD contrast spans two launches.
Tag: verified.
Whether the recorded token usage matches the real thinking volume of each request is unchecked.
Tag: unchecked.
All 11 methods reached the ceiling of 15/15, so the functional mark cannot rank them.
Tag: recomputed.
One model ran every agent, claude-opus-5-5 at medium effort.
Tag: verified.
Wall times were measured while all methods ran in parallel on one machine.
Tag: verified.
Bench times come from one machine and hold only for it.
Tag: verified.
Each rater saw a rebuilt clone whose history names no method and a prompt that names none. Comments inside src and test stay as written, so a comment could still hint at a method. The single review per method that the rating supersedes had no second reviewer.
Tag: verified.
The calibration gate was revised on 2026-10-26 after round-a failed. C5 now checks only problem_alignment on the starter stub (D20). C2 now passes a drop of 1 from a parent median of 3 (D19). The modular mutant was rebuilt as one god class (D21). The stub's round-b medians would fail the original C5, which needed all eight principles at 1 or 2.
Tag: verified.
Mutant counts differ by method, so mutation scores compare only loosely.
Tag: recomputed.
Alterability is not measured. No change case was carried out and the rating leaves it out.
Tag: verified.
Testability in the course sense and cohesion, complexity and coupling are not measured.
Tag: verified.
Weighted tokens are relative prices and state no dollar cost.
Tag: verified.
The 15 harness and review agents that belong to no method are left out of every per-method cost.
Tag: recomputed.
The agent transcripts behind Q9 and Q10 stay outside the repo, so those counts cannot be rechecked from this page.
Tag: verified.
The overall rank sum is descriptive. Alignment uses all eight principles, including those with low or unestimable agreement. E is an additive composite count and does not count unique defects. verified
M = T × (1 − APFDc-H) is a trapezoidal estimate over recorded checkpoints. Checkpoint spacing can miss changes between observations. M_io recomputes the curve without output tokens as a sensitivity check. It does not establish comparability across CLI versions. C pools costs without correcting those differences. verified
Gaps in the records
Each row is derived from facts.json at the snapshot sha, so it changes when the snapshot does. recomputed
rep-2/rebuild rebuilt the 5 Held-out TDD replicates rep-2-r04, rep-2-r08, rep-2-r14, rep-2-r17 and rep-2-r20 at harness commit cc8c8cb, where the rest of rep-2 records 809589d. The rebuild records a harness delta of sha256 8fde5d73. Both record agent CLI 2.1.309.
recomputed
Each run records the sha256 of the methods doc it was built under, shown here by its first 8 characters. 64606950: full-1, pilot-1 and pilot-2. 12f026f6, the methods text these pages quote: rep-1, rep-2, rep-2/rebuild and tcp-1-list2. 1f11dd37: tcp-1-list and tcp-1.
recomputed
7 of 9 runs record no harness commit in args.json: full-1, pilot-1, pilot-2, rep-1, tcp-1-list, tcp-1-list2 and tcp-1. The harness version they ran is not on record.
recomputed
Docstrings state 2 limits of the analysis code. stats.py mann_whitney_exact: “The C(n, len(xs)) splits suit small samples only.” awards.py overall: “A method a criterion leaves out ranks last on it.”
recomputed