Each card quotes a column or term definition from the snapshot with its claim tag. The lines under it give the typed formula where the page has one, the inputs when the definition names no Source, and the module that computes it. Formula, Inputs, Module and Also labelled lines, this lead and the Figure 4.1 caption are written by this site. Tag: unchecked.
The methods text on these pages is from methods doc 12f026f6. The runs this page names were built under these doc versions, by short sha256: full-1 64606950, rep-1 12f026f6, rep-2 12f026f6, tcp-1 1f11dd37, tcp-1-list2 12f026f6 and tcp-1-list 1f11dd37.
Grade, size, mutation, static analysis, bench and rating columns read the method's final clone. Tokens, weighted tokens, agent calls and audit flags sum every clone of the method. Wall time spans every clone of the method from the first to the last transcript timestamp. Multi-start TDD has a final clone and 3 start clones. Costs, times, agent calls and audit flags leave out the 15 unattributed agents listed in RUN.md and in the Run section of report.html. The reference row's medians come from calibration round round-b.
Strict PMD compliance. Source: metrics.json from pmd.txt. Tag: recomputed.
Definition: Strict PMD compliance passes when PMD with the starter's kalah_pmd.xml finds no violation in src/ and otherwise fails with the count of distinct violations. The lint sets no minimum priority, so rules of every priority count. It covers src/ only, the scope of the starter's ./run.sh lint. A02-RUBRIC.md awards no marks for PMD, so the functional mark leaves it out.
Module: not stated in the snapshot.
Static analysis violations. Source: metrics.json from pmd.txt. Tag: recomputed.
Definition: Distinct PMD violations against the starter's kalah_pmd.xml. Reports alike in file, line, rule and message count once.
Module: not stated in the snapshot.
Also labelled: PMD violations.
Strict PMD violations. Source: metrics.json from pmd.txt. Tag: recomputed.
Definition: Distinct PMD violations against the starter's kalah_pmd.xml. Reports alike in file, line, rule and message count once.
Module: not stated in the snapshot.
Mutation score. Source: metrics.json from mutate.txt. Tag: recomputed.
Definition: Killed plus timed-out mutants out of those that compiled (K/N), then mutants left unrun. Only the method's own tests run, without the starter's two Example tests and their ExampleSpec helper. The reference row names the tests it ran.
Module: not stated in the snapshot.
Also labelled: Mutants killed; Score %.
Mutation score and K/N. Tag: verified.
Definition: Mutation testing makes small changes to the code, called mutants, and counts those the tests catch. K/N is killed plus timed-out mutants out of the N that compiled.
Inputs: the Source of Mutation score (metrics.json from mutate.txt).
Module: not stated in the snapshot.
Also labelled: Killed; Compiled mutants.
Adjusted mutation. Source: metrics.json. Tag: recomputed.
Definition: The mutants the build's own tests kill or time out, over those plus the survivors the hidden suite kills. Survivors missed by the hidden suite are excluded. They may be equivalent or expose gaps in that suite.
Module: not stated in the snapshot.
Also labelled: Adjusted %; Adjusted mutation score; Median adjusted mutation.
Unadjusted mutation. Source: metrics.json. Tag: recomputed.
Definition: Killed mutants over all compiled mutants.
Module: not stated in the snapshot.
Tokens. Source: costs.json from transcripts. Tag: recomputed.
Definition: Input, cache creation, cache read and output tokens over every agent of the method's clones.
Module: not stated in the snapshot.
Weighted tokens. Source: costs.json from transcripts. Tag: recomputed.
Definition: Weighted tokens count input 1, cache creation 1.25, cache read 0.05 and output 5 per token. These are claude-opus-5-5 prices relative to an input token and state no dollar cost. Summed over every agent of the method's clones.
Formula: 1 × input + 1.25 × cache creation + 0.05 × cache read + 5 × output
Module: not stated in the snapshot.
Wall time. Source: costs.json. Tag: recomputed.
Definition: First to last transcript timestamp over the method's clones, as h:mm:ss.
Module: not stated in the snapshot.
Agent calls. Source: costs.json. Tag: recomputed.
Definition: Workflow agents started for the method's clones, any without a transcript included.
Module: not stated in the snapshot.
Run order. Source: costs.json. Tag: recomputed.
Definition: Parallel when the method's first to last transcript span overlaps every other method's, else its start rank as start k of N methods with costs.
Module: not stated in the snapshot.
Cost to completion. Source: curve.json from costs.json with steps.csv for a method or batch.json tcp.calls for an arm. Tag: recomputed.
Definition: Weighted tokens of the lead cost plus the final clone's step costs, measure and review agents left out. A method's steps are the steps.csv rows of its graded points. An arm's steps are its loop calls, each costed by the costs.json agent its batch.json tcp.calls label names. A stopped call without a sha counts though it grades no point.
Module: harness/report/curve.py, as the glossary row Cost to completion states.
Also labelled: Weighted tokens to completion; Median cost to completion.
M. Source: curve.json mean_cost_to_pass with Cost to completion's costs.json, steps.csv or batch.json tcp.calls. Tag: recomputed.
Definition: Trapezoidal checkpoint-area estimate of weighted-token cost per hidden test: the cost to completion times 1 minus APFDc-H. Lower M costs less.
Formula: cost to completion × (1 − APFDc-H)
Module: harness/report/curve.py, as the glossary row M and M_io states.
Also labelled: M, weighted tokens per hidden test passed.
M_io. Source: curve.json mean_cost_to_pass_io with Cost to completion's costs.json, steps.csv or batch.json tcp.calls. Tag: recomputed.
Definition: M_io recomputes the curve without output tokens as a sensitivity check. This does not establish comparability across CLI builds.
Module: harness/report/curve.py, as the glossary row M and M_io states.
Clear Presentation /5. Source: rating.json. Tag: unchecked.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Clear Presentation. Alpha over the finals is 0.62 [0.23, 0.85], so its status is unchecked.
Module: not stated in the snapshot.
Explanatory Language /5. Source: rating.json. Tag: unchecked.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Explanatory Language. Alpha over the finals is 0.41 [-0.19, 0.88], so its status is unchecked.
Module: not stated in the snapshot.
Consistent Code /5. Source: rating.json. Tag: unchecked.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Consistent Code. Alpha over the finals is not measured, so its status is unchecked.
Module: not stated in the snapshot.
Used Content /5. Source: rating.json. Tag: unchecked.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Used Content. Alpha over the finals is 0.58 [-0.10, 0.91], so its status is unchecked.
Module: not stated in the snapshot.
Simple Constructs /5. Source: rating.json. Tag: recomputed.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Simple Constructs. Alpha over the finals is 1.00 [not estimable, 34.7% of resamples skipped], so its status is accepted.
Module: not stated in the snapshot.
Minimal Duplication /5. Source: rating.json. Tag: recomputed.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Minimal Duplication. Alpha over the finals is 0.88 [-0.03, 1.00], so its status is accepted.
Module: not stated in the snapshot.
Modular Structure /5. Source: rating.json. Tag: recomputed.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Modular Structure. Alpha over the finals is 0.85 [0.51, 1.00], so its status is accepted.
Module: not stated in the snapshot.
Problem Alignment (tentative) /5. Source: rating.json. Tag: recomputed.
Definition: Median of the valid blind ratings from 1 to 5 by three raters on the cqp.html principle Problem Alignment. Alpha over the finals is 0.75 [0.29, 1.00], so its status is tentative.
Module: not stated in the snapshot.
CQP. Source: rating.json. Tag: recomputed.
Definition: The summed median blind rating over the four cqp.html principles, 4 to 20.
Module: not stated in the snapshot.
Also labelled: Median CQP.
CQP accepted. Source: rating.json. Tag: recomputed.
Definition: The summed median rating over the principles accepted on full-1's finals, fixed before any replicate build.
Module: not stated in the snapshot.
Awards and E. Tag: verified.
Definition: The Headline's awards rank the survey methods, lower first, with ties sharing a place. E sums a final's distinct PMD findings, adjusted surviving mutants, hidden failures and unseen failures. Alignment needs the blind CQP rating to agree with PMD on every principle and PMD to report no issue. Best overall is the plain sum of each method's ranks on M among 15/15 methods, E and alignment. Cheapest, the lowest cost to completion, and most consistent design, the lowest C pooled over every replicate of rep-1 and rep-2, take bronze badges only and never count toward best overall. A medal only styles a place and carries no score. harness/report/awards.py defines them.
Formula: distinct PMD findings + adjusted surviving mutants + hidden failures + unseen failures
Inputs: the Sources of Static analysis violations (metrics.json from pmd.txt), Adjusted mutation (metrics.json), Hidden pass /76 (summary.json) and Unseen pass /69 (summary.json).
Function error_count doc: E: distinct PMD findings, adjusted surviving mutants, hidden failures and unseen failures, None when any is unmeasured.
Module: harness/report/awards.py.
Relative spread. Source: replicate.py. Tag: recomputed.
Definition: A method's mean absolute deviation from its own median divided by the pooled mean absolute deviation across eligible builds off the metric's bound. 1 is the pooled reference spread. The reference weights builds, including unequal replicate counts. At bound means at least 60 percent of builds sit at the bound and not comparable that fewer than two methods sit off it.
Module: replicate.py.
Consistency score C. Source: replicate.py. Tag: recomputed.
Definition: The mean relative spread over unadjusted mutation, classes and cost to completion. Exploratory.
Module: replicate.py.
Also labelled: Consistency score.
Holm p. Source: contrast.json. Tag: recomputed.
Definition: The planned contrast's p after Holm correction over the planned pairs. Below 0.05 separates the pair.
Module: not stated in the snapshot.
Each card defines a table header or axis label of the results and data pages that names no metric above. This site writes each definition, and a quoted glossary or report line keeps its source's tag. The Table header line names the report sections whose tables use the field. Tag: unchecked.
Slug. Tag: unchecked.
Definition: The method's short lowercase name.
The glossary row Method and slug says: Its slug is its short lowercase name, which the Branch column shows after method/. Tag: verified.
Table header in: Survey (full-1).
Build. Tag: unchecked.
Definition: The build a method's row stands for.
The report section Replicate table (rep-1) says: Build names the best, median and worst of each method by adjusted mutation, then lower cost to completion. Tag: recomputed.
Table header in: Planned contrasts (rep-1), Planned contrasts (rep-2), Replicate table (rep-1) and Replicate table (rep-2).
Source. Tag: unchecked.
Definition: The run a build comes from, and whether it is its method's representative build.
Table header in: Planned contrasts (rep-1) and Planned contrasts (rep-2).
Replicate. Tag: unchecked.
Definition: One build of a method in a replicated run. A Replicates column counts them.
The glossary row Replicate and representative build says: A replicate is one of the 5 builds of a method in run rep-1 or rep-2. Tag: verified.
Table header in: Replicate table (rep-1), Replicate table (rep-2) and Planned contrasts (rep-1).
Also labelled: Replicates.
Representative build. Tag: unchecked.
Definition: The one build of a method that a per-method row reports.
The glossary row Replicate and representative build says: The representative build is the middle replicate by adjusted mutation score, then by lower cost to completion. Tag: verified.
Table header in: Planned contrasts (rep-1).
Run ms. Tag: unchecked.
Definition: The time of each bench run of a build in milliseconds. Their median is the Bench median ms.
Table header in: Survey (full-1).
Also labelled: Bench run time.
Outcome. Tag: unchecked.
Definition: The measure a planned contrast compares.
The glossary row Planned contrast says: A pair of methods and an outcome fixed before any build of its run. Tag: verified.
Table header in: Contrast table (rep-1) and Contrast table (rep-2). Table header on Results: planned contrasts.
n. Tag: unchecked.
Definition: The count of builds or items a statistic is computed over, one count per group when it compares two.
Table header in: Contrast table (rep-1), Contrast table (rep-2) and List reliability.
U. Tag: unchecked.
Definition: The Mann-Whitney U statistic of a contrast.
Function mann_whitney_exact doc: (U, p) of the exact two-sided Mann-Whitney test. U counts the pairs with x above y, ties as half. p is the share of every split of the pooled average ranks into a group of len(xs) whose rank sum lies at least as far from its mean as the observed, so ties stay exact. The C(n, len(xs)) splits suit small samples only.
Table header in: Contrast table (rep-1) and Contrast table (rep-2). Table header on Results: planned contrasts.
p. Tag: unchecked.
Definition: The p value of a test before Holm correction. The Holm p column corrects it.
Table header in: Contrast table (rep-1) and Contrast table (rep-2).
Permutation p. Tag: unchecked.
Definition: The p value of a seeded permutation test.
Function brown_forsythe_permutation doc: (F, p) of brown_forsythe. p is the share of seeded shuffles of the pooled deviations from each value's own group median over groups of the same sizes, counting the observed one, whose F reaches the observed. Shuffling raw values would mix the group medians into the spread.
Table header in: Consistency table.
Metric. Tag: unchecked.
Definition: The measure a row's statistic is computed on.
Table header in: Consistency table.
Statistic. Tag: unchecked.
Definition: The test statistic a row reports, named with its value, such as a Brown-Forsythe F.
Function brown_forsythe doc: Brown-Forsythe F: one-way ANOVA F of each value's absolute deviation from its group median.
Table header in: Consistency table and List reliability.
Value. Tag: unchecked.
Definition: The value a row reports for its statistic, cap or setting.
Table header in: List reliability and Run record (full-1). Table header on Results: test order (tcp-1).
95% interval. Tag: unchecked.
Definition: A 95% interval around a row's value. A percent interval of a mutation score is a Wilson interval.
Function wilson doc: (lo, hi) Wilson score interval of k successes in n trials, which stays inside 0 to 1 at k = 0 and k = n.
Table header in: List reliability and Survey (full-1).
Also labelled: 95% interval %.
Principle. Tag: unchecked.
Definition: One of the review principles each build is rated on.
The report section Rating agreement (full-1) says: Three blind raters scored each item from 1 to 5 on each cqp.html principle. Tag: recomputed.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
Alpha. Tag: unchecked.
Definition: Rater agreement on a principle. A Finals or All prefix names the set of rated items it covers.
The report section Rating agreement (full-1) says: Alpha is Krippendorff's ordinal alpha over the three rater slots, with a seeded bootstrap 95% interval in brackets. Tag: recomputed.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
Also labelled: Finals alpha [95% CI]; All alpha [95% CI].
Finals q. Tag: unchecked.
Definition: The bootstrap shares that show how sure a principle's status is.
The report section Rating agreement (full-1) says: q is the share of bootstrap alphas below 0.8 and below 0.7. Tag: recomputed.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
Also labelled: Finals q 0.8, 0.7.
Status. Tag: unchecked.
Definition: Whether a principle's ratings are accepted, tentative or unchecked.
The report section Rating agreement (full-1) says: The finals alpha sets each status: accepted at 0.8 or more, tentative at 0.7 or more and unchecked below. Tag: recomputed.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
LOC and PMD rho. Tag: unchecked.
Definition: How closely a principle's ratings follow code size and PMD violations.
The report section Rating agreement (full-1) says: LOC and PMD rho are Spearman correlations of the finals medians with lines of code and with distinct PMD violations, with Holm-adjusted q. Tag: recomputed.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
Also labelled: LOC rho (q); PMD rho (q).
Round. Tag: unchecked.
Definition: One rating round of the blind raters, such as round-a.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
Role. Tag: unchecked.
Definition: What a rating round was for, such as a calibration that passed or failed.
The report section Rating agreement (full-1) says: A calibration round's role comes from its committed verdict.json. Tag: recomputed.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
Agent time. Tag: unchecked.
Definition: The sum over a round's agents of each agent's first to last timestamp, beside its Wall time. Agents that ran side by side add up.
Each method page of run full-1 says: Agent time sums each agent's first to last timestamp, so agents that ran side by side add up.
Table header in: Rating agreement (full-1), Rating agreement (rep-1) and Rating agreement (rep-2).
Label. Tag: unchecked.
Definition: The harness label of an agent call, such as review:r18.
Table header in: Run record (full-1), Run record (rep-1) and Run record (rep-2).
Cap. Tag: unchecked.
Definition: The harness limit a labelled call reached, such as cycles.
Table header in: Run record (full-1).
Agent id. Tag: unchecked.
Definition: The id of an agent transcript that the cost records attribute to no method.
Table header in: Run record (full-1), Run record (rep-1) and Run record (rep-2).
Rating. Tag: unchecked.
Definition: The list phase rating a reliability row checks, such as Risk.
The report section List reliability says: A second lister in run tcp-1-list2 rated the clauses and candidate tests of the list of run tcp-1-list blind. Tag: recomputed.
Table header in: List reliability.
Run. Tag: unchecked.
Definition: The snapshot run a row belongs to.
Table header on Results: planned contrasts.
Setting. Tag: unchecked.
Definition: A setting of run tcp-1.
Table header on Results: test order (tcp-1).
Arm. Tag: unchecked.
Definition: One arm of run tcp-1.
The glossary row Arm says: In tcp-1 each arm follows one order of the same test list. Tag: verified.
Table header on Results: test order (tcp-1).
Order. Tag: unchecked.
Definition: The order of the test list an arm follows.
Table header on Results: test order (tcp-1).
Clause. Tag: unchecked.
Definition: A numbered functional rule of the spec.
List phase step 1 of run tcp-1-list: Number each functional rule in README.md and docs/SPEC.md as a clause R1 to Rn in CLAUSES.md. Leave out code quality and PMD rules.
Table header on Results: test order (tcp-1).
Risk. Tag: unchecked.
Definition: A clause's 1 to 3 rating of the chance a first implementation gets it wrong.
List phase step 2 of run tcp-1-list: Rate each clause 1 to 3 for the chance a first implementation gets it wrong. Rate its usage 1 to 100 as the percent of moves in a typical game that exercise it. Rate a clause of one variant as the percent of games that use the variant times the share of their moves that exercise it.
Table header on Results: test order (tcp-1).
Usage. Tag: unchecked.
Definition: A clause's 1 to 100 rating of how much of a game exercises it.
List phase step 2 of run tcp-1-list: Rate each clause 1 to 3 for the chance a first implementation gets it wrong. Rate its usage 1 to 100 as the percent of moves in a typical game that exercise it. Rate a clause of one variant as the percent of games that use the variant times the share of their moves that exercise it.
Table header on Results: test order (tcp-1).
Candidate. Tag: unchecked.
Definition: A candidate test of the list. A Candidates in order cell lists an order's candidates first to last.
List phase step 3 of run tcp-1-list: List at most 12 candidate tests in TESTLIST.md. Each names one behaviour.
Table header on Results: test order (tcp-1).
Also labelled: Candidates in order.
Checks. Tag: unchecked.
Definition: The clauses a candidate's assertions observe.
List phase step 4 of run tcp-1-list: Give each candidate two fields. checks lists every clause its assertions observe, variants included. relies lists clauses it needs and does not assert.
Table header on Results: test order (tcp-1).
Relies. Tag: unchecked.
Definition: The clauses a candidate needs and does not assert.
List phase step 4 of run tcp-1-list: Give each candidate two fields. checks lists every clause its assertions observe, variants included. relies lists clauses it needs and does not assert.
Table header on Results: test order (tcp-1).
Estimate. Tag: unchecked.
Definition: The production lines a candidate needs on the starter alone.
List phase step 5 of run tcp-1-list: Estimate the production lines each candidate needs on the starter alone.
Table header on Results: test order (tcp-1).
Stage. Tag: unchecked.
Definition: One stage of an arm's harness flow.
Table header on Results: test order (tcp-1).
Runs as. Tag: unchecked.
Definition: How a stage runs, once throughout an arm or repeated per unit.
Table header on Results: test order (tcp-1).
Steps. Tag: unchecked.
Definition: The steps a stage takes.
Table header on Results: test order (tcp-1).
Details. Tag: unchecked.
Definition: Further detail of a stage.
Table header on Results: test order (tcp-1).
Note. Tag: unchecked.
Definition: A note on a row.
Table header on Results: test order (tcp-1).
Hidden-pass curve area. Tag: unchecked.
Definition: The area under an arm's curve of hidden tests passed, which APFD-I scores.
Axis label on Results: test order (tcp-1).
Each report column with its source file, its claim tag and the section of this page that defines it.