How far a second lister agreed on the test list of run tcp-1-list, and how far the three blind raters agreed in runs full-1, rep-1 and rep-2.
List reliability
A second lister in run tcp-1-list2 rated the clauses and candidate tests of the list of run tcp-1-list blind. Risk agreement is a linearly weighted kappa, usage agreement a Spearman rho with a seeded bootstrap 95% interval, and checks agreement the mean Jaccard overlap of the clauses each candidate test checks. Tag: recomputed.
Rating agreement (full-1)
Three blind raters scored each item from 1 to 5 on each cqp.html principle. Alpha is Krippendorff's ordinal alpha over the three rater slots, with a seeded bootstrap 95% interval in brackets. A resample whose values do not vary has no alpha and is skipped, so above 20% of the draws skipped an interval leans on the few varying items and is not estimable. An alpha's q is then not estimable too. q is the share of bootstrap alphas below 0.8 and below 0.7. The finals alpha sets each status: accepted at 0.8 or more, tentative at 0.7 or more and unchecked below. Only accepted and tentative principles join the headline. The finals set holds the 11 final clones. The all set holds every rated item of the published rounds, 20 in all. LOC and PMD rho are Spearman correlations of the finals medians with lines of code and with distinct PMD violations, with Holm-adjusted q. Tag: recomputed.
Rating cost sums costs.py over each round's Workflow transcripts, so the table gives the full cost of the rating. A calibration round's role comes from its committed verdict.json. A failed calibration round adds only its cost. Its ratings join no alpha and no median. Wall time runs from the first to the last timestamp over a round's workflows. Weighted tokens count input 1, cache creation 1.25, cache read 0.05 and output 5 per token. These are claude-opus-5-5 prices relative to an input token and state no dollar cost. Tag: recomputed.
Rating agreement (rep-1)
Three blind raters scored each item from 1 to 5 on each cqp.html principle. Alpha is Krippendorff's ordinal alpha over the three rater slots, with a seeded bootstrap 95% interval in brackets. A resample whose values do not vary has no alpha and is skipped, so above 20% of the draws skipped an interval leans on the few varying items and is not estimable. An alpha's q is then not estimable too. q is the share of bootstrap alphas below 0.8 and below 0.7. The finals alpha sets each status: accepted at 0.8 or more, tentative at 0.7 or more and unchecked below. Only accepted and tentative principles join the headline. The finals set holds the 20 final clones. The all set holds every rated item of the published rounds, 30 in all. LOC and PMD rho are Spearman correlations of the finals medians with lines of code and with distinct PMD violations, with Holm-adjusted q. Tag: recomputed.
Rating cost sums costs.py over each round's Workflow transcripts, so the table gives the full cost of the rating. A calibration round's role comes from its committed verdict.json. A failed calibration round adds only its cost. Its ratings join no alpha and no median. Wall time runs from the first to the last timestamp over a round's workflows. Weighted tokens count input 1, cache creation 1.25, cache read 0.05 and output 5 per token. These are claude-opus-5-5 prices relative to an input token and state no dollar cost. Tag: recomputed.
Rating agreement (rep-2)
Three blind raters scored each item from 1 to 5 on each cqp.html principle. Alpha is Krippendorff's ordinal alpha over the three rater slots, with a seeded bootstrap 95% interval in brackets. A resample whose values do not vary has no alpha and is skipped, so above 20% of the draws skipped an interval leans on the few varying items and is not estimable. An alpha's q is then not estimable too. q is the share of bootstrap alphas below 0.8 and below 0.7. The finals alpha sets each status: accepted at 0.8 or more, tentative at 0.7 or more and unchecked below. Only accepted and tentative principles join the headline. The finals set holds the 20 final clones. The all set holds every rated item of the published rounds, 30 in all. LOC and PMD rho are Spearman correlations of the finals medians with lines of code and with distinct PMD violations, with Holm-adjusted q. Tag: recomputed.
Rating cost sums costs.py over each round's Workflow transcripts, so the table gives the full cost of the rating. A calibration round's role comes from its committed verdict.json. A failed calibration round adds only its cost. Its ratings join no alpha and no median. Wall time runs from the first to the last timestamp over a round's workflows. Weighted tokens count input 1, cache creation 1.25, cache read 0.05 and output 5 per token. These are claude-opus-5-5 prices relative to an input token and state no dollar cost. Tag: recomputed.