The planned contrasts, the consistency table and every replicate build of runs rep-1 and rep-2.
Contrast table (rep-1)
The planned contrasts of run rep-1, each an exact two-sided Mann-Whitney U over the replicates with Holm correction over the planned pairs of each outcome. The sensitivity rows repeat each pair on the measure their outcome names. The outcome fell back from CQP to adjusted mutation because rater agreement over the finals fell short. Simple Constructs alpha 0.67, below 0.7. Minimal Duplication alpha 0.44, below 0.7. Tag: recomputed.
| Contrast | Outcome | n | U |
|---|---|---|---|
| Functions first, classes later against Noun-verb map | adjusted mutation | 5 and 5 | 16 |
| Anchor Engineering against Classic greedy TDD | adjusted mutation | 5 and 5 | 25 |
| Functions first, classes later against Noun-verb map | CQP accepted, sensitivity. Finals alpha below 0.70: simple constructs 0.67, minimal duplication 0.44 | 5 and 5 | 7.5 |
| Anchor Engineering against Classic greedy TDD | CQP accepted, sensitivity. Finals alpha below 0.70: simple constructs 0.67, minimal duplication 0.44 | 5 and 5 | 19 |
| Contrast | p | Holm p |
|---|---|---|
| Functions first, classes later against Noun-verb map | 0.548 | 0.548 |
| Anchor Engineering against Classic greedy TDD | 0.008 | 0.016 |
| Functions first, classes later against Noun-verb map | 0.167 | 0.333 |
| Anchor Engineering against Classic greedy TDD | 0.286 | 0.333 |
Contrast table (rep-2)
The planned contrasts of run rep-2, each an exact two-sided Mann-Whitney U over the replicates with Holm correction over the planned pairs of each outcome. The sensitivity rows repeat each pair on the measure their outcome names. The outcome fell back from CQP to adjusted mutation because rater agreement over the finals fell short. Simple Constructs alpha 0.52, below 0.7. Tag: recomputed.
| Contrast | Outcome | n | U |
|---|---|---|---|
| Held-out TDD against Classic greedy TDD | M | 5 and 5 | 11 |
| Coarse-to-fine TDD against Classic greedy TDD | M | 5 and 5 | 9 |
| Most informative next test against Classic greedy TDD | M | 5 and 5 | 17 |
| Held-out TDD against Classic greedy TDD | M_io, sensitivity | 5 and 5 | 15 |
| Coarse-to-fine TDD against Classic greedy TDD | M_io, sensitivity | 5 and 5 | 10 |
| Most informative next test against Classic greedy TDD | M_io, sensitivity | 5 and 5 | 18 |
| Held-out TDD against Classic greedy TDD | adjusted mutation | 5 and 5 | 0 |
| Coarse-to-fine TDD against Classic greedy TDD | adjusted mutation | 5 and 5 | 11 |
| Most informative next test against Classic greedy TDD | adjusted mutation | 5 and 5 | 15 |
| Held-out TDD against Classic greedy TDD | CQP accepted, sensitivity. Finals alpha below 0.70: simple constructs 0.52 | 5 and 5 | 24 |
| Coarse-to-fine TDD against Classic greedy TDD | CQP accepted, sensitivity. Finals alpha below 0.70: simple constructs 0.52 | 5 and 5 | 21 |
| Most informative next test against Classic greedy TDD | CQP accepted, sensitivity. Finals alpha below 0.70: simple constructs 0.52 | 5 and 5 | 21.5 |
| Contrast | p | Holm p |
|---|---|---|
| Held-out TDD against Classic greedy TDD | 0.841 | 1.000 |
| Coarse-to-fine TDD against Classic greedy TDD | 0.548 | 1.000 |
| Most informative next test against Classic greedy TDD | 0.421 | 1.000 |
| Held-out TDD against Classic greedy TDD | 0.690 | 1.000 |
| Coarse-to-fine TDD against Classic greedy TDD | 0.690 | 1.000 |
| Most informative next test against Classic greedy TDD | 0.310 | 0.929 |
| Held-out TDD against Classic greedy TDD | 0.008 | 0.024 |
| Coarse-to-fine TDD against Classic greedy TDD | 1.000 | 1.000 |
| Most informative next test against Classic greedy TDD | 1.000 | 1.000 |
| Held-out TDD against Classic greedy TDD | 0.024 | 0.071 |
| Coarse-to-fine TDD against Classic greedy TDD | 0.119 | 0.175 |
| Most informative next test against Classic greedy TDD | 0.087 | 0.175 |
Consistency table
Exploratory, since only the replicated methods enter it. It pools every replicate of runs rep-1 and rep-2 per method. Each cell is a relative spread, the method's mean absolute deviation from its own median over the mean deviation of the methods off the metric's bound. Lower is more consistent and 1 is the average. At bound means at least 60 percent of builds sit at the metric's bound. Not comparable means fewer than two methods sit off it. C is the mean over unadjusted mutation, classes and cost to completion. The tests are seeded permutation tests with 10,000 draws that shuffle each build's deviations from its own method's median. Tag: recomputed.
| Method | Adjusted mutation | Unadjusted mutation | Classes | Cost to completion |
|---|---|---|---|---|
| Functions first, classes later | 1.37 | 1.98 | 1.14 | 0.30 |
| Noun-verb map | 1.61 | 2.18 | 1.43 | 1.34 |
| Anchor Engineering | at bound | 0.21 | 1.43 | 1.53 |
| Classic greedy TDD | 0.33 | 0.75 | 0.86 | 1.00 |
| Held-out TDD | 1.36 | 1.73 | 0.57 | 0.59 |
| Coarse-to-fine TDD | at bound | 0.25 | 1.71 | 1.23 |
| Most informative next test | at bound | 0.14 | 0.00 | 1.01 |
| Method | Strict PMD violations | CQP accepted | Consistency score C |
|---|---|---|---|
| Functions first, classes later | at bound | 1.68 | 1.14 |
| Noun-verb map | at bound | 0.00 | 1.65 |
| Anchor Engineering | at bound | 0.84 | 1.06 |
| Classic greedy TDD | 0.97 | 1.47 | 0.87 |
| Held-out TDD | 1.12 | 0.84 | 0.96 |
| Coarse-to-fine TDD | 1.22 | 1.26 | 1.06 |
| Most informative next test | 0.71 | 0.42 | 0.38 |
| Metric | Statistic | Permutation p | Holm p |
|---|---|---|---|
| Unadjusted mutation | Brown-Forsythe F 1.80 | 0.113 | 0.339 |
| Classes | Brown-Forsythe F 0.77 | 0.613 | 0.613 |
| Cost to completion | Brown-Forsythe F 1.35 | 0.259 | 0.518 |
| Consistency score C | variance 0.121 | 0.464 | - |
Replicate table (rep-1)
Every replicate build of run rep-1 in clone order. Build names the best, median and worst of each method by adjusted mutation, then lower cost to completion. Tag: recomputed.
Replicates (rep-1), 20 rows and 10 columns
| Replicate | Method | Adjusted mutation | Unadjusted mutation |
|---|---|---|---|
| rep-1-r01 | Classic greedy TDD | 0.992 | 0.944 |
| rep-1-r02 | Anchor Engineering | 1.000 | 0.955 |
| rep-1-r03 | Functions first, classes later | 0.848 | 0.763 |
| rep-1-r04 | Anchor Engineering | 1.000 | 0.966 |
| rep-1-r05 | Functions first, classes later | 0.992 | 0.910 |
| rep-1-r06 | Anchor Engineering | 1.000 | 0.959 |
| rep-1-r07 | Noun-verb map | 0.989 | 0.940 |
| rep-1-r08 | Anchor Engineering | 1.000 | 0.962 |
| rep-1-r09 | Classic greedy TDD | 0.993 | 0.965 |
| rep-1-r10 | Noun-verb map | 0.776 | 0.750 |
| rep-1-r11 | Functions first, classes later | 1.000 | 0.950 |
| rep-1-r12 | Functions first, classes later | 0.927 | 0.891 |
| rep-1-r13 | Noun-verb map | 0.835 | 0.783 |
| rep-1-r14 | Anchor Engineering | 1.000 | 0.941 |
| rep-1-r15 | Classic greedy TDD | 0.981 | 0.922 |
| rep-1-r16 | Functions first, classes later | 0.841 | 0.795 |
| rep-1-r17 | Classic greedy TDD | 0.903 | 0.861 |
| rep-1-r18 | Classic greedy TDD | 0.984 | 0.920 |
| rep-1-r19 | Noun-verb map | 0.964 | 0.922 |
| rep-1-r20 | Noun-verb map | 0.978 | 0.927 |
| Replicate | Classes | Cost to completion | CQP |
|---|---|---|---|
| rep-1-r01 | 1 | 1,179,133 | 13 |
| rep-1-r02 | 5 | 1,225,495 | 15 |
| rep-1-r03 | 6 | 1,367,029 | 17 |
| rep-1-r04 | 7 | 830,836 | 16 |
| rep-1-r05 | 7 | 1,337,295 | 15 |
| rep-1-r06 | 5 | 938,885 | 16 |
| rep-1-r07 | 15 | 1,174,050 | 16 |
| rep-1-r08 | 6 | 1,039,535 | 17 |
| rep-1-r09 | 1 | 1,033,995 | 14 |
| rep-1-r10 | 19 | 1,615,015 | 16 |
| rep-1-r11 | 5 | 1,339,770 | 16 |
| rep-1-r12 | 7 | 1,271,901 | 14 |
| rep-1-r13 | 16 | 1,334,836 | 16 |
| rep-1-r14 | 8 | 1,271,702 | 16 |
| rep-1-r15 | 3 | 1,024,817 | 16 |
| rep-1-r16 | 5 | 1,385,387 | 13 |
| rep-1-r17 | 2 | 1,147,883 | 14 |
| rep-1-r18 | 3 | 856,188 | 16 |
| rep-1-r19 | 15 | 1,260,350 | 16 |
| rep-1-r20 | 15 | 1,454,264 | 16 |
| Replicate | CQP accepted | Strict PMD violations | Build |
|---|---|---|---|
| rep-1-r01 | 10 | 10 | - |
| rep-1-r02 | 11 | 0 | - |
| rep-1-r03 | 13 | 0 | - |
| rep-1-r04 | 12 | 0 | best |
| rep-1-r05 | 11 | 0 | - |
| rep-1-r06 | 12 | 0 | - |
| rep-1-r07 | 12 | 0 | best |
| rep-1-r08 | 13 | 0 | median |
| rep-1-r09 | 11 | 7 | best |
| rep-1-r10 | 12 | 0 | worst |
| rep-1-r11 | 12 | 0 | best |
| rep-1-r12 | 11 | 0 | median |
| rep-1-r13 | 12 | 0 | - |
| rep-1-r14 | 12 | 0 | worst |
| rep-1-r15 | 12 | 4 | - |
| rep-1-r16 | 10 | 0 | worst |
| rep-1-r17 | 11 | 9 | worst |
| rep-1-r18 | 12 | 4 | median |
| rep-1-r19 | 12 | 0 | median |
| rep-1-r20 | 12 | 0 | - |
Replicate table (rep-2)
Every replicate build of run rep-2 in clone order. Build names the best, median and worst of each method by adjusted mutation, then lower cost to completion. Tag: recomputed.
Replicates (rep-2), 20 rows and 10 columns
| Replicate | Method | Adjusted mutation | Unadjusted mutation |
|---|---|---|---|
| rep-2-r01 | Classic greedy TDD | 1.000 | 0.948 |
| rep-2-r02 | Classic greedy TDD | 1.000 | 0.961 |
| rep-2-r03 | Coarse-to-fine TDD | 0.993 | 0.952 |
| rep-2-r04 | Held-out TDD | 0.777 | 0.725 |
| rep-2-r05 | Most informative next test | 1.000 | 0.927 |
| rep-2-r06 | Coarse-to-fine TDD | 1.000 | 0.928 |
| rep-2-r07 | Classic greedy TDD | 1.000 | 0.913 |
| rep-2-r08 | Held-out TDD | 0.777 | 0.734 |
| rep-2-r09 | Most informative next test | 1.000 | 0.918 |
| rep-2-r10 | Classic greedy TDD | 1.000 | 0.930 |
| rep-2-r11 | Classic greedy TDD | 0.988 | 0.958 |
| rep-2-r12 | Coarse-to-fine TDD | 1.000 | 0.949 |
| rep-2-r13 | Coarse-to-fine TDD | 0.990 | 0.935 |
| rep-2-r14 | Held-out TDD | 0.736 | 0.709 |
| rep-2-r15 | Most informative next test | 1.000 | 0.932 |
| rep-2-r16 | Coarse-to-fine TDD | 1.000 | 0.944 |
| rep-2-r17 | Held-out TDD | 0.723 | 0.688 |
| rep-2-r18 | Most informative next test | 1.000 | 0.923 |
| rep-2-r19 | Most informative next test | 1.000 | 0.919 |
| rep-2-r20 | Held-out TDD | 0.983 | 0.927 |
| Replicate | Classes | Cost to completion | CQP |
|---|---|---|---|
| rep-2-r01 | 1 | 1,011,641 | 13 |
| rep-2-r02 | 1 | 946,961 | 14 |
| rep-2-r03 | 7 | 967,802 | 15 |
| rep-2-r04 | 3 | 1,021,062 | 17 |
| rep-2-r05 | 3 | 878,599 | 15 |
| rep-2-r06 | 3 | 926,033 | 16 |
| rep-2-r07 | 1 | 1,256,020 | 13 |
| rep-2-r08 | 5 | 904,350 | 16 |
| rep-2-r09 | 3 | 927,123 | 16 |
| rep-2-r10 | 2 | 1,079,012 | 16 |
| rep-2-r11 | 1 | 908,035 | 13 |
| rep-2-r12 | 5 | 1,089,432 | 15 |
| rep-2-r13 | 3 | 794,216 | 16 |
| rep-2-r14 | 3 | 973,901 | 16 |
| rep-2-r15 | 3 | 1,068,727 | 16 |
| rep-2-r16 | 3 | 681,209 | 17 |
| rep-2-r17 | 3 | 926,018 | 17 |
| rep-2-r18 | 3 | 1,109,946 | 16 |
| rep-2-r19 | 3 | 1,175,603 | 16 |
| rep-2-r20 | 3 | 1,088,356 | 17 |
| Replicate | CQP accepted | Strict PMD violations | Build |
|---|---|---|---|
| rep-2-r01 | 10 | 6 | - |
| rep-2-r02 | 11 | 4 | best |
| rep-2-r03 | 11 | 8 | - |
| rep-2-r04 | 13 | 12 | median |
| rep-2-r05 | 11 | 2 | best |
| rep-2-r06 | 12 | 5 | - |
| rep-2-r07 | 10 | 7 | - |
| rep-2-r08 | 12 | 6 | - |
| rep-2-r09 | 12 | 5 | - |
| rep-2-r10 | 12 | 4 | median |
| rep-2-r11 | 10 | 4 | worst |
| rep-2-r12 | 11 | 2 | median |
| rep-2-r13 | 12 | 4 | worst |
| rep-2-r14 | 12 | 7 | - |
| rep-2-r15 | 12 | 4 | median |
| rep-2-r16 | 13 | 10 | best |
| rep-2-r17 | 13 | 8 | worst |
| rep-2-r18 | 12 | 6 | - |
| rep-2-r19 | 12 | 2 | worst |
| rep-2-r20 | 13 | 3 | best |
Each method page holds its builds, records and conclusion.
- Classic greedy TDD: method page
- Noun-verb map: method page
- Mapped-path TDD: method page
- Functions first, classes later: method page
- Transformation Priority Premise TDD: method page
- Coarse-to-fine TDD: method page
- Annealed refactoring: method page
- Multi-start TDD: method page
- Most informative next test: method page
- Property-based TDD: method page
- Held-out TDD: method page
- Anchor Engineering: method page