Kalah TDD study · Part 7 of 29 · Results

Results: planned contrasts

Planned contrasts over five replicates per arm in runs rep-1 and rep-2.

Source runs rep-1, rep-2 and full-1 at tdd-experiment c3dd988

Contents

Adjusted mutation per replicate in rep-1 is a headline figure, so it is on the study landing page. This page holds the other planned contrast figures.

Holm p across the planned contrasts

The contrast cells of rep-1 hold 2 and rep-2 hold 6 planned contrasts, each an exact Mann-Whitney U test with Holm correction. Figure 7.1 shows where each Holm p falls against 0.05.

Figure 7.1

Holm p of each planned contrast

Adjusted mutationMHolm p 0.05rep-1: Functions first, classes later against Noun-verb maprep-1: Anchor Engineering against Classic greedy TDDrep-2: Held-out TDD against Classic greedy TDDrep-2: Coarse-to-fine TDD against Classic greedy TDDrep-2: Most informative next test against Classic greedy TDDrep-1 adjusted mutation, Functions first, classes later against Noun-verb map: U 16, Holm p 0.548rep-1 adjusted mutation, Anchor Engineering against Classic greedy TDD: U 25, Holm p 0.016rep-2 M, Held-out TDD against Classic greedy TDD: U 11, Holm p 1.000rep-2 M, Coarse-to-fine TDD against Classic greedy TDD: U 9, Holm p 1.000rep-2 M, Most informative next test against Classic greedy TDD: U 17, Holm p 1.000rep-2 adjusted mutation, Held-out TDD against Classic greedy TDD: U 0, Holm p 0.024rep-2 adjusted mutation, Coarse-to-fine TDD against Classic greedy TDD: U 11, Holm p 1.000rep-2 adjusted mutation, Most informative next test against Classic greedy TDD: U 15, Holm p 1.00000.20.40.60.81Holm p

The 8 planned contrasts of rep-1 and rep-2, read from the contrast cells in the sections below. Both sides of each contrast agree and each Holm p is in its section lead. Filled marks fall below the dashed line at 0.05 and hollow marks do not. Tag: recomputed.

Data
Text version: Holm p of each planned contrast
  1. rep-1 adjusted mutation, Functions first, classes later against Noun-verb map: U 16, Holm p 0.548
  2. rep-1 adjusted mutation, Anchor Engineering against Classic greedy TDD: U 25, Holm p 0.016
  3. rep-2 M, Held-out TDD against Classic greedy TDD: U 11, Holm p 1.000
  4. rep-2 M, Coarse-to-fine TDD against Classic greedy TDD: U 9, Holm p 1.000
  5. rep-2 M, Most informative next test against Classic greedy TDD: U 17, Holm p 1.000
  6. rep-2 adjusted mutation, Held-out TDD against Classic greedy TDD: U 0, Holm p 0.024
  7. rep-2 adjusted mutation, Coarse-to-fine TDD against Classic greedy TDD: U 11, Holm p 1.000
  8. rep-2 adjusted mutation, Most informative next test against Classic greedy TDD: U 15, Holm p 1.000

Planned contrasts with U and Holm p

U is that of the method named first. Tag: recomputed.

RunOutcomeContrastUHolm p
rep-1Adjusted mutationFunctions first, classes later against Noun-verb map160.548
rep-1Adjusted mutationAnchor Engineering against Classic greedy TDD250.016
rep-2MHeld-out TDD against Classic greedy TDD111.000
rep-2MCoarse-to-fine TDD against Classic greedy TDD91.000
rep-2MMost informative next test against Classic greedy TDD171.000
rep-2Adjusted mutationHeld-out TDD against Classic greedy TDD00.024
rep-2Adjusted mutationCoarse-to-fine TDD against Classic greedy TDD111.000
rep-2Adjusted mutationMost informative next test against Classic greedy TDD151.000

Planned contrasts (rep-1)

Both contrasts were fixed before any rep-1 build, each on adjusted mutation score by an exact Mann-Whitney U test with Holm correction over the two. Anchor Engineering against Classic greedy TDD gave U 25 and Holm p 0.016. Functions first, classes later against Noun-verb map gave U 16 and Holm p 0.548. The CQP sensitivity check separated neither pair, Holm p 0.333 for both. Tag: recomputed.

MethodReplicatesMedian adjusted mutationMedian classesMedian cost to completion
Functions first, classes later50.92761,339,770
Noun-verb map50.964151,334,836
Anchor Engineering51.00061,039,535
Classic greedy TDD50.98421,033,995
MethodMedian CQPPlanned contrastRepresentative build
Functions first, classes later15against Noun-verb map: U 16, Holm p 0.548rep-1-r12
Noun-verb map16against Functions first, classes later: U 9, Holm p 0.548rep-1-r19
Anchor Engineering16against Classic greedy TDD: U 25, Holm p 0.016rep-1-r08
Classic greedy TDD14against Anchor Engineering: U 0, Holm p 0.016rep-1-r18

Figure 7.2

Classes per replicate, rep-1

Informs Q6.

Medians over each method's replicates: Functions first, classes later 6, Noun-verb map 15, Anchor Engineering 6, Classic greedy TDD 2.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Hollow diamonds are the single survey build of each method from the earlier full-1 run, a different batch. They stay out of every median, span and test. Tag: recomputed.

05101520Classes in src/Functions first, classes laterFunctions first, classes later: median 6rep-1-r03: 6 classesrep-1-r05: 7 classesrep-1-r11: 5 classesrep-1-r12: 7 classesrep-1-r12: 7 classesrep-1-r16: 5 classesfunctions-first, full-1 survey build: 6 classesFunctions first, classes later: median 6Noun-verb mapNoun-verb map: median 15rep-1-r07: 15 classesrep-1-r10: 19 classesrep-1-r13: 16 classesrep-1-r19: 15 classesrep-1-r19: 15 classesrep-1-r20: 15 classesnoun-verb, full-1 survey build: 14 classesNoun-verb map: median 15Anchor EngineeringAnchor Engineering: median 6rep-1-r02: 5 classesrep-1-r04: 7 classesrep-1-r06: 5 classesrep-1-r08: 6 classesrep-1-r08: 6 classesrep-1-r14: 8 classesAnchor Engineering: median 6Classic greedy TDDClassic greedy TDD: median 2rep-1-r01: 1 classesrep-1-r09: 1 classesrep-1-r15: 3 classesrep-1-r17: 2 classesrep-1-r18: 3 classesrep-1-r18: 3 classesclassic, full-1 survey build: 1 classesClassic greedy TDD: median 2
Functions first, classes laterNoun-verb mapAnchor EngineeringClassic greedy TDDSurvey build, earlier batch
Data
Table view, 23 rows and 4 columns
MethodBuildSourceClasses
Functions first, classes laterrep-1-r03rep-1 replicate6
Functions first, classes laterrep-1-r05rep-1 replicate7
Functions first, classes laterrep-1-r11rep-1 replicate5
Functions first, classes laterrep-1-r12rep-1 replicate, representative7
Functions first, classes laterrep-1-r16rep-1 replicate5
Noun-verb maprep-1-r07rep-1 replicate15
Noun-verb maprep-1-r10rep-1 replicate19
Noun-verb maprep-1-r13rep-1 replicate16
Noun-verb maprep-1-r19rep-1 replicate, representative15
Noun-verb maprep-1-r20rep-1 replicate15
Anchor Engineeringrep-1-r02rep-1 replicate5
Anchor Engineeringrep-1-r04rep-1 replicate7
Anchor Engineeringrep-1-r06rep-1 replicate5
Anchor Engineeringrep-1-r08rep-1 replicate, representative6
Anchor Engineeringrep-1-r14rep-1 replicate8
Classic greedy TDDrep-1-r01rep-1 replicate1
Classic greedy TDDrep-1-r09rep-1 replicate1
Classic greedy TDDrep-1-r15rep-1 replicate3
Classic greedy TDDrep-1-r17rep-1 replicate2
Classic greedy TDDrep-1-r18rep-1 replicate, representative3
Functions first, classes laterfunctions-firstfull-1 survey build6
Noun-verb mapnoun-verbfull-1 survey build14
Classic greedy TDDclassicfull-1 survey build1

Figure 7.3

Cost to completion per replicate, rep-1

Informs Q8.

Medians over each method's replicates: Functions first, classes later 1,339,770, Noun-verb map 1,334,836, Anchor Engineering 1,039,535, Classic greedy TDD 1,033,995.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Hollow diamonds are the single survey build of each method from the earlier full-1 run, a different batch. They stay out of every median, span and test. Tag: recomputed.

00.511.52Weighted tokens to completion, millionsFunctions first, classes laterFunctions first, classes later: median 1,339,770rep-1-r03: 1,367,029 cost to completionrep-1-r05: 1,337,295 cost to completionrep-1-r11: 1,339,770 cost to completionrep-1-r12: 1,271,901 cost to completionrep-1-r12: 1,271,901 cost to completionrep-1-r16: 1,385,387 cost to completionfunctions-first, full-1 survey build: 1,195,424 cost to completionFunctions first, classes later: median 1,339,770Noun-verb mapNoun-verb map: median 1,334,836rep-1-r07: 1,174,050 cost to completionrep-1-r10: 1,615,015 cost to completionrep-1-r13: 1,334,836 cost to completionrep-1-r19: 1,260,350 cost to completionrep-1-r19: 1,260,350 cost to completionrep-1-r20: 1,454,264 cost to completionnoun-verb, full-1 survey build: 918,336 cost to completionNoun-verb map: median 1,334,836Anchor EngineeringAnchor Engineering: median 1,039,535rep-1-r02: 1,225,495 cost to completionrep-1-r04: 830,836 cost to completionrep-1-r06: 938,885 cost to completionrep-1-r08: 1,039,535 cost to completionrep-1-r08: 1,039,535 cost to completionrep-1-r14: 1,271,702 cost to completionAnchor Engineering: median 1,039,535Classic greedy TDDClassic greedy TDD: median 1,033,995rep-1-r01: 1,179,133 cost to completionrep-1-r09: 1,033,995 cost to completionrep-1-r15: 1,024,817 cost to completionrep-1-r17: 1,147,883 cost to completionrep-1-r18: 856,188 cost to completionrep-1-r18: 856,188 cost to completionclassic, full-1 survey build: 804,727 cost to completionClassic greedy TDD: median 1,033,995
Functions first, classes laterNoun-verb mapAnchor EngineeringClassic greedy TDDSurvey build, earlier batch
Data
Table view, 23 rows and 4 columns
MethodBuildSourceCost to completion
Functions first, classes laterrep-1-r03rep-1 replicate1,367,029
Functions first, classes laterrep-1-r05rep-1 replicate1,337,295
Functions first, classes laterrep-1-r11rep-1 replicate1,339,770
Functions first, classes laterrep-1-r12rep-1 replicate, representative1,271,901
Functions first, classes laterrep-1-r16rep-1 replicate1,385,387
Noun-verb maprep-1-r07rep-1 replicate1,174,050
Noun-verb maprep-1-r10rep-1 replicate1,615,015
Noun-verb maprep-1-r13rep-1 replicate1,334,836
Noun-verb maprep-1-r19rep-1 replicate, representative1,260,350
Noun-verb maprep-1-r20rep-1 replicate1,454,264
Anchor Engineeringrep-1-r02rep-1 replicate1,225,495
Anchor Engineeringrep-1-r04rep-1 replicate830,836
Anchor Engineeringrep-1-r06rep-1 replicate938,885
Anchor Engineeringrep-1-r08rep-1 replicate, representative1,039,535
Anchor Engineeringrep-1-r14rep-1 replicate1,271,702
Classic greedy TDDrep-1-r01rep-1 replicate1,179,133
Classic greedy TDDrep-1-r09rep-1 replicate1,033,995
Classic greedy TDDrep-1-r15rep-1 replicate1,024,817
Classic greedy TDDrep-1-r17rep-1 replicate1,147,883
Classic greedy TDDrep-1-r18rep-1 replicate, representative856,188
Functions first, classes laterfunctions-firstfull-1 survey build1,195,424
Noun-verb mapnoun-verbfull-1 survey build918,336
Classic greedy TDDclassicfull-1 survey build804,727

Figure 7.4

CQP per replicate, rep-1

Informs Q12.

Medians over each method's replicates: Functions first, classes later 15, Noun-verb map 16, Anchor Engineering 16, Classic greedy TDD 14.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Tag: recomputed.

05101520CQP, 4 to 20Functions first, classes laterFunctions first, classes later: median 15rep-1-r03: 17 CQPrep-1-r05: 15 CQPrep-1-r11: 16 CQPrep-1-r12: 14 CQPrep-1-r12: 14 CQPrep-1-r16: 13 CQPFunctions first, classes later: median 15Noun-verb mapNoun-verb map: median 16rep-1-r07: 16 CQPrep-1-r10: 16 CQPrep-1-r13: 16 CQPrep-1-r19: 16 CQPrep-1-r19: 16 CQPrep-1-r20: 16 CQPNoun-verb map: median 16Anchor EngineeringAnchor Engineering: median 16rep-1-r02: 15 CQPrep-1-r04: 16 CQPrep-1-r06: 16 CQPrep-1-r08: 17 CQPrep-1-r08: 17 CQPrep-1-r14: 16 CQPAnchor Engineering: median 16Classic greedy TDDClassic greedy TDD: median 14rep-1-r01: 13 CQPrep-1-r09: 14 CQPrep-1-r15: 16 CQPrep-1-r17: 14 CQPrep-1-r18: 16 CQPrep-1-r18: 16 CQPClassic greedy TDD: median 14
Functions first, classes laterNoun-verb mapAnchor EngineeringClassic greedy TDD
Data
Table view, 20 rows and 4 columns
MethodBuildSourceCQP
Functions first, classes laterrep-1-r03rep-1 replicate17
Functions first, classes laterrep-1-r05rep-1 replicate15
Functions first, classes laterrep-1-r11rep-1 replicate16
Functions first, classes laterrep-1-r12rep-1 replicate, representative14
Functions first, classes laterrep-1-r16rep-1 replicate13
Noun-verb maprep-1-r07rep-1 replicate16
Noun-verb maprep-1-r10rep-1 replicate16
Noun-verb maprep-1-r13rep-1 replicate16
Noun-verb maprep-1-r19rep-1 replicate, representative16
Noun-verb maprep-1-r20rep-1 replicate16
Anchor Engineeringrep-1-r02rep-1 replicate15
Anchor Engineeringrep-1-r04rep-1 replicate16
Anchor Engineeringrep-1-r06rep-1 replicate16
Anchor Engineeringrep-1-r08rep-1 replicate, representative17
Anchor Engineeringrep-1-r14rep-1 replicate16
Classic greedy TDDrep-1-r01rep-1 replicate13
Classic greedy TDDrep-1-r09rep-1 replicate14
Classic greedy TDDrep-1-r15rep-1 replicate16
Classic greedy TDDrep-1-r17rep-1 replicate14
Classic greedy TDDrep-1-r18rep-1 replicate, representative16

Planned contrasts (rep-2)

Three contrasts were fixed before any rep-2 build, each survey leader against Classic greedy TDD on M by an exact Mann-Whitney U test with Holm correction over the three. Held-out TDD gave U 11 and p 0.841, Coarse-to-fine TDD U 9 and p 0.548 and Most informative next test U 17 and p 0.421, each with Holm p 1.000. The M_io sensitivity check gave p 0.690, 0.690 and 0.310 and Holm p 1.000, 1.000 and 0.929. The planned quality outcome fell back to adjusted mutation score because Simple Constructs reached alpha 0.52 on the rep-2 finals, below 0.7. Held-out TDD scored lower than Classic greedy TDD, a median of 0.777 against 1.000, with U 0, p 0.008 and Holm p 0.024. The other two leaders gave Holm p 1.000. The CQP sensitivity check gave Held-out TDD a median of 13 against 10, U 24, p 0.024 and Holm p 0.071, and the other two Holm p 0.175. CQP accepted sums the three principles accepted on the full-1 finals, and the rep-2 rating accepts none of them, as the Limits explain. Tag: recomputed.

Held-out TDD. Replicates: 5. Median adjusted mutation: 0.777. Median classes: 3. Median cost to completion: 973,901. Median CQP: 17. Median M: 607,308. Representative build: rep-2-r04.

Planned contrast: M against Classic greedy TDD: U 11, Holm p 1.000; adjusted mutation against Classic greedy TDD: U 0, Holm p 0.024

Classic greedy TDD. Replicates: 5. Median adjusted mutation: 1.000. Median classes: 1. Median cost to completion: 1,011,641. Median CQP: 13. Median M: 601,555. Representative build: rep-2-r10.

Planned contrast: M against Held-out TDD: U 14, Holm p 1.000; M against Coarse-to-fine TDD: U 16, Holm p 1.000; M against Most informative next test: U 8, Holm p 1.000; adjusted mutation against Held-out TDD: U 25, Holm p 0.024; adjusted mutation against Coarse-to-fine TDD: U 14, Holm p 1.000; adjusted mutation against Most informative next test: U 10, Holm p 1.000

Coarse-to-fine TDD. Replicates: 5. Median adjusted mutation: 1.000. Median classes: 3. Median cost to completion: 926,033. Median CQP: 16. Median M: 559,186. Representative build: rep-2-r12.

Planned contrast: M against Classic greedy TDD: U 9, Holm p 1.000; adjusted mutation against Classic greedy TDD: U 11, Holm p 1.000

Most informative next test. Replicates: 5. Median adjusted mutation: 1.000. Median classes: 3. Median cost to completion: 1,068,727. Median CQP: 16. Median M: 735,111. Representative build: rep-2-r15.

Planned contrast: M against Classic greedy TDD: U 17, Holm p 1.000; adjusted mutation against Classic greedy TDD: U 15, Holm p 1.000

Figure 7.5

Adjusted mutation per replicate, rep-2

Informs Q12.

Medians over each method's replicates: Held-out TDD 0.777, Classic greedy TDD 1.000, Coarse-to-fine TDD 1.000, Most informative next test 1.000.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Hollow diamonds are the single survey build of each method from the earlier full-1 run, a different batch. They stay out of every median, span and test. Tag: recomputed.

0.70.80.91Adjusted mutation scoreHeld-out TDDHeld-out TDD: median 0.777rep-2-r04: 0.777 adjusted mutationrep-2-r04: 0.777 adjusted mutationrep-2-r08: 0.777 adjusted mutationrep-2-r14: 0.736 adjusted mutationrep-2-r17: 0.723 adjusted mutationrep-2-r20: 0.983 adjusted mutationheld-out, full-1 survey build: 0.794 adjusted mutationHeld-out TDD: median 0.777Classic greedy TDDClassic greedy TDD: median 1.000rep-2-r01: 1.000 adjusted mutationrep-2-r02: 1.000 adjusted mutationrep-2-r07: 1.000 adjusted mutationrep-2-r10: 1.000 adjusted mutationrep-2-r10: 1.000 adjusted mutationrep-2-r11: 0.988 adjusted mutationclassic, full-1 survey build: 1.000 adjusted mutationClassic greedy TDD: median 1.000Coarse-to-fine TDDCoarse-to-fine TDD: median 1.000rep-2-r03: 0.993 adjusted mutationrep-2-r06: 1.000 adjusted mutationrep-2-r12: 1.000 adjusted mutationrep-2-r12: 1.000 adjusted mutationrep-2-r13: 0.990 adjusted mutationrep-2-r16: 1.000 adjusted mutationcoarse-to-fine, full-1 survey build: 1.000 adjusted mutationCoarse-to-fine TDD: median 1.000Most informative next testMost informative next test: median 1.000rep-2-r05: 1.000 adjusted mutationrep-2-r09: 1.000 adjusted mutationrep-2-r15: 1.000 adjusted mutationrep-2-r15: 1.000 adjusted mutationrep-2-r18: 1.000 adjusted mutationrep-2-r19: 1.000 adjusted mutationmutation-guided, full-1 survey build: 1.000 adjusted mutationMost informative next test: median 1.000
Held-out TDDClassic greedy TDDCoarse-to-fine TDDMost informative next testSurvey build, earlier batch
Data
Table view, 24 rows and 4 columns
MethodBuildSourceAdjusted mutation
Held-out TDDrep-2-r04rep-2 replicate, representative0.777
Held-out TDDrep-2-r08rep-2 replicate0.777
Held-out TDDrep-2-r14rep-2 replicate0.736
Held-out TDDrep-2-r17rep-2 replicate0.723
Held-out TDDrep-2-r20rep-2 replicate0.983
Classic greedy TDDrep-2-r01rep-2 replicate1.000
Classic greedy TDDrep-2-r02rep-2 replicate1.000
Classic greedy TDDrep-2-r07rep-2 replicate1.000
Classic greedy TDDrep-2-r10rep-2 replicate, representative1.000
Classic greedy TDDrep-2-r11rep-2 replicate0.988
Coarse-to-fine TDDrep-2-r03rep-2 replicate0.993
Coarse-to-fine TDDrep-2-r06rep-2 replicate1.000
Coarse-to-fine TDDrep-2-r12rep-2 replicate, representative1.000
Coarse-to-fine TDDrep-2-r13rep-2 replicate0.990
Coarse-to-fine TDDrep-2-r16rep-2 replicate1.000
Most informative next testrep-2-r05rep-2 replicate1.000
Most informative next testrep-2-r09rep-2 replicate1.000
Most informative next testrep-2-r15rep-2 replicate, representative1.000
Most informative next testrep-2-r18rep-2 replicate1.000
Most informative next testrep-2-r19rep-2 replicate1.000
Held-out TDDheld-outfull-1 survey build0.794
Classic greedy TDDclassicfull-1 survey build1.000
Coarse-to-fine TDDcoarse-to-finefull-1 survey build1.000
Most informative next testmutation-guidedfull-1 survey build1.000

Figure 7.6

Classes per replicate, rep-2

Informs Q6.

Medians over each method's replicates: Held-out TDD 3, Classic greedy TDD 1, Coarse-to-fine TDD 3, Most informative next test 3.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Hollow diamonds are the single survey build of each method from the earlier full-1 run, a different batch. They stay out of every median, span and test. Tag: recomputed.

02468Classes in src/Held-out TDDHeld-out TDD: median 3rep-2-r04: 3 classesrep-2-r04: 3 classesrep-2-r08: 5 classesrep-2-r14: 3 classesrep-2-r17: 3 classesrep-2-r20: 3 classesheld-out, full-1 survey build: 5 classesHeld-out TDD: median 3Classic greedy TDDClassic greedy TDD: median 1rep-2-r01: 1 classesrep-2-r02: 1 classesrep-2-r07: 1 classesrep-2-r10: 2 classesrep-2-r10: 2 classesrep-2-r11: 1 classesclassic, full-1 survey build: 1 classesClassic greedy TDD: median 1Coarse-to-fine TDDCoarse-to-fine TDD: median 3rep-2-r03: 7 classesrep-2-r06: 3 classesrep-2-r12: 5 classesrep-2-r12: 5 classesrep-2-r13: 3 classesrep-2-r16: 3 classescoarse-to-fine, full-1 survey build: 3 classesCoarse-to-fine TDD: median 3Most informative next testMost informative next test: median 3rep-2-r05: 3 classesrep-2-r09: 3 classesrep-2-r15: 3 classesrep-2-r15: 3 classesrep-2-r18: 3 classesrep-2-r19: 3 classesmutation-guided, full-1 survey build: 3 classesMost informative next test: median 3
Held-out TDDClassic greedy TDDCoarse-to-fine TDDMost informative next testSurvey build, earlier batch
Data
Table view, 24 rows and 4 columns
MethodBuildSourceClasses
Held-out TDDrep-2-r04rep-2 replicate, representative3
Held-out TDDrep-2-r08rep-2 replicate5
Held-out TDDrep-2-r14rep-2 replicate3
Held-out TDDrep-2-r17rep-2 replicate3
Held-out TDDrep-2-r20rep-2 replicate3
Classic greedy TDDrep-2-r01rep-2 replicate1
Classic greedy TDDrep-2-r02rep-2 replicate1
Classic greedy TDDrep-2-r07rep-2 replicate1
Classic greedy TDDrep-2-r10rep-2 replicate, representative2
Classic greedy TDDrep-2-r11rep-2 replicate1
Coarse-to-fine TDDrep-2-r03rep-2 replicate7
Coarse-to-fine TDDrep-2-r06rep-2 replicate3
Coarse-to-fine TDDrep-2-r12rep-2 replicate, representative5
Coarse-to-fine TDDrep-2-r13rep-2 replicate3
Coarse-to-fine TDDrep-2-r16rep-2 replicate3
Most informative next testrep-2-r05rep-2 replicate3
Most informative next testrep-2-r09rep-2 replicate3
Most informative next testrep-2-r15rep-2 replicate, representative3
Most informative next testrep-2-r18rep-2 replicate3
Most informative next testrep-2-r19rep-2 replicate3
Held-out TDDheld-outfull-1 survey build5
Classic greedy TDDclassicfull-1 survey build1
Coarse-to-fine TDDcoarse-to-finefull-1 survey build3
Most informative next testmutation-guidedfull-1 survey build3

Figure 7.7

Cost to completion per replicate, rep-2

Informs Q8.

Medians over each method's replicates: Held-out TDD 973,901, Classic greedy TDD 1,011,641, Coarse-to-fine TDD 926,033, Most informative next test 1,068,727.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Hollow diamonds are the single survey build of each method from the earlier full-1 run, a different batch. They stay out of every median, span and test. Tag: recomputed.

00.511.5Weighted tokens to completion, millionsHeld-out TDDHeld-out TDD: median 973,901rep-2-r04: 1,021,062 cost to completionrep-2-r04: 1,021,062 cost to completionrep-2-r08: 904,350 cost to completionrep-2-r14: 973,901 cost to completionrep-2-r17: 926,018 cost to completionrep-2-r20: 1,088,356 cost to completionheld-out, full-1 survey build: 801,898 cost to completionHeld-out TDD: median 973,901Classic greedy TDDClassic greedy TDD: median 1,011,641rep-2-r01: 1,011,641 cost to completionrep-2-r02: 946,961 cost to completionrep-2-r07: 1,256,020 cost to completionrep-2-r10: 1,079,012 cost to completionrep-2-r10: 1,079,012 cost to completionrep-2-r11: 908,035 cost to completionclassic, full-1 survey build: 804,727 cost to completionClassic greedy TDD: median 1,011,641Coarse-to-fine TDDCoarse-to-fine TDD: median 926,033rep-2-r03: 967,802 cost to completionrep-2-r06: 926,033 cost to completionrep-2-r12: 1,089,432 cost to completionrep-2-r12: 1,089,432 cost to completionrep-2-r13: 794,216 cost to completionrep-2-r16: 681,209 cost to completioncoarse-to-fine, full-1 survey build: 926,685 cost to completionCoarse-to-fine TDD: median 926,033Most informative next testMost informative next test: median 1,068,727rep-2-r05: 878,599 cost to completionrep-2-r09: 927,123 cost to completionrep-2-r15: 1,068,727 cost to completionrep-2-r15: 1,068,727 cost to completionrep-2-r18: 1,109,946 cost to completionrep-2-r19: 1,175,603 cost to completionmutation-guided, full-1 survey build: 708,760 cost to completionMost informative next test: median 1,068,727
Held-out TDDClassic greedy TDDCoarse-to-fine TDDMost informative next testSurvey build, earlier batch
Data
Table view, 24 rows and 4 columns
MethodBuildSourceCost to completion
Held-out TDDrep-2-r04rep-2 replicate, representative1,021,062
Held-out TDDrep-2-r08rep-2 replicate904,350
Held-out TDDrep-2-r14rep-2 replicate973,901
Held-out TDDrep-2-r17rep-2 replicate926,018
Held-out TDDrep-2-r20rep-2 replicate1,088,356
Classic greedy TDDrep-2-r01rep-2 replicate1,011,641
Classic greedy TDDrep-2-r02rep-2 replicate946,961
Classic greedy TDDrep-2-r07rep-2 replicate1,256,020
Classic greedy TDDrep-2-r10rep-2 replicate, representative1,079,012
Classic greedy TDDrep-2-r11rep-2 replicate908,035
Coarse-to-fine TDDrep-2-r03rep-2 replicate967,802
Coarse-to-fine TDDrep-2-r06rep-2 replicate926,033
Coarse-to-fine TDDrep-2-r12rep-2 replicate, representative1,089,432
Coarse-to-fine TDDrep-2-r13rep-2 replicate794,216
Coarse-to-fine TDDrep-2-r16rep-2 replicate681,209
Most informative next testrep-2-r05rep-2 replicate878,599
Most informative next testrep-2-r09rep-2 replicate927,123
Most informative next testrep-2-r15rep-2 replicate, representative1,068,727
Most informative next testrep-2-r18rep-2 replicate1,109,946
Most informative next testrep-2-r19rep-2 replicate1,175,603
Held-out TDDheld-outfull-1 survey build801,898
Classic greedy TDDclassicfull-1 survey build804,727
Coarse-to-fine TDDcoarse-to-finefull-1 survey build926,685
Most informative next testmutation-guidedfull-1 survey build708,760

Figure 7.8

CQP per replicate, rep-2

Informs Q12.

Medians over each method's replicates: Held-out TDD 17, Classic greedy TDD 13, Coarse-to-fine TDD 16, Most informative next test 16.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Tag: recomputed.

05101520CQP, 4 to 20Held-out TDDHeld-out TDD: median 17rep-2-r04: 17 CQPrep-2-r04: 17 CQPrep-2-r08: 16 CQPrep-2-r14: 16 CQPrep-2-r17: 17 CQPrep-2-r20: 17 CQPHeld-out TDD: median 17Classic greedy TDDClassic greedy TDD: median 13rep-2-r01: 13 CQPrep-2-r02: 14 CQPrep-2-r07: 13 CQPrep-2-r10: 16 CQPrep-2-r10: 16 CQPrep-2-r11: 13 CQPClassic greedy TDD: median 13Coarse-to-fine TDDCoarse-to-fine TDD: median 16rep-2-r03: 15 CQPrep-2-r06: 16 CQPrep-2-r12: 15 CQPrep-2-r12: 15 CQPrep-2-r13: 16 CQPrep-2-r16: 17 CQPCoarse-to-fine TDD: median 16Most informative next testMost informative next test: median 16rep-2-r05: 15 CQPrep-2-r09: 16 CQPrep-2-r15: 16 CQPrep-2-r15: 16 CQPrep-2-r18: 16 CQPrep-2-r19: 16 CQPMost informative next test: median 16
Held-out TDDClassic greedy TDDCoarse-to-fine TDDMost informative next test
Data
Table view, 20 rows and 4 columns
MethodBuildSourceCQP
Held-out TDDrep-2-r04rep-2 replicate, representative17
Held-out TDDrep-2-r08rep-2 replicate16
Held-out TDDrep-2-r14rep-2 replicate16
Held-out TDDrep-2-r17rep-2 replicate17
Held-out TDDrep-2-r20rep-2 replicate17
Classic greedy TDDrep-2-r01rep-2 replicate13
Classic greedy TDDrep-2-r02rep-2 replicate14
Classic greedy TDDrep-2-r07rep-2 replicate13
Classic greedy TDDrep-2-r10rep-2 replicate, representative16
Classic greedy TDDrep-2-r11rep-2 replicate13
Coarse-to-fine TDDrep-2-r03rep-2 replicate15
Coarse-to-fine TDDrep-2-r06rep-2 replicate16
Coarse-to-fine TDDrep-2-r12rep-2 replicate, representative15
Coarse-to-fine TDDrep-2-r13rep-2 replicate16
Coarse-to-fine TDDrep-2-r16rep-2 replicate17
Most informative next testrep-2-r05rep-2 replicate15
Most informative next testrep-2-r09rep-2 replicate16
Most informative next testrep-2-r15rep-2 replicate, representative16
Most informative next testrep-2-r18rep-2 replicate16
Most informative next testrep-2-r19rep-2 replicate16

Figure 7.9

M per replicate, rep-2

Informs Q1 and Q8.

Medians over each method's replicates: Held-out TDD 607308, Classic greedy TDD 601555, Coarse-to-fine TDD 559186, Most informative next test 735111.

One row per method in planned pair order. Each dot is one replicate build, the tick is the median and the whisker spans the lowest to the highest. The ringed dot is the representative build, the middle of five by adjusted mutation, then cost. Hollow diamonds are the single survey build of each method from the earlier full-1 run, a different batch. They stay out of every median, span and test. Tag: recomputed.

00.20.40.60.81M, weighted tokens per hidden test passed, millionsHeld-out TDDHeld-out TDD: median 607308rep-2-r04: 623923 Mrep-2-r04: 623923 Mrep-2-r08: 469385 Mrep-2-r14: 607308 Mrep-2-r17: 589595 Mrep-2-r20: 674611 Mheld-out, full-1 survey build: 519499 MHeld-out TDD: median 607308Classic greedy TDDClassic greedy TDD: median 601555rep-2-r01: 601555 Mrep-2-r02: 598650 Mrep-2-r07: 950708 Mrep-2-r10: 636452 Mrep-2-r10: 636452 Mrep-2-r11: 539897 Mclassic, full-1 survey build: 542369 MClassic greedy TDD: median 601555Coarse-to-fine TDDCoarse-to-fine TDD: median 559186rep-2-r03: 700373 Mrep-2-r06: 559186 Mrep-2-r12: 857141 Mrep-2-r12: 857141 Mrep-2-r13: 534710 Mrep-2-r16: 446307 Mcoarse-to-fine, full-1 survey build: 537832 MCoarse-to-fine TDD: median 559186Most informative next testMost informative next test: median 735111rep-2-r05: 552939 Mrep-2-r09: 735111 Mrep-2-r15: 859769 Mrep-2-r15: 859769 Mrep-2-r18: 711019 Mrep-2-r19: 945338 Mmutation-guided, full-1 survey build: 477347 MMost informative next test: median 735111
Held-out TDDClassic greedy TDDCoarse-to-fine TDDMost informative next testSurvey build, earlier batch
Data
Table view, 24 rows and 4 columns
MethodBuildSourceM
Held-out TDDrep-2-r04rep-2 replicate, representative623923
Held-out TDDrep-2-r08rep-2 replicate469385
Held-out TDDrep-2-r14rep-2 replicate607308
Held-out TDDrep-2-r17rep-2 replicate589595
Held-out TDDrep-2-r20rep-2 replicate674611
Classic greedy TDDrep-2-r01rep-2 replicate601555
Classic greedy TDDrep-2-r02rep-2 replicate598650
Classic greedy TDDrep-2-r07rep-2 replicate950708
Classic greedy TDDrep-2-r10rep-2 replicate, representative636452
Classic greedy TDDrep-2-r11rep-2 replicate539897
Coarse-to-fine TDDrep-2-r03rep-2 replicate700373
Coarse-to-fine TDDrep-2-r06rep-2 replicate559186
Coarse-to-fine TDDrep-2-r12rep-2 replicate, representative857141
Coarse-to-fine TDDrep-2-r13rep-2 replicate534710
Coarse-to-fine TDDrep-2-r16rep-2 replicate446307
Most informative next testrep-2-r05rep-2 replicate552939
Most informative next testrep-2-r09rep-2 replicate735111
Most informative next testrep-2-r15rep-2 replicate, representative859769
Most informative next testrep-2-r18rep-2 replicate711019
Most informative next testrep-2-r19rep-2 replicate945338
Held-out TDDheld-outfull-1 survey build519499
Classic greedy TDDclassicfull-1 survey build542369
Coarse-to-fine TDDcoarse-to-finefull-1 survey build537832
Most informative next testmutation-guidedfull-1 survey build477347