Kalah TDD study · Part 6 of 29 · Results

Results: survey (full-1)

One build of each design in the survey run full-1.

Source runs full-1 and rep-2 at tdd-experiment c3dd988

Contents

Arguments

Each figure backs one argument. Colour and shape mark the method family of the Summary table in runs/full-1/tdd-methodologies.md, the A02 reference is drawn in ink, and each figure has its table view.

Survey (full-1)

Each survey method ran once, so its figures describe one build. On the share of hidden tests passed at 500000 weighted tokens Held-out TDD led at 0.42, then Coarse-to-fine TDD and Most informative next test at 0.39, against 0.29 for Classic greedy TDD. Held-out TDD also had the lowest adjusted mutation score of the survey at 0.794. In rep-2, a different batch from the survey, the median shares at 500000 weighted tokens were 0.00 for Held-out TDD, 0.34 for Coarse-to-fine TDD, 0.09 for Most informative next test and 0.29 for Classic greedy TDD. Tag: recomputed.

Figure 6.1

PMD violations against classes

Informs Q6, Q11 and Q13.

The 3 methods that pass strict PMD compliance have 6, 14 and 16 classes, and the other 8 have at most 5.

One mark per method's final clone. Labelled marks pass strict PMD compliance. Hollow marks are start clones, which the permutation test leaves out. An exact one-sided permutation test over all 165 ways to pick 3 of the 11 methods gives p = 0.006 for clean methods having this many classes or more. Spearman rho = -0.73 between classes and violations. Both are post hoc and treat each method as one observation, since each ran once. The hollow start clone with the most classes is multi-start-1 with 7 classes and 3 PMD violations. The A02 reference has 10 classes and 0 PMD violations. Tag: recomputed.

024681005101520Classes in src/PMD violationsmulti-start-1, start clone: 7 classes, 3 PMD violationsmulti-start-2, start clone: 1 classes, 5 PMD violationsmulti-start-3, start clone: 3 classes, 5 PMD violationsclassic: 1 classes, 10 PMD violationsnoun-verb: 14 classes, 0 PMD violationsmapped-path: 16 classes, 0 PMD violationsfunctions-first: 6 classes, 0 PMD violationstpp: 3 classes, 7 PMD violationscoarse-to-fine: 3 classes, 6 PMD violationsannealed: 1 classes, 2 PMD violationsmulti-start: 3 classes, 5 PMD violationsmutation-guided: 3 classes, 9 PMD violationsproperty-based: 4 classes, 5 PMD violationsheld-out: 5 classes, 4 PMD violationsA02 reference: 10 classes, 0 PMD violationsnoun-verbmapped-pathfunctions-firstA02 reference
BaselineStructure firstStep sizeSearch and validationA02 referenceStart clone
Data
Table view, 15 rows and 5 columns
MethodSlugFamilyClassesPMD violations
Classic greedy TDDclassicBaseline110
Noun-verb mapnoun-verbStructure first140
Mapped-path TDDmapped-pathStructure first160
Functions first, classes laterfunctions-firstStructure first60
Transformation Priority Premise TDDtppStep size37
Coarse-to-fine TDDcoarse-to-fineStep size36
Annealed refactoringannealedStep size12
Multi-start TDDmulti-startSearch and validation35
Most informative next testmutation-guidedSearch and validation39
Property-based TDDproperty-basedSearch and validation45
Held-out TDDheld-outSearch and validation54
Multi-start TDD, start clonemulti-start-1Search and validation73
Multi-start TDD, start clonemulti-start-2Search and validation15
Multi-start TDD, start clonemulti-start-3Search and validation35
A02 referencereferenceReference100

Figure 6.2

Mutation score per method

Informs Q12.

Mutation scores run from 75% (held-out) to 95% (mapped-path).

Each dot is the share of a method's N compiled mutants its tests killed or timed out, labelled killed/N, and its whisker is the Wilson 95% interval, descriptive only since one suite judges all of a method's mutants. Only each method's own tests ran, without the starter's two Example tests and their ExampleSpec helper. Over the scores a seeded permutation chi-square gives 63.0 with p below 0.001, and Fisher exact tests with Holm correction separate 11 of the 55 pairs at 0.05. Over the adjusted scores a seeded permutation chi-square gives 120.5 with p below 0.001, and Fisher exact tests with Holm correction separate 22 of the 55 pairs at 0.05. Both treat each mutant as independent within one run per method. Surviving mutants missed by the hidden suite are excluded. They may be equivalent or expose gaps in that suite. The A02 reference ran the reference's own TestKalah tests from the hidden jar and the dashed line marks its score. Tag: recomputed.

60708090100Mutants killed, percent of those that compiledmapped-path, 107/113mapped-path: 107 of 113 compiled mutants killed, 95%, 95% interval 88.9 to 97.5mapped-path: 107 of 113 compiled mutants killed, 95%, 95% interval 88.9 to 97.5property-based, 116/123property-based: 116 of 123 compiled mutants killed, 94%, 95% interval 88.7 to 97.2property-based: 116 of 123 compiled mutants killed, 94%, 95% interval 88.7 to 97.2mutation-guided, 112/119mutation-guided: 112 of 119 compiled mutants killed, 94%, 95% interval 88.4 to 97.1mutation-guided: 112 of 119 compiled mutants killed, 94%, 95% interval 88.4 to 97.1annealed, 115/123annealed: 115 of 123 compiled mutants killed, 93%, 95% interval 87.7 to 96.7annealed: 115 of 123 compiled mutants killed, 93%, 95% interval 87.7 to 96.7classic, 105/113classic: 105 of 113 compiled mutants killed, 93%, 95% interval 86.6 to 96.4classic: 105 of 113 compiled mutants killed, 93%, 95% interval 86.6 to 96.4coarse-to-fine, 102/111coarse-to-fine: 102 of 111 compiled mutants killed, 92%, 95% interval 85.3 to 95.7coarse-to-fine: 102 of 111 compiled mutants killed, 92%, 95% interval 85.3 to 95.7functions-first, 117/132functions-first: 117 of 132 compiled mutants killed, 89%, 95% interval 82.1 to 93.0functions-first: 117 of 132 compiled mutants killed, 89%, 95% interval 82.1 to 93.0multi-start, 103/123multi-start: 103 of 123 compiled mutants killed, 84%, 95% interval 76.2 to 89.2multi-start: 103 of 123 compiled mutants killed, 84%, 95% interval 76.2 to 89.2tpp, 94/114tpp: 94 of 114 compiled mutants killed, 82%, 95% interval 74.4 to 88.3tpp: 94 of 114 compiled mutants killed, 82%, 95% interval 74.4 to 88.3noun-verb, 88/115noun-verb: 88 of 115 compiled mutants killed, 77%, 95% interval 68.0 to 83.3noun-verb: 88 of 115 compiled mutants killed, 77%, 95% interval 68.0 to 83.3held-out, 112/149held-out: 112 of 149 compiled mutants killed, 75%, 95% interval 67.7 to 81.4held-out: 112 of 149 compiled mutants killed, 75%, 95% interval 67.7 to 81.4A02 reference, 154/165A02 reference: 154 of 165 compiled mutants killed, 93%, 95% interval 88.5 to 96.2A02 reference: 154 of 165 compiled mutants killed, 93%, 95% interval 88.5 to 96.2A02 reference: 93%
BaselineStructure firstStep sizeSearch and validationA02 reference
Data
Table view, 12 rows and 8 columns
MethodSlugFamilyAdjusted %Killed
Mapped-path TDDmapped-pathStructure first98107
Property-based TDDproperty-basedSearch and validation100116
Most informative next testmutation-guidedSearch and validation100112
Annealed refactoringannealedStep size98115
Classic greedy TDDclassicBaseline100105
Coarse-to-fine TDDcoarse-to-fineStep size100102
Functions first, classes laterfunctions-firstStructure first96117
Multi-start TDDmulti-startSearch and validation87103
Transformation Priority Premise TDDtppStep size8894
Noun-verb mapnoun-verbStructure first8188
Held-out TDDheld-outSearch and validation79112
A02 referencereferenceReferencenot graded154
MethodCompiled mutantsScore %95% interval %
Mapped-path TDD1139588.9 to 97.5
Property-based TDD1239488.7 to 97.2
Most informative next test1199488.4 to 97.1
Annealed refactoring1239387.7 to 96.7
Classic greedy TDD1139386.6 to 96.4
Coarse-to-fine TDD1119285.3 to 95.7
Functions first, classes later1328982.1 to 93.0
Multi-start TDD1238476.2 to 89.2
Transformation Priority Premise TDD1148274.4 to 88.3
Noun-verb map1157768.0 to 83.3
Held-out TDD1497567.7 to 81.4
A02 reference1659388.5 to 96.2

Figure 6.3

Bench run times per method

Informs Q7.

Bench medians run from 205 ms (tpp) to 606 ms (held-out).

Each dot is one bench run, the first hollow, and the tick marks the median. Each run plays 1000 random games from seed 701. The first run is the slowest in 11 of 11 methods, 68 to 468 ms over the median. The dashed line marks the A02 reference median of 500 ms. Tag: recomputed.

025050075010001250Bench run time, msclassicclassic run 1: 1020 msclassic run 2: 610 msclassic run 3: 605 msclassic run 4: 600 msclassic run 5: 599 msclassic: median 605 msnoun-verbnoun-verb run 1: 896 msnoun-verb run 2: 562 msnoun-verb run 3: 606 msnoun-verb run 4: 529 msnoun-verb run 5: 536 msnoun-verb: median 562 msmapped-pathmapped-path run 1: 858 msmapped-path run 2: 518 msmapped-path run 3: 520 msmapped-path run 4: 520 msmapped-path run 5: 517 msmapped-path: median 520 msfunctions-firstfunctions-first run 1: 838 msfunctions-first run 2: 554 msfunctions-first run 3: 521 msfunctions-first run 4: 518 msfunctions-first run 5: 515 msfunctions-first: median 521 mstpptpp run 1: 273 mstpp run 2: 225 mstpp run 3: 205 mstpp run 4: 194 mstpp run 5: 194 mstpp: median 205 mscoarse-to-finecoarse-to-fine run 1: 903 mscoarse-to-fine run 2: 614 mscoarse-to-fine run 3: 605 mscoarse-to-fine run 4: 602 mscoarse-to-fine run 5: 583 mscoarse-to-fine: median 605 msannealedannealed run 1: 938 msannealed run 2: 526 msannealed run 3: 544 msannealed run 4: 520 msannealed run 5: 512 msannealed: median 526 msmulti-startmulti-start run 1: 916 msmulti-start run 2: 548 msmulti-start run 3: 553 msmulti-start run 4: 523 msmulti-start run 5: 627 msmulti-start: median 553 msmutation-guidedmutation-guided run 1: 867 msmutation-guided run 2: 595 msmutation-guided run 3: 576 msmutation-guided run 4: 576 msmutation-guided run 5: 568 msmutation-guided: median 576 msproperty-basedproperty-based run 1: 1021 msproperty-based run 2: 591 msproperty-based run 3: 540 msproperty-based run 4: 553 msproperty-based run 5: 547 msproperty-based: median 553 msheld-outheld-out run 1: 985 msheld-out run 2: 641 msheld-out run 3: 606 msheld-out run 4: 600 msheld-out run 5: 593 msheld-out: median 606 msA02 referenceA02 reference run 1: 831 msA02 reference run 2: 500 msA02 reference run 3: 502 msA02 reference run 4: 495 msA02 reference run 5: 495 msA02 reference: median 500 msA02 reference median: 500 ms
BaselineStructure firstStep sizeSearch and validationA02 referenceFirst run
Data
Table view, 12 rows and 5 columns
MethodSlugFamilyMedian msRun ms
Classic greedy TDDclassicBaseline6051020, 610, 605, 600, 599
Noun-verb mapnoun-verbStructure first562896, 562, 606, 529, 536
Mapped-path TDDmapped-pathStructure first520858, 518, 520, 520, 517
Functions first, classes laterfunctions-firstStructure first521838, 554, 521, 518, 515
Transformation Priority Premise TDDtppStep size205273, 225, 205, 194, 194
Coarse-to-fine TDDcoarse-to-fineStep size605903, 614, 605, 602, 583
Annealed refactoringannealedStep size526938, 526, 544, 520, 512
Multi-start TDDmulti-startSearch and validation553916, 548, 553, 523, 627
Most informative next testmutation-guidedSearch and validation576867, 595, 576, 576, 568
Property-based TDDproperty-basedSearch and validation5531021, 591, 540, 553, 547
Held-out TDDheld-outSearch and validation606985, 641, 606, 600, 593
A02 referencereferenceReference500831, 500, 502, 495, 495