Test order
Run tcp-1 built one list of 12 candidate tests in 5 orders, once each. A second blind lister in run tcp-1-list2 agreed with the first on usage with rho 0.96 and on risk with weighted kappa 0.53. Tag: recomputed.
Figure 9.1 plots the APFD-I and APFDc-H values that Q20 reports for each order.
Figure 9.1
APFD-I and APFDc-H by test-list order
Each order ran once, so the values are descriptive. Hollow marks are at or below random. Tag: recomputed, from Q20.
Text version: APFD-I and APFDc-H by test-list order
- coverage: APFD-I 0.5515
- coverage: APFDc-H 0.4715
- cost: APFD-I 0.6634
- cost: APFDc-H 0.5716
- risk: APFD-I 0.5274
- risk: APFDc-H 0.4166
- usage: APFD-I 0.6129
- usage: APFDc-H 0.5309
APFD-I and APFDc-H per order
Tag: recomputed, from Q20.
| Order | APFD-I | APFDc-H |
|---|---|---|
| random | 0.6316 | 0.5111 |
| coverage | 0.5515 | 0.4715 |
| cost | 0.6634 | 0.5716 |
| risk | 0.5274 | 0.4166 |
| usage | 0.6129 | 0.5309 |
Design of run tcp-1
The settings, list and orders of run tcp-1 as its run facts give them. Tag: verified.
Settings
| Setting | Value |
|---|---|
| Seed | 20261020 |
| Lister | tcp-1-list-lister |
| List run | tcp-1-list |
| List sha | 7788e4fd |
| Methods doc, tcp-1 | 1f11dd37 |
| Methods doc, tcp-1-list | 1f11dd37 |
| Methods doc, tcp-1-list2 | 12f026f6 |
List phase steps
The lister of run tcp-1-list followed these steps.
- Number each functional rule in README.md and docs/SPEC.md as a clause R1 to Rn in CLAUSES.md. Leave out code quality and PMD rules.
- Rate each clause 1 to 3 for the chance a first implementation gets it wrong. Rate its usage 1 to 100 as the percent of moves in a typical game that exercise it. Rate a clause of one variant as the percent of games that use the variant times the share of their moves that exercise it.
- List at most 12 candidate tests in TESTLIST.md. Each names one behaviour.
- Give each candidate two fields. checks lists every clause its assertions observe, variants included. relies lists clauses it needs and does not assert.
- Estimate the production lines each candidate needs on the starter alone.
- Commit both files and return them as JSON.
Clauses
| Clause | Risk | Usage |
|---|---|---|
| R1 | 1 | 50 |
| R2 | 2 | 50 |
| R3 | 1 | 3 |
| R4 | 2 | 95 |
| R5 | 3 | 48 |
| R6 | 3 | 48 |
| R7 | 3 | 50 |
| R8 | 1 | 65 |
| R9 | 1 | 20 |
| R10 | 2 | 10 |
| R11 | 2 | 3 |
| R12 | 2 | 3 |
| R13 | 3 | 3 |
| R14 | 2 | 3 |
| R15 | 1 | 3 |
| R16 | 2 | 50 |
| R17 | 2 | 50 |
| R18 | 1 | 100 |
| R19 | 1 | 3 |
Candidates
| Candidate | Checks | Relies | Estimate |
|---|---|---|---|
| T1 | R1, R3, R15, R17, R18 | none | 25 |
| T2 | R2, R3, R15, R16, R18 | none | 25 |
| T3 | R1, R4, R5, R8, R16, R18 | R3, R15 | 40 |
| T4 | R1, R4, R6, R8, R17, R18 | R3, R15 | 40 |
| T5 | R7, R9, R18 | R1, R4, R5, R16, R15 | 45 |
| T6 | R7, R8 | R1, R4, R6, R17, R18, R15 | 50 |
| T7 | R10, R8 | R1, R4, R5, R16, R18, R15 | 55 |
| T8 | R11, R8 | R1, R4, R5, R10, R16, R18, R15 | 55 |
| T9 | R19, R18 | R1, R4, R5, R16, R15 | 45 |
| T10 | R4, R7, R10 | R1, R6, R17, R18, R15 | 55 |
| T11 | R12, R13, R14 | R1, R4, R6, R8, R9, R10, R17, R18 | 70 |
| T12 | R12, R13, R14 | R1, R4, R5, R8, R9, R10, R16, R18 | 70 |
Orders
| Order | Arm | Candidates in order |
|---|---|---|
| untreated | not run | T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12 |
| random | tcp-1-r01 | T7, T10, T6, T1, T2, T11, T8, T12, T5, T9, T3, T4 |
| coverage | tcp-1-r02 | T3, T1, T11, T10, T2, T8, T5, T9, T4, T12, T7, T6 |
| cost | tcp-1-r03 | T1, T3, T5, T11, T2, T4, T9, T7, T8, T10, T12, T6 |
| risk | tcp-1-r04 | T10, T11, T12, T6, T5, T3, T4, T7, T8, T2, T1, T9 |
| usage | tcp-1-r05 | T3, T4, T5, T2, T7, T11, T8, T9, T1, T10, T6, T12 |
| costOnly | not run | T1, T2, T3, T4, T5, T9, T6, T7, T10, T8, T11, T12 |
Arm steps
Every arm followed these steps.
- Read CLAUSES.md and TESTLIST.md. Change neither.
- Build candidates in the order the harness gives. Log a new candidate in ADDED.md with its checks and relies.
- Write one failing test for the first open candidate and watch it fail.
- Pass it with the smallest change. Refactor on green. Commit each green.
Figure 9.2
Stage flow of a tcp-1 arm
- Steps 1 and 2. Lane: Throughout.
- Steps 3 and 4. Returns candidate. Lane: Agent.
Loop: repeat per candidate, back to Steps 3 and 4.
Every arm and tcp design lists these stages. Details are in the stage table. Tag: verified.
Text version: Stage flow of a tcp-1 arm
- Steps 1 and 2. Lane: Throughout.
- Steps 3 and 4. Returns candidate. Lane: Agent.
Loop: repeat per candidate, back to Steps 3 and 4.
Stages
| Stage | Runs as | Steps | Details | Note |
|---|---|---|---|---|
| F1-2 | throughout: | 1 and 2 | none | none |
| L3-4 | repeat per candidate until none is open: | 3 and 4 | unit candidate; returns candidate | UNITS.md holds one line per TESTLIST.md candidate id plus one per added candidate. Reorder the open lines to the order in the facts from the harness, then work the first. Return its id in candidate and any new candidate in added. |