Methodology: harness and protocol

How a design runs

Each design runs as a list of stages in harness/workflow/stages.json. A stage takes numbered steps of its method from tdd-methodologies.md, quoted through args.json. 18 stage sets hold 58 stages. recomputed

S: once

Once means one agent runs the steps. 32 stages.

L: repeat per unit

Repeat per unit means one agent per unit until no unit is open. 15 stages.

F: throughout

Throughout means the rule joins every prompt. 9 stages.

starts: for up to n starts

For up to n starts means separate attempts with the best kept. 1 stage.

H: the harness does

The harness does means the harness runs the step without an agent. 1 stage.

If names the event that starts a stage, and with adds a rule borrowed from another method. Steps are quoted from args.json with their numbers. verified

Agent roles

Examples are agent labels in the full-1 report and its method pages. Lister labels join each list-only run's lister field to :list. recomputed

Where each role works paraphrases the stage notes, the review schema and the harness sources batch.js and rate.js. verified

Transfer agents run fixed git commands. Ranking the starts is the method's own stage s6. verified

Stages

Each table lists one family's stage sets as stages.json holds them now. Rules name a commit cap, the event that starts a stage and the steps borrowed from another method. The stage notes follow each table. verified

Baseline

Classic greedy TDD L2-7

An impasse is a bar that stays red for 3 red-green cycles. Log each impasse in the LOG.md actions.

Structure first

Noun-verb map s6

One prompt in the step is one commit here.

Noun-verb map s7

One prompt in the step is one commit here.

Functions first, classes later s3

Test-drive one verb per commit.

Step size

Transformation Priority Premise TDD s1

Martin's list from highest to lowest rank is {}->nil, constant, constant+, scalar, statements, if, array, container, recursion, while, function and variable->assignment. Write it with one-line definitions to TPP.md and commit it. Later stages read TPP.md as the prompt list.

Transformation Priority Premise TDD L2-7

Read TPP.md first. An impasse is a bar that stays red for 3 red-green cycles. Log each impasse in the LOG.md actions.

Coarse-to-fine TDD L1,3-8

10 minutes red is 2 red-green cycles. An attempt stays uncommitted until green, and a shrink reverts it with git restore.

Annealed refactoring F2-3

The timeboxes are red-green cycles: hot 9, warm 3 and cold 1.

Annealed refactoring L1,4-8

If TESTS.md is missing, first write the original test list to it with one line per planned test and commit it. Step 8 counts thirds of that list. Energy comes from ./run.sh lint and ./run.sh deps. A move is tried uncommitted in place of a throwaway branch and reverted with git restore.

Search and validation

Multi-start TDD s1

Lock the acceptance tests with a JUnit test that compares their SHA-256 with recorded values, so ./run.sh test runs the check.

Multi-start TDD s2

Write the score rule to SCORE.md and commit it. The harness gives each attempt two halves of up to the cycle limit each, with the drop rule between them.

Multi-start TDD s3

Return one line per start strategy in starts. Create no branch or clone, because each start gets its own clone.

Multi-start TDD s4-5

Report passed and total from ./run.sh test on the locked acceptance tests.

Multi-start TDD s6

Rank the finished attempts by the score rule in SCORE.md, using the attempt results among the facts below. Pick the winner only among the finished starts. A partial mutation score covers only the mutants its run reached, so when any finished attempt's mutation score is partial, mutation score ranks none of them and the next criterion in SCORE.md decides. When no finished attempt is green, rank them by tests passed, as step 6 says. Write ATTEMPTS.md with each attempt's results, its score and the drop decisions, commit it and return the winning start number.

Multi-start TDD s7

The winner is at the local branch winner. First run git restore --source=winner --staged --worktree -- src test and commit, then archive the others in ATTEMPTS.md.

Most informative next test L1-8

Mutation testing is ./run.sh mutate. It mutates all of src/, newest file first. It runs the method's own tests without the starter's Example tests.

Property-based TDD L1-7

No property library is on the classpath. Write generators over java.util.Random with a logged seed.

Held-out TDD s2

You are the separate agent. Return blocked if {cases} holds anything. Write at least 3 black-box cases per requirement as {cases}/<requirement>/<variant>/caseNN.txt, where <requirement> names a noun and <variant> is a variant folder name from examples/EXAMPLES.md, in the spec format of examples/specs. Keep generators and other scratch files in {scratch}, never in the clone, which the method agents read. Log counts only in LOG.md and commit LOG.md.

Held-out TDD s3

Run {split} {cases} {visible}, which keeps two thirds visible, and commit visible/. Log only the line it prints.

Held-out TDD L6-8

Drive with ./run.sh specs visible and ./run.sh test. Once the unit is green run ./run.sh hidden once and append a GAP.md row: time, label, requirement, visible pass/total, hidden pass/total and gap. Write a fresh visible test for each requirement ticked in UNITS.md that fails hidden. Write each fresh visible test under test/, never under visible/, which the harness re-splits. A spec file there goes in a variant folder, so ./run.sh specs test plays it. The harness does the rest of step 8 after this loop and replaces the hidden cases through separate agents, so leave the hidden set alone. Return in failing only the requirements ticked in UNITS.md that fail hidden, because the others are not built yet.

Held-out TDD s8-writer

You are the separate agent. Add fresh cases to {cases} for each requirement the facts below list as failing hidden, in the layout of {cases}. Keep generators and other scratch files in {scratch}, never in the clone, which the method agents read. Log counts only in LOG.md and commit LOG.md.

Held-out TDD s8-split

Run {split} {cases} {visible} again and commit visible/. Log the line it prints and the count of earlier cases it moved into visible/. Then run ./run.sh specs visible and ./run.sh hidden once and append a closing GAP.md row.

Test-list order

tcp-untreated L3-4

UNITS.md holds one line per TESTLIST.md candidate id plus one per added candidate. Reorder the open lines to the order in the facts from the harness, then work the first. Return its id in candidate and any new candidate in added.

tcp-random L3-4

UNITS.md holds one line per TESTLIST.md candidate id plus one per added candidate. Reorder the open lines to the order in the facts from the harness, then work the first. Return its id in candidate and any new candidate in added.

tcp-coverage L3-4

UNITS.md holds one line per TESTLIST.md candidate id plus one per added candidate. Reorder the open lines to the order in the facts from the harness, then work the first. Return its id in candidate and any new candidate in added.

tcp-cost L3-4

UNITS.md holds one line per TESTLIST.md candidate id plus one per added candidate. Reorder the open lines to the order in the facts from the harness, then work the first. Return its id in candidate and any new candidate in added.

tcp-risk L3-4

UNITS.md holds one line per TESTLIST.md candidate id plus one per added candidate. Reorder the open lines to the order in the facts from the harness, then work the first. Return its id in candidate and any new candidate in added.

tcp-usage L3-4

UNITS.md holds one line per TESTLIST.md candidate id plus one per added candidate. Reorder the open lines to the order in the facts from the harness, then work the first. Return its id in candidate and any new candidate in added.

Stages as run

Each run records the stages it ran. 16 builds recorded a stage field unlike stages.json. As run, with holds the quoted step texts where stages.json holds step numbers. recomputed

Prompts and the shared test list

The test-list order arms share one prompt and one test list. The methods doc states how the list is written. verified

One lister agent works once per run in its own clone and writes the list every arm shares.

1. Number each functional rule in README.md and docs/SPEC.md as a clause R1 to Rn in CLAUSES.md. Leave out code quality and PMD rules.
2. Rate each clause 1 to 3 for the chance a first implementation gets it wrong. Rate its usage 1 to 100 as the percent of moves in a typical game that exercise it. Rate a clause of one variant as the percent of games that use the variant times the share of their moves that exercise it.
3. List at most 12 candidate tests in TESTLIST.md. Each names one behaviour.
4. Give each candidate two fields. checks lists every clause its assertions observe, variants included. relies lists clauses it needs and does not assert.
5. Estimate the production lines each candidate needs on the starter alone.
6. Commit both files and return them as JSON.

The harness checks the JSON before any arm starts. Ids are unique, checks and relies name real clauses, there are 1 to 12 candidates, risk is 1 to 3, usage is 1 to 100 and each estimate is a whole number. A failed check stops the family. The harness divides each usage by the sum over all clauses, so the usage shares sum to 1.

tcp-1-list ran the list phase in 6 steps.

1. Number each functional rule in README.md and docs/SPEC.md as a clause R1 to Rn in CLAUSES.md. Leave out code quality and PMD rules.
2. Rate each clause 1 to 3 for the chance a first implementation gets it wrong. Rate its usage 1 to 100 as the percent of moves in a typical game that exercise it. Rate a clause of one variant as the percent of games that use the variant times the share of their moves that exercise it.
3. List at most 12 candidate tests in TESTLIST.md. Each names one behaviour.
4. Give each candidate two fields. checks lists every clause its assertions observe, variants included. relies lists clauses it needs and does not assert.
5. Estimate the production lines each candidate needs on the starter alone.
6. Commit both files and return them as JSON.

tcp-1-list test list fields.

lister

tcp-1-list-lister

seed

20261020

tcp-1-list2 ran the list phase in 5 steps.

1. CLAUSES.md and TESTLIST.md already hold every clause and candidate with blank Risk, Usage, Checks, Relies and Estimate cells. Fill only those cells by the steps below. Keep every other cell and row as written and add no clause or candidate.
2. Rate each clause 1 to 3 for the chance a first implementation gets it wrong. Rate its usage 1 to 100 as the percent of moves in a typical game that exercise it. Rate a clause of one variant as the percent of games that use the variant times the share of their moves that exercise it.
3. Give each candidate two fields. checks lists every clause its assertions observe, variants included. relies lists clauses it needs and does not assert.
4. Estimate the production lines each candidate needs on the starter alone.
5. Commit both files and return them as JSON.

tcp-1-list2 test list fields.

lister

tcp-1-list2-lister

seed

20261020

rates

tcp-1-list

tcp-1 test list fields.

list_tag

tcp-1-list

lister

tcp-1-list-lister

seed

20261020

sha

7788e4fd8dbb9ce5b077c3478208d9fed4f6327b

tcp-1 built 5 arms on the list of tcp-1-list: 19 clauses and 12 candidates. recomputed

Runs and harness versions

67 build entries in 9 run records include 5 rebuild entries and exclude start clones. Doc version is the short sha256 of the methods doc a run was built under. Redacted keys name run record fields the snapshot leaves out. recomputed

2 runs record the harness they ran. rep-2/rebuild rebuilt 5 rep-2 replicates on a changed harness and records the sha256 of the change. recomputed

The held-out split

Held-out TDD as the methods doc states it. Its design page covers its builds. verified

Visible tests drive the red-green loop. Hidden tests that the implementer never sees score milestones.

Mirrors: a train and validation split. The gap between visible and hidden pass rates measures overfitting.

1. Pin the public interface first such as the CLI or the noun-verb verbs.
2. A human or separate agent writes several black-box cases per requirement.
3. Move about a third of each requirement's cases into a hidden set.
4. Keep hidden tests in a private repo that only CI checks out. Keep its CI token out of the agent environment.
5. Clone it outside the agent workspace. Add a permissions deny entry for that path in the assistant settings.
6. Drive TDD and AI loops with the visible set alone.
7. On each merge CI prints hidden pass counts per requirement. Log the gap.
8. On a hidden failure write a fresh visible test. Replace that hidden case.

Pairs with noun-verb: pin the map's public verbs as the interface both sets test. The gap per noun shows where the code overfits.

Measure: visible and hidden pass rate per merge. Gap per noun. Hidden cases replaced after disclosure.

Fits when: an AI loop iterates hard against tests and a human or separate agent owns the hidden set.

Risk: each look at the hidden score leaks a little so the set ages with reuse. A solo author has seen the hidden tests so only the AI side is held out. CI logs that print assertions leak the set.

The separate agent writes the cases in stage s2 and stage s3 splits them. Their notes follow. verified

s2. You are the separate agent. Return blocked if {cases} holds anything. Write at least 3 black-box cases per requirement as {cases}/<requirement>/<variant>/caseNN.txt, where <requirement> names a noun and <variant> is a variant folder name from examples/EXAMPLES.md, in the spec format of examples/specs. Keep generators and other scratch files in {scratch}, never in the clone, which the method agents read. Log counts only in LOG.md and commit LOG.md.

s3. Run {split} {cases} {visible}, which keeps two thirds visible, and commit visible/. Log only the line it prints.

Blind review

Blind review of one method clone. A reviewer agent with a blind id scores src/ and test/ from git diff refs/tags/starter..HEAD. Every score runs from 1, worst, to 5, best, with a reason of one or two sentences. verified

Every run records the same review schema, with 11 scored items. recomputed

Alterability

How little code each change case of docs/SPEC.md touches, given that all combinations of both must work.

Code quality principles

The eight principles of cqp.html.

Test quality

How well the method's own tests in test/ pin the rules and variants of the spec, with clear names and independent cases.

A review agent scored each method's code and tests once from a prompt that left out the method's name, and the blind rating supersedes those scores. In the blind rating three raters score each item from 1 to 5 on each cqp.html principle. An item is a rebuilt clone whose history names no method. The finals are the 11 final clones. The calibration items add the A02 reference, the starter stub and 7 mutants of functions-first, each damaging one or two named principles. The scores are on the ratings page. verified

Collect and audit

After the build stages collect.sh collects each run. Its branchStep records a step marked skippable as skipped when fetchBack failed, as the method branch may then be stale. verified

collectOne runs 16 steps in order.

1. fetchBack
2. export skippable
3. logs skippable
4. grade skippable
5. measure skippable
6. survivors skippable
7. bench skippable
8. metrics skippable
9. diagram skippable
10. costs
11. curve skippable
12. apfd skippable
13. audit
14. adherence skippable
15. review
16. report

collectList runs 7 steps in order.

1. fetchBack
2. pin skippable
3. export skippable
4. list skippable
5. costs
6. audit
7. orders

Audit flag. A command or commit by an agent that may show access to the hidden tests. Each high flag gets a disposition. verified

3 rows judge 10 high flags. The evidence for each is in the audit flag dispositions. recomputed