Held-out TDD

Search and validation

What it changes: The check. Hidden tests score milestones.

Bronze Most cost effective M 519,499, M_io 443,244

One method of the study, from runs full-1 and rep-2. The survey build is the method's single build in run full-1. A replicate is one of its builds in run rep-2, which ran in a different batch. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

How it builds code

The procedure supplied to the agents. Indented lines run under the line above them.

Family: Search and validation. Idea: Visible tests drive the loop. Hidden tests measure overfitting. What it changes in the search: The check. Hidden tests score milestones.

once:1. Pin the public interface first such as the CLI or the noun-verb verbs.2. A human or separate agent writes several black-box cases per requirement.3. Move about a third of each requirement's cases into a hidden set.the harness does:4. Keep hidden tests in a private repo that only CI checks out. Keep its CI token out of the agent environment.5. Clone it outside the agent workspace. Add a permissions deny entry for that path in the assistant settings.repeat per requirement until none is open:6. Drive TDD and AI loops with the visible set alone.7. On each merge CI prints hidden pass counts per requirement. Log the gap.8. On a hidden failure write a fresh visible test. Replace that hidden case.if hidden failure, once:with Held-out TDD step 2. A human or separate agent writes several black-box cases per requirement.with Held-out TDD step 8. On a hidden failure write a fresh visible test. Replace that hidden case.if hidden failure, once:with Held-out TDD step 3. Move about a third of each requirement's cases into a hidden set.with Held-out TDD step 8. On a hidden failure write a fresh visible test. Replace that hidden case.

Once means one agent runs the steps. Repeat per unit means one agent per unit until no unit is open. Throughout means the rule joins every prompt. For up to n starts means separate attempts with the best kept. The harness does means the harness runs the step without an agent. If names the event that starts a stage, and with adds a rule borrowed from another method. Steps are quoted from args.json with their numbers. Tag: verified.

Figure 21.1

Stage flow of Held-out TDD

Once: step 1. Lane: Agent.
Next: Once: step 2.
Once: step 2. Lane: Agent.
Next: Once: step 3.
Once: step 3. Lane: Agent.
Next: Steps 4 and 5.
Steps 4 and 5. Lane: Harness.
Next: Steps 6 to 8.
Steps 6 to 8. Returns hiddenFail. Lane: Agent.
Next: With held-out steps 2 and 8 (on fail).

Loop: repeat per requirement, back to Steps 6 to 8.
With held-out steps 2 and 8. Lane: When hidden failure.
Next: With held-out steps 3 and 8.
With held-out steps 3 and 8. Lane: When hidden failure.

7 stages from the run facts, each checked against its block of the steps above. Details and notes are in the stage table. Tag: recomputed.

Data

Text version: Stage flow of Held-out TDD

Once: step 1. Lane: Agent.
Next: Once: step 2.
Once: step 2. Lane: Agent.
Next: Once: step 3.
Once: step 3. Lane: Agent.
Next: Steps 4 and 5.
Steps 4 and 5. Lane: Harness.
Next: Steps 6 to 8.
Steps 6 to 8. Returns hiddenFail. Lane: Agent.
Next: With held-out steps 2 and 8 (on fail).

Loop: repeat per requirement, back to Steps 6 to 8.
With held-out steps 2 and 8. Lane: When hidden failure.
Next: With held-out steps 3 and 8.
With held-out steps 3 and 8. Lane: When hidden failure.

Stages

Each stage as the run facts list it. Runs as and Note are quoted verbatim and Steps repeats the steps above. Tag: verified.

Stage	Runs as	Steps	Details	Note
Once: step 1	once:	1. Pin the public interface first such as the CLI or the noun-verb verbs.	none	none
Once: step 2	once:	2. A human or separate agent writes several black-box cases per requirement.	none	You are the separate agent. Return blocked if {cases} holds anything. Write at least 3 black-box cases per requirement as {cases}/<requirement>/<variant>/caseNN.txt, where <requirement> names a noun and <variant> is a variant folder name from examples/EXAMPLES.md, in the spec format of examples/specs. Keep generators and other scratch files in {scratch}, never in the clone, which the method agents read. Log counts only in LOG.md and commit LOG.md.
Once: step 3	once:	3. Move about a third of each requirement's cases into a hidden set.	none	Run {split} {cases} {visible}, which keeps two thirds visible, and commit visible/. Log only the line it prints.
Steps 4 and 5	the harness does:	4. Keep hidden tests in a private repo that only CI checks out. Keep its CI token out of the agent environment. 5. Clone it outside the agent workspace. Add a permissions deny entry for that path in the assistant settings.	none	none
Steps 6 to 8	repeat per requirement until none is open:	6. Drive TDD and AI loops with the visible set alone. 7. On each merge CI prints hidden pass counts per requirement. Log the gap. 8. On a hidden failure write a fresh visible test. Replace that hidden case.	unit requirement; returns hiddenFail	Drive with ./run.sh specs visible and ./run.sh test. Once the unit is green run ./run.sh hidden once and append a GAP.md row: time, label, requirement, visible pass/total, hidden pass/total and gap. Write a fresh visible test for each requirement ticked in UNITS.md that fails hidden. Write each fresh visible test under test/, never under visible/, which the harness re-splits. A spec file there goes in a variant folder, so ./run.sh specs test plays it. The harness does the rest of step 8 after this loop and replaces the hidden cases through separate agents, so leave the hidden set alone. Return in failing only the requirements ticked in UNITS.md that fail hidden, because the others are not built yet.
With held-out steps 2 and 8	if hidden failure, once:	with Held-out TDD step 2. A human or separate agent writes several black-box cases per requirement. with Held-out TDD step 8. On a hidden failure write a fresh visible test. Replace that hidden case.	when hidden_failure; with held-out steps 2 and 8	You are the separate agent. Add fresh cases to {cases} for each requirement the facts below list as failing hidden, in the layout of {cases}. Keep generators and other scratch files in {scratch}, never in the clone, which the method agents read. Log counts only in LOG.md and commit LOG.md.
With held-out steps 3 and 8	if hidden failure, once:	with Held-out TDD step 3. Move about a third of each requirement's cases into a hidden set. with Held-out TDD step 8. On a hidden failure write a fresh visible test. Replace that hidden case.	when hidden_failure; with held-out steps 3 and 8	Run {split} {cases} {visible} again and commit visible/. Log the line it prints and the count of earlier cases it moved into visible/. Then run ./run.sh specs visible and ./run.sh hidden once and append a closing GAP.md row.

Survey build

One build in run full-1.

15/15Functional mark

33E

801,898Cost to completion

519,499M

443,244M_io

4PMD findings

noAlignment

Replicates

Repeat builds: 5 in run rep-2.

0.777Median adjusted mutation, over 5 replicates in rep-2

3Median classes, over 5 replicates in rep-2

973,901Median cost to completion, over 5 replicates in rep-2

17Median CQP, over 5 replicates in rep-2

M against Classic greedy TDD: U 11, Holm p 1.000; adjusted mutation against Classic greedy TDD: U 0, Holm p 0.024Planned contrast, exact, Holm, rep-2

0.96Consistency score C, exploratory over rep-1 and rep-2, lower is more consistent

Verdict

One survey build and 5 replicates in rep-2. The survey build passed the largest share of hidden tests at 500000 weighted tokens, 0.42, and had the lowest adjusted mutation score of the survey, 0.794. The M test did not detect a difference from Classic greedy TDD (U 11 and Holm p 1.000). The replicates' median share at 500000 weighted tokens was 0.00 in a different batch from the survey build. Their median adjusted mutation of 0.777 was lower than Classic greedy TDD's 1.000, Holm p 0.024, in line with the survey build. Its 9 survey and 46 rep-2 high audit flags were each judged a false positive. Tag: recomputed.

Report summary

The report's class diagram for this design is identical to Figure 21.2 below, so it is shown once. Tag: recomputed.

Functional mark /15	15/15
Hidden pass /76	76/76
Unseen pass /69	69/69
Strict PMD compliance	fail (4 violations)
Mutation score	112/149, 0 not run, adjusted 112/141 (29 of 37 survivors fail hidden tests)
Classes	5
Simple Constructs /5	4
Minimal Duplication /5	4
Modular Structure /5	5
Problem Alignment (tentative) /5	4
Weighted tokens	844,959
Bench median ms	606

Score card

Survey tiles place the survey build among the single survey builds. One build per method describes that build and does not rank the methods. Contrast tiles are medians over the replicates. Tag: recomputed.

Adjusted mutation0.79411th highest of 11 single survey builds, one per method

Classes54th highest of 11 single survey builds, one per method

Cost to completion801,8982nd lowest of 11 single survey builds, one per method

Strict PMD violations45th lowest of 11 single survey builds, one per method

Median adjusted mutation0.777over 5 replicates in rep-2

Median classes3over 5 replicates in rep-2

Median cost to completion973,901over 5 replicates in rep-2

Median CQP17over 5 replicates in rep-2

Planned contrastM against Classic greedy TDD: U 11, Holm p 1.000; adjusted mutation against Classic greedy TDD: U 0, Holm p 0.024exact, Holm, rep-2

Consistency score C0.96exploratory over rep-1 and rep-2, lower is more consistent

Builds

Best, median and worst order a method's replicates by adjusted mutation, highest first, then lower cost to completion. The median build is the representative. Tag: recomputed.

Survey build (full-1)

Build	held-out
Adjusted mutation	0.794
Classes	5
Cost to completion	801,898
Strict PMD violations	4

Figure 21.2

Class diagram of method/held-out

5 classes and 9 links. Tag: recomputed.

Data

Text version: Class diagram of method/held-out

Package kalah: Board, BoardView, Game, Kalah, Protocol
BoardView uses Board
Game uses Board
Game holds Board
Game uses BoardView
Game holds BoardView
Game uses Protocol
Kalah uses Board
Kalah uses BoardView
Kalah uses Game

Best replicate (rep-2-r20)

Replicate	rep-2-r20
Method	Held-out TDD
Adjusted mutation	0.983
Unadjusted mutation	0.927
Classes	3
Cost to completion	1,088,356
CQP	17
CQP accepted	13
Strict PMD violations	3
Build	best

Figure 21.3

Class diagram of rep-2-r20

3 classes and 3 links. Tag: recomputed.

Data

Text version: Class diagram of rep-2-r20

Package kalah: Board, BoardView, Kalah
BoardView uses Board
Kalah uses Board
Kalah uses BoardView

Median replicate (rep-2-r04)

Replicate	rep-2-r04
Method	Held-out TDD
Adjusted mutation	0.777
Unadjusted mutation	0.725
Classes	3
Cost to completion	1,021,062
CQP	17
CQP accepted	13
Strict PMD violations	12
Build	median

Figure 21.4

Class diagram of rep-2-r04

3 classes and 3 links. Tag: recomputed.

Data

Text version: Class diagram of rep-2-r04

Package kalah: Board, BoardPrinter, Kalah
BoardPrinter uses Board
Kalah uses Board
Kalah uses BoardPrinter

Worst replicate (rep-2-r17)

Replicate	rep-2-r17
Method	Held-out TDD
Adjusted mutation	0.723
Unadjusted mutation	0.688
Classes	3
Cost to completion	926,018
CQP	17
CQP accepted	13
Strict PMD violations	8
Build	worst

Figure 21.5

Class diagram of rep-2-r17

3 classes and 3 links. Tag: recomputed.

Data

Text version: Class diagram of rep-2-r17

Package kalah: Board, BoardView, Kalah
BoardView uses Board
Kalah uses Board
Kalah uses BoardView

Same family comes from the methods doc. Nearest and farthest average the rank differences over adjusted mutation, classes and cost to completion of the survey builds, over the metrics both methods have. Anchor Engineering has no survey build, so the replicate medians of adjusted mutation and classes stand in. Cost to completion leaves out such a method, since costs of a different batch do not compare. Tag: recomputed.

Records

Replicates (rep-2)

Replicates (rep-2), 5 rows and 10 columns

Replicate	Method	Adjusted mutation	Unadjusted mutation
rep-2-r04	Held-out TDD	0.777	0.725
rep-2-r08	Held-out TDD	0.777	0.734
rep-2-r14	Held-out TDD	0.736	0.709
rep-2-r17	Held-out TDD	0.723	0.688
rep-2-r20	Held-out TDD	0.983	0.927

Replicate	Classes	Cost to completion	CQP
rep-2-r04	3	1,021,062	17
rep-2-r08	5	904,350	16
rep-2-r14	3	973,901	16
rep-2-r17	3	926,018	17
rep-2-r20	3	1,088,356	17

Replicate	CQP accepted	Strict PMD violations	Build
rep-2-r04	13	12	median
rep-2-r08	12	6	-
rep-2-r14	12	7	-
rep-2-r17	13	8	worst
rep-2-r20	13	3	best

Consistency

Method	Held-out TDD
Adjusted mutation	1.36
Unadjusted mutation	1.73
Classes	0.57
Cost to completion	0.59
Strict PMD violations	1.12
CQP accepted	0.84
Consistency score C	0.96

Clone reports

Each clone's own report, as in its REPORT.md.

Held-out TDD: functional mark 15/15, strict PMD fail (4 violations), classes 5, weighted tokens 844,959

Branch method/held-out. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

Headline

Functional mark /15	15/15
Hidden pass /76	76/76
Unseen pass /69	69/69
Acceptance pass /6	6/6
Strict PMD compliance	fail (4 violations)

Recomputed from the grader's grade.tsv and summary.json and from metrics.json. The functional mark follows A02-RUBRIC.md: 2 for the original set with no failure, 3 per other set less one per failure, and 4 when every test passes. A02 is the course assignment whose spec and rubric the experiment uses. STA means anticlockwise and STC clockwise sowing. 6-4 and 7-5 are houses and seeds. A02-RUBRIC.md names the original set "Clockwise 6 Houses 4 Seeds". This page names each set by the spec folder its tests play, so the original set reads STA 6-4. Strict PMD compliance passes when PMD with the starter's kalah_pmd.xml finds no violation in src/ and otherwise fails with the count of distinct violations. The lint sets no minimum priority, so rules of every priority count. It covers src/ only, the scope of the starter's ./run.sh lint. A02-RUBRIC.md awards no marks for PMD, so the functional mark leaves it out.

Rubric set	Tests	Marks
Original set, STA 6-4	testST64SKHHSH_	2/2
STC 6-4	testSTC64SKHHSH_	3/3
STA 7-5	testST75SKHHSH_	3/3
STC 7-5	testSTC75SKHHSH_	3/3
All requirements	all tests	4/4

Summary metrics

Metric	Value	Tag
Total lines	417 (src 269, own tests 148)	recomputed
Lines of code	331 (src 210, own tests 121)	recomputed
Classes	5	recomputed
Methods	26	recomputed
Fields	13	recomputed
Bench median ms	606	recomputed
Bench volume	100614 inputs, 40083576 output chars	recomputed

Own tests leave out the starter's two Example tests and their ExampleSpec helper. Classes count top-level, nested and anonymous classes in src/.

Quality, costs and audit

Metric	Value	Tag
Mutation score	112/149, 0 not run, adjusted 112/141 (29 of 37 survivors fail hidden tests)	recomputed
PMD lines	4 (4 violations)	recomputed
Dependency edges	7	recomputed
Review scores (superseded)	Alterability 5, Code quality principles 4.1, Test quality 3	unchecked
Tokens	4,966,338 (input 268, cache creation 408,339, cache read 4,536,240, output 21,491)	recomputed
Weighted tokens	844,959	recomputed
Wall time	0:21:52	recomputed
Agent time	0:21:03	recomputed
Agent calls	11	recomputed
Audit flags	9 high, 4 low, 9 of 9 high have a disposition	recomputed

PMD lines count violations once per file and line. Agent time sums each agent's first to last timestamp, so agents that ran side by side add up. Weighted tokens count input 1, cache creation 1.25, cache read 0.05 and output 5 per token. These are claude-opus-5-5 prices relative to an input token and state no dollar cost. Costs, times and agent calls cover this clone only. The comparison row covers every clone of the method as its count basis states.

Review scores

A blind reviewer agent scored src/ and test/ from 1, worst, to 5, best, so the scores are unchecked. Reasons are the reviewer's own words, verbatim. This review is superseded by the blind rating of three raters in rating.json, whose medians fill the comparison's principle columns.

Review scores, 11 rows and 4 columns

Alterability. Item: Change Case 1: sow seeds anti-clockwise. Score /5: 5.

Reason: Direction is a pure view concern here: Board sows the same way in every variant and only BoardView's clockwise branch picks the row layout, so the change touches one small conditional.

Alterability. Item: Change Case 2: 7 houses with 5 seeds. Score /5: 5.

Reason: House count and seed count are constructor parameters used throughout Board and BoardView, so the variant is a single ternary in Kalah.play and all four combinations work without other edits.

Code quality principles. Item: 1. Clear Presentation. Score /5: 4.

Reason: The files are small and well formatted, with a clear order and related helpers next to each other. The sow loop's side/pit state is dense to scan.

Code quality principles. Item: 2. Explanatory Language. Score /5: 4.

Reason: Names and class Javadocs state intent well, and CANCELLED and the Protocol constants give literals meaning. The magic expressions 3 - side, 2 - player and player - 1 are never named.

Code quality principles. Item: 3. Consistent Code. Score /5: 4.

Reason: Players and houses are 1-based throughout and Protocol handles every message the same way. Opponent indexing switches between 3 - side and 2 - player, which is an inconsistent idiom.

Code quality principles. Item: 4. Used Content. Score /5: 5.

Reason: Every class, method and constant is used. The stub output was removed and nothing dead remains.

Code quality principles. Item: 5. Simple Constructs. Score /5: 3.

Reason: Game.run is a simple loop. Board.sow, though, encodes the last-seed-in-store case through the side flip and pit reset, and needs a comment to explain its return tests.

Code quality principles. Item: 6. Minimal Duplication. Score /5: 4.

Reason: Row rendering shares a houses helper with an ascending flag, and draw is factored out. Game over plus draw appears in two branches of run, and pit and pitEnd nearly duplicate each other.

Code quality principles. Item: 7. Modular Structure. Score /5: 5.

Reason: The responsibilities split cleanly into Board as the model, BoardView for rendering, Protocol for text and Game for the turn loop. Classes are package-private with narrow interfaces.

Code quality principles. Item: 8. Problem Alignment. Score /5: 4.

Reason: Board, house, store, sow and capture follow the domain vocabulary, and the variants map directly to constructor args and the view. The integer side/pit cursor is a lower-level encoding than the domain suggests.

Test quality. Item: Test quality. Score /5: 3.

Reason: BoardTest pins sowing, an extra turn, capture and scoring with clear names, and BoardViewTest covers two variant layouts. No own test exercises Game flow such as quit, empty house or end of game. ProtocolTest and PublicInterfaceTest mostly restate constants and signatures.

Audit flags

9 high and 4 low flags from agent tool calls and the method's commits. Only audit.json lists the low flags.

Channel	Agent or commit	Tool	Path
ho_reflection	held-out:s1	Write	test/kalah/PublicInterfaceTest.java
ho_reflection	held-out:s1	Write	test/kalah/PublicInterfaceTest.java
ho_reflection	held-out:s1	Write	test/kalah/PublicInterfaceTest.java
hidden	held-out:s2	Bash	$CLONES_DIR/_heldout
hidden	held-out:s2	Bash	$CLONES_DIR/_heldout
ho_reflection	9640df5925beb1dca6dded74b969b49f64fd7fad	-	test/kalah/PublicInterfaceTest.java:3
ho_reflection	9640df5925beb1dca6dded74b969b49f64fd7fad	-	test/kalah/PublicInterfaceTest.java:4
ho_reflection	9640df5925beb1dca6dded74b969b49f64fd7fad	-	test/kalah/PublicInterfaceTest.java:13
ho_reflection	9640df5925beb1dca6dded74b969b49f64fd7fad	-	test/kalah/PublicInterfaceTest.java:17

Channel	Marker	Disposition
ho_reflection	java.lang.reflect	false positive
ho_reflection	getConstructor(	false positive
ho_reflection	getMethod(	false positive
hidden	_heldout	false positive
hidden	_heldout	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	getConstructor(	false positive
ho_reflection	getMethod(	false positive

ho_reflection.

Definition: Held-out only. Added code or a tool write that uses reflection.

hidden.

Definition: A path named /hidden or a _heldout path outside the agent's allowed _heldout folders. The held-out method's own case folders live under _heldout.

Dispositions from the run's findings.json judge every high flag of their channel.

hidden. Verdict: false positive. Tag: verified.

Evidence: Both flags are held-out:s2 naming _heldout itself. One Bash call lists _heldout and _heldout/cases and one changes into _heldout to write cases with its scratch generator. Neither reads _heldout/hidden, and s2 may use _heldout/cases and _heldout/scratch.

ho_reflection. Verdict: false positive. Tag: verified.

Evidence: 3 flags mark held-out:s1's Write of test/kalah/PublicInterfaceTest.java and 4 mark its lines in commit 9640df5. The test reflects only on Kalah to check a public no-argument constructor and a public play(IO, boolean, boolean). It reads no spec and no MockIO state.

Step log

Method agents wrote the LOG.md cells. Agent-written text is shown verbatim and unchecked. Words such as recomputed inside it are the agent's own and are not tags of this page. Its Measure lines describe measurement during the run. Harness scores were measured separately after the run. Agent text may quote git commit times in the host's local zone. Harness times are UTC. Agent, tokens and seconds are recomputed from transcripts by row time. A dash in Agent means no agent run spans the row.

Step log, 10 rows and 10 columns

2026-10-15T23:54:08Z. Step: s1: 1 pin public interface. Unit: public interface: play(IO, boolean, boolean) and CLI protocol. Tests added: 8. Cycles: 1. Sha: 9640df5. Agent: held-out:s1. Tokens: 375190. Seconds: 63.417.

Actions: Added PublicInterfaceTest (reflection pin of public constructor and play signature, green on stub as a pin of the given contract). Added ProtocolTest red (no Protocol class) then src/kalah/Protocol.java green with the quit verb, prompt, empty-house, game-over, score, winner and tie texts from examples/specs. "Player 1 wins!" inferred by symmetry. Ran ./run.sh test ProtocolTest and PublicInterfaceTest only. Conflict logged: the step names no checks while standing guidance asks for static checks and related tests, so only the two new test classes ran. Full ./run.sh test stays red on the six example tests by design of the starter.

2026-10-15T23:57:39Z. Step: s2: 2 write black-box cases per requirement. Unit: held-out case set: 7 requirements x 4 variants. Tests added: 84. Cycles: 0. Sha: 060fc46. Agent: held-out:s2. Tokens: 786487. Seconds: 212.054.

Actions: Wrote 84 spec-format cases outside the clone: 7 requirements, 3 cases per requirement per variant, 12 per requirement. Generated and checked outside the clone. The generator replays all 6 example specs exactly. No ./run.sh call because the step names no checks. Conflict logged: standing guidance asks to commit each work group, while the step keeps cases out of the clone, so only LOG.md is committed.

2026-10-15T23:57:58Z. Step: s3: 3 move a third of each requirement into a hidden set. Unit: held-out split: 7 requirements. Tests added: 0. Cycles: 0. Sha: ab6140f. Agent: held-out:s3. Tokens: 94348. Seconds: 11.044.

Actions: split: 84 cases in 7 requirements, 56 visible, 28 hidden

2026-10-16T00:00:45Z. Step: L6-8 iter 1: 6 visible-only TDD, 7 hidden gap, 8 fresh visible tests. Unit: quit. Tests added: 5. Cycles: 3. Sha: e30c970. Agent: held-out:L6-8#1. Tokens: 1344397. Seconds: 172.235.

Actions: Wrote UNITS.md (7 requirements) and committed it. Red-green: BoardViewTest (board drawing both directions), BoardTest sowing, BoardTest capture. Added Board, BoardView, Game and wired Kalah.play. Visible quit 8/8 needed capture because quit STC 7-5 case03 ends a move with a capture, so capture was built in this unit; logged as a deviation from one requirement per iteration. ./run.sh test: 4 own classes green, example tests still red for later units (empty house, full games). ./run.sh hidden once: quit 4/4, gap 0, so no fresh visible test. Conflict logged: standing guidance asks for static checks and full CI green, while the step names only specs, test and hidden, so lint was not run and CI stays red on later-unit examples.

2026-10-16T00:09:13Z. Step: L6-8 iter 2: 6 visible-only TDD, 7 hidden gap, 8 fresh visible tests. Unit: sowing. Tests added: 1. Cycles: 1. Sha: 3aa6c9d. Agent: held-out:L6-8#2. Tokens: 398681. Seconds: 494.569.

Actions: Visible sowing was 4/8: each case03 ends a move in the own store and expects another turn. Red BoardTest.testLastSeedInOwnStoreGivesAnotherTurn, then Board.sow returns the same player when the last seed lands in the own store. Visible sowing 8/8; store and lap visible also turned 8/8 from the same rule. ./run.sh test: 4 own classes green, example tests still red on empty house and full games for later units. ./run.sh hidden once: sowing 4/4, gap 0, quit still 4/4, so no fresh visible test. Conflict logged: standing guidance asks for static checks and green CI, while the step names only specs, test and hidden, so lint was not run and CI stays red on later-unit examples.

2026-10-16T00:10:43Z. Step: L6-8 iter 3: 6 visible-only TDD, 7 hidden gap, 8 fresh visible tests. Unit: store. Tests added: 0. Cycles: 0. Sha: 984a0be. Agent: held-out:L6-8#3. Tokens: 413144. Seconds: 98.648.

Actions: ./run.sh specs visible: store already 8/8 from the iteration 2 another-turn rule, so the visible set gave no red case and no code or test changed (0 cycles). ./run.sh test: 4 own classes green, example tests still red on empty house and full games for later units. ./run.sh hidden once: store 4/4, gap 0, quit and sowing still 4/4, so no fresh visible test. Ticked store. Conflict logged: standing guidance asks for new tests per change and static checks, while the step drives only from the visible set and names only specs, test and hidden, so no test was added and lint was not run.

2026-10-16T00:11:23Z. Step: L6-8 iter 4: 6 visible-only TDD, 7 hidden gap, 8 fresh visible tests. Unit: capture. Tests added: 0. Cycles: 0. Sha: 0017295. Agent: held-out:L6-8#4. Tokens: 216835. Seconds: 28.061.

Actions: ./run.sh specs visible: capture already 8/8 because iteration 1 built capture for a quit case, so the visible set gave no red case and no code or test changed (0 cycles). Specs ran twice to filter output, with the same result. ./run.sh test: 4 own classes green, example tests still red on empty house and full games for later units. ./run.sh hidden once: capture 4/4, gap 0, quit sowing and store still 4/4, so no fresh visible test. Ticked capture. Conflict logged: standing guidance asks for new tests per change and static checks, while the step drives only from the visible set and names only specs, test and hidden, so no test was added and lint was not run.

2026-10-16T00:12:19Z. Step: L6-8 iter 5: 6 visible-only TDD, 7 hidden gap, 8 fresh visible tests. Unit: emptyhouse. Tests added: 1. Cycles: 1. Sha: 47cffd9. Agent: held-out:L6-8#5. Tokens: 503949. Seconds: 61.467.

Actions: ./run.sh specs visible: emptyhouse 0/8, all other built requirements 8/8. Red BoardTest.testHouseIsEmptyOnlyAfterItIsSown (no Board.isEmpty), then Board.isEmpty and Game prints Protocol.EMPTY_HOUSE and reprompts without sowing. Visible emptyhouse 8/8. ./run.sh test: BoardTest, BoardViewTest, ProtocolTest and PublicInterfaceTest green, example full-game tests still red on game end for a later unit. ./run.sh hidden once: emptyhouse 4/4, gap 0, quit sowing store capture lap 4/4, gameend 0/4 (not built), so no fresh visible test. Ticked emptyhouse. Conflict logged: standing guidance asks for static checks and green CI, while the step names only specs, test and hidden, so lint was not run and CI stays red on game-end examples.

2026-10-16T00:12:56Z. Step: L6-8 iter 6: 6 visible-only TDD, 7 hidden gap, 8 fresh visible tests. Unit: lap. Tests added: 0. Cycles: 0. Sha: 657940a. Agent: held-out:L6-8#6. Tokens: 215211. Seconds: 28.592.

Actions: ./run.sh specs visible: lap already 8/8 from the iteration 2 another-turn rule and sowing across the opponent store skip, so the visible set gave no red case and no code or test changed (0 cycles). Specs ran twice to filter output, with the same result. ./run.sh test: own classes green, example full-game tests still red on game end for a later unit. ./run.sh hidden once: lap 4/4, gap 0, other built requirements 4/4, gameend 0/4 (not built), so no fresh visible test. Ticked lap. Conflict logged: standing guidance asks for new tests per change and static checks, while the step drives only from the visible set and names only specs, test and hidden, so no test was added and lint was not run.

2026-10-16T00:14:05Z. Step: L6-8 iter 7: 6 visible-only TDD, 7 hidden gap, 8 fresh visible tests. Unit: gameend. Tests added: 1. Cycles: 1. Sha: 0232958. Agent: held-out:L6-8#7. Tokens: 511102. Seconds: 71.004.

Actions: ./run.sh test red: BoardTest.testNoMovesOnceEveryOwnHouseIsEmptyAndScoreAddsHousesToStore did not compile (no Board.hasMoves or Board.score). Added both, and Game now prints Game over, the board, both score lines and the winner or a tie when the player to move has no seeds. ./run.sh test: 6 classes green, example full-game tests now pass. ./run.sh specs visible: 56/56, gameend 8/8. ./run.sh hidden once: every requirement 4/4, gameend gap 0, so no fresh visible test. Ticked gameend, UNITS.md has no open lines. Conflict logged: standing guidance asks for static checks, while the step names only specs, test and hidden, so lint was not run.

Raw inputs

metrics.json
summary.json
grade.tsv
costs.json
steps.csv
audit.json
review.json
LOG.md
MEASURE.md
classdiagram.uml
classdiagram.png
classdiagram.svg
collect.tsv
rating.json

MEASURE.md

The method's measure agent answered its Measure line from LOG.md and git. Agent-written text is shown verbatim and unchecked. Words such as recomputed inside it are the agent's own and are not tags of this page. Its Measure lines describe measurement during the run. Harness scores were measured separately after the run. Agent text may quote git commit times in the host's local zone. Harness times are UTC.

# Measure

Measure: visible and hidden pass rate per merge. Gap per noun. Hidden cases replaced after disclosure.

Sources: LOG.md, GAP.md and git log. ATTEMPTS.md does not exist. Credits and time are measured by the harness and are not recorded here.

The run has no merge commits. Each held-out iteration lands as one tick commit on the work branch, so each tick commit counts as one merge.

## Pass rate per merge

Case set: 84 cases over 7 requirements. 56 visible (8 per requirement) and 28 hidden (4 per requirement).

| merge | sha | unit | unit visible | unit hidden | all visible | all hidden |
|---|---|---|---|---|---|---|
| iter1 | e30c970 | quit | 8/8 | 4/4 | not recorded | not recorded |
| iter2 | 3aa6c9d | sowing | 8/8 | 4/4 | not recorded | not recorded |
| iter3 | 984a0be | store | 8/8 | 4/4 | not recorded | not recorded |
| iter4 | 0017295 | capture | 8/8 | 4/4 | not recorded | not recorded |
| iter5 | 47cffd9 | emptyhouse | 8/8 | 4/4 | 48/56 | 24/28 |
| iter6 | 657940a | lap | 8/8 | 4/4 | 48/56 | 24/28 |
| iter7 | 0232958 | gameend | 8/8 | 4/4 | 56/56 | 28/28 |

Unit columns come from GAP.md. All-set columns come from LOG.md. Iterations 1 to 4 logged only the built requirements, so their all-set totals are not recorded. At iterations 5 and 6 LOG.md states every requirement except gameend at 8/8 visible and 4/4 hidden, and gameend 0/4 hidden. Gameend visible 0/8 at those merges is inferred from it being unbuilt.

## Gap per noun

Gap is visible pass rate minus hidden pass rate for the iteration's requirement.

| noun | visible | hidden | gap |
|---|---|---|---|
| quit | 8/8 | 4/4 | 0 |
| sowing | 8/8 | 4/4 | 0 |
| store | 8/8 | 4/4 | 0 |
| capture | 8/8 | 4/4 | 0 |
| emptyhouse | 8/8 | 4/4 | 0 |
| lap | 8/8 | 4/4 | 0 |
| gameend | 8/8 | 4/4 | 0 |

Every noun closed with a gap of 0 at its first hidden run.

## Hidden cases replaced after disclosure

0. No hidden run showed a gap, so no hidden case was disclosed and none was moved to the visible set or replaced. Each iteration ran ./run.sh hidden once.

## Other counts

- Red-green cycles: 7 (s1 1, iter1 3, iter2 1, iter5 1, iter7 1). Iterations 3, 4 and 6 ran 0 cycles because earlier rules already passed their visible cases.
- Tests added per LOG.md: 8 in s1 and 8 over the iterations. The 84 held-out spec cases live outside the clone.
- Deviation: capture was built in iteration 1 because a visible quit case ends with a capture.

Replicate rep-2-r04: functional mark 15/15, strict PMD fail (12 violations), classes 3, weighted tokens 1,069,622

Replicate rep-2-r04

Branch method/rep-2-r04. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

Headline

Functional mark /15	15/15
Hidden pass /76	76/76
Unseen pass /69	69/69
Acceptance pass /6	6/6
Strict PMD compliance	fail (12 violations)

Rubric set	Tests	Marks
Original set, STA 6-4	testST64SKHHSH_	2/2
STC 6-4	testSTC64SKHHSH_	3/3
STA 7-5	testST75SKHHSH_	3/3
STC 7-5	testSTC75SKHHSH_	3/3
All requirements	all tests	4/4

Summary metrics

Metric	Value	Tag
Total lines	305 (src 188, own tests 117)	recomputed
Lines of code	248 (src 148, own tests 100)	recomputed
Classes	3	recomputed
Methods	18	recomputed
Fields	6	recomputed
Bench median ms	571	recomputed
Bench volume	100614 inputs, 40083576 output chars	recomputed

Own tests leave out the starter's two Example tests and their ExampleSpec helper. Classes count top-level, nested and anonymous classes in src/.

Quality, costs and audit

Metric	Value	Tag
Mutation score	87/120, 0 not run, adjusted 87/112 (25 of 33 survivors fail hidden tests)	recomputed
PMD lines	12 (12 violations)	recomputed
Dependency edges	3	recomputed
Review scores (superseded)	Alterability 5, Code quality principles 4.2, Test quality 3	unchecked
Tokens	5,315,209 (input 290, cache creation 410,956, cache read 4,841,248, output 62,715)	recomputed
Weighted tokens	1,069,622	recomputed
Wall time	0:18:03	recomputed
Agent time	0:16:47	recomputed
Agent calls	12	recomputed
Audit flags	9 high, 5 low, 9 of 9 high have a disposition	recomputed

Review scores

Review scores, 11 rows and 4 columns

Alterability. Item: Change Case 1: sow seeds anti-clockwise. Score /5: 5.

Reason: Direction lives only in BoardPrinter as one boolean, because sowing in rising house order is the same for both variants and only the drawn layout mirrors. Board and the game loop never read the flag.

Alterability. Item: Change Case 2: 7 houses with 5 seeds. Score /5: 5.

Reason: The variant is a single constructor choice in play(). Board, printer borders and the input range all come from houseCount(), so nothing else changes.

Code quality principles. Item: 1. Clear Presentation. Score /5: 4.

Reason: Formatting is tidy and the three classes are small and easy to scan. The QUIT constant sits at the bottom of Kalah, away from where it is used, and the clockwise and anticlockwise print branches are dense string concatenations.

Code quality principles. Item: 2. Explanatory Language. Score /5: 4.

Reason: Names such as sow, captureAt, isEmpty and score match the domain, and the javadoc explains why direction affects only drawing. P1, P2 and QUIT are symbolic, but Kalah still uses bare 0, 1 and 1 - player for players.

Code quality principles. Item: 3. Consistent Code. Score /5: 4.

Reason: Style is mostly uniform. Players are named constants in BoardPrinter but literals in Kalah and Board, and java.util.Arrays is written fully qualified three times where an import would do.

Code quality principles. Item: 4. Used Content. Score /5: 4.

Reason: The production code has no dead elements. Two added transcript files, emptyhouse/p2_empty.txt and quit/p1_quit_after_extra_move.txt, are referenced by no test and so never run.

Code quality principles. Item: 5. Simple Constructs. Score /5: 4.

Reason: The control flow is short and readable. sow() nests a store check inside a wrap check inside the loop with an early return, which is the hardest part to follow.

Code quality principles. Item: 6. Minimal Duplication. Score /5: 4.

Reason: Row drawing and store formatting are shared helpers. The play loop repeats the 'Game over' plus print sequence on its two exit paths.

Code quality principles. Item: 7. Modular Structure. Score /5: 5.

Reason: Board holds the state and rules, BoardPrinter handles rendering and Kalah handles the turn loop and I/O. Each class has one clear job and package-private scope.

Code quality principles. Item: 8. Problem Alignment. Score /5: 5.

Reason: The classes and methods match Kalah concepts directly: houses, stores, sowing, capture, extra move, empty house and score. Seeing that direction is only a display mirror keeps the rules in one model.

Test quality. Item: Test quality. Score /5: 3.

Reason: BoardTest covers sowing, extra moves, skipping the opponent store, captures and laps with independent cases, and InterfaceTest pins quit for all four variants. Empty-house and game-over transcripts are not run, the added spec files are never referenced, and some capture setups are opaque move sequences.

Audit flags

9 high and 5 low flags from agent tool calls and the method's commits. Only audit.json lists the low flags.

Channel	Agent or commit	Tool	Path
ho_reflection	rep-2-r04:s1	Write	test/kalah/InterfaceTest.java
ho_reflection	rep-2-r04:s1	Write	test/kalah/InterfaceTest.java
ho_reflection	rep-2-r04:s1	Write	test/kalah/InterfaceTest.java
hidden	rep-2-r04:s2	Bash	$CLONES_DIR/_heldout/rep-2-r04
hidden	rep-2-r04:s2	Bash	$CLONES_DIR/_heldout/rep-2-r04
ho_reflection	1362bccab8a7ddf9f3c4f4bd28623610fce8245f	-	test/kalah/InterfaceTest.java:4
ho_reflection	1362bccab8a7ddf9f3c4f4bd28623610fce8245f	-	test/kalah/InterfaceTest.java:5
ho_reflection	1362bccab8a7ddf9f3c4f4bd28623610fce8245f	-	test/kalah/InterfaceTest.java:11
ho_reflection	1362bccab8a7ddf9f3c4f4bd28623610fce8245f	-	test/kalah/InterfaceTest.java:14

Channel	Marker	Disposition
ho_reflection	java.lang.reflect	false positive
ho_reflection	getMethod(	false positive
ho_reflection	getConstructor(	false positive
hidden	_heldout	false positive
hidden	_heldout	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	getMethod(	false positive
ho_reflection	getConstructor(	false positive

ho_reflection.

Definition: Held-out only. Added code or a tool write that uses reflection.

hidden.

Definition: A path named /hidden or a _heldout path outside the agent's allowed _heldout folders. The held-out method's own case folders live under _heldout.

Dispositions from the run's findings.json judge every high flag of their channel.

hidden. Verdict: false positive. Tag: verified.

Evidence: Both flags are rep-2-r04:s2 in agent-ab883340dd8047793.jsonl. At line 14 it lists its own _heldout/rep-2-r04 and its cases before writing, and at line 96 it copies scratch/out into cases. It names no hidden path and no other build's dir.

ho_reflection. Verdict: false positive. Tag: verified.

Evidence: All 7 flags mark rep-2-r04:s1's InterfaceTest, 3 as a Write and 4 in commit 1362bcc. It reflects only on Kalah to check its public constructor and play. None reads a spec or MockIO state.

Step log

Step log, 11 rows and 10 columns

2026-10-30T06:01:56Z. Step: s1. Unit: public interface: play(IO, boolean, boolean) and quit CLI per variant. Tests added: 5. Cycles: 1. Sha: 1362bcc. Agent: rep-2-r04:s1. Tokens: 762474. Seconds: 131.446.

Actions: Pinned play() signature by reflection and the start board, prompt, 'q' and Game over transcript for all four variants (3 new specs under test/pin plus example quit.txt). Added Board and BoardPrinter for the start board. Ran ./run.sh test InterfaceTest only: red 4 errors then green 5/5. Conflict: Write and Edit strip trailing spaces, so the prompt's trailing space in the 3 spec files was restored with one sed call against the no-scripted-edits rule.

2026-10-30T06:05:20Z. Step: s2. Unit: black-box cases per requirement (separate agent). Tests added: 0. Cycles: 0. Sha: 839e9d5. Agent: rep-2-r04:s2. Tokens: 1038901. Seconds: 203.465.

Actions: Wrote 100 held-out cases outside the clone: 8 requirements x 4 variants, 3 per variant (gameend 4). Counts per requirement: board 12, sowing 12, store 12, capture 12, emptyhouse 12, quit 12, lap 12, gameend 16. No ./run.sh call; this step names no check.

2026-10-30T06:05:46Z. Step: s3. Unit: hidden split of the held-out cases. Tests added: 0. Cycles: 0. Sha: 9cd29c9. Agent: rep-2-r04:s3. Tokens: 124481. Seconds: 20.22.

Actions: split: 100 cases in 8 requirements, 67 visible, 33 hidden

2026-10-30T06:07:47Z. Step: L6-8 iter 1. Unit: board. Tests added: 3. Cycles: 2. Sha: 36b4161. Agent: rep-2-r04:L6-8#1. Tokens: 647897. Seconds: 114.186.

Actions: Wrote UNITS.md with 8 requirements. Red: ./run.sh specs visible/board 0/8. Added Board.sow (rising house order, skip opponent store, extra move on store, capture) and turn alternation in Kalah. Green: visible board 8/8. Added BoardTest (3 tests). First capture test used an invented move list and failed, fixed to the case03 sequence. ./run.sh test: BoardTest and InterfaceTest green, Example tests still red on empty-house and game-end units not built yet. ./run.sh hidden once: specs 0, so the runner has no hidden set and the gap is not measurable. Conflict: used sed for the UNITS.md tick and printf for GAP.md against the no-scripted-edits rule.

2026-10-30T06:08:37Z. Step: L6-8 iter 2. Unit: sowing. Tests added: 1. Cycles: 0. Sha: 549b964. Agent: rep-2-r04:L6-8#2. Tokens: 285100. Seconds: 55.755.

Actions: ./run.sh specs visible/sowing was already 8/8 from iteration 1, so no code change and no red phase. Added BoardTest.testSowDropsOneSeedPerFollowingHouse (7 houses, 5 seeds) as a characterization test. ./run.sh test: BoardTest 4/4 and InterfaceTest green, Example tests still red on units not built yet. ./run.sh hidden once: specs 0, so the gap is not measurable. Ticked sowing and logged GAP.md.

2026-10-30T06:09:38Z. Step: L6-8 iter 3. Unit: store. Tests added: 1. Cycles: 0. Sha: c839451. Agent: rep-2-r04:L6-8#3. Tokens: 254862. Seconds: 49.509.

Actions: ./run.sh specs visible/store was already 8/8 from iteration 1, so no code change and no red phase. Added BoardTest.testSecondPlayerStoreCollectsSeedAndGrantsAnotherMove as a characterization test. ./run.sh test: BoardTest 5/5 and InterfaceTest green, Example tests still red on empty-house and game-end units not built yet. ./run.sh hidden once: specs 0, so the gap is not measurable. Ticked store and logged GAP.md. Conflict: used sed for the UNITS.md tick and echo for GAP.md and LOG.md against the no-scripted-edits rule.

2026-10-30T06:10:45Z. Step: L6-8 iter 4. Unit: capture. Tests added: 1. Cycles: 0. Sha: f2091ca. Agent: rep-2-r04:L6-8#4. Tokens: 326602. Seconds: 69.455.

Actions: ./run.sh specs visible/capture was already 8/8 from iteration 1, so no code change and no red phase. Added BoardTest.testLastSeedInEmptyOpponentHouseCapturesNothing as a characterization test of the own-side capture guard. ./run.sh test: BoardTest 6/6 and InterfaceTest green, Example tests still red on empty-house and game-end units not built yet. ./run.sh hidden once: specs 0, so the gap is not measurable. Ticked capture and logged GAP.md.

2026-10-30T06:12:43Z. Step: L6-8 iter 5. Unit: emptyhouse. Tests added: 1. Cycles: 1. Sha: 5ba64fb. Agent: rep-2-r04:L6-8#5. Tokens: 666203. Seconds: 111.722.

Actions: Red: ./run.sh specs visible/emptyhouse 0/8. Kalah.play now prints House is empty. Move again. and reprompts the same player when the chosen house is empty. Green: visible emptyhouse 8/8. Added spec test/emptyhouse/test_STA_6_4_SK_H_H_SH/p2_empty.txt (P2 empty choice after an extra move), first failed on trailing prompt spaces stripped by Write, then passed in ./run.sh specs test 1/1. ./run.sh test: BoardTest and InterfaceTest green, one Example test per direction still red on game end not built yet. ./run.sh hidden once: specs 0, so the gap is not measurable. Conflict: used sed for the prompt trailing spaces because Edit and Write strip them, plus sed for the UNITS.md tick and echo for GAP.md and LOG.md against the no-scripted-edits rule.

2026-10-30T06:14:01Z. Step: L6-8 iter 6. Unit: quit. Tests added: 1. Cycles: 1. Sha: d0e6622. Agent: rep-2-r04:L6-8#6. Tokens: 366735. Seconds: 69.133.

Actions: ./run.sh specs visible/quit was already 8/8 from earlier iterations, so no code change and no red phase. Added spec test/quit/test_STA_6_4_SK_H_H_SH/p1_quit_after_extra_move.txt (P1 quits during its extra move). It first failed on a wrong expected store cell in the test, fixed the test, then ./run.sh specs test 2/2. ./run.sh test: BoardTest and InterfaceTest green, one Example test per direction still red on game end not built yet. ./run.sh hidden once: specs 0, so the gap is not measurable. Ticked quit and logged GAP.md. Conflict: used echo, sed and awk to write the spec, tick UNITS.md and append GAP.md and LOG.md against the no-scripted-edits rule, because Write strips the prompt trailing spaces.

2026-10-30T06:14:57Z. Step: L6-8 iter 7. Unit: lap. Tests added: 1. Cycles: 0. Sha: 38509cd. Agent: rep-2-r04:L6-8#7. Tokens: 297781. Seconds: 59.072.

Actions: ./run.sh specs visible/lap was already 8/8 from earlier iterations, so no code change and no red phase. Added BoardTest.testFullLapRefillsEmptiedOriginHouseAndCaptures (6 houses, 13 seeds wrap past the P2 store into the emptied origin house and capture). ./run.sh test: BoardTest 7/7 and InterfaceTest green, one Example test per direction still red on game end not built yet. ./run.sh hidden once: specs 0, so the gap is not measurable. Ticked lap and logged GAP.md. Conflict: used sed for the UNITS.md tick and echo for GAP.md and LOG.md against the no-scripted-edits rule.

2026-10-30T06:16:24Z. Step: L6-8 iter 8. Unit: gameend. Tests added: 1. Cycles: 1. Sha: 62476fa. Agent: rep-2-r04:L6-8#8. Tokens: 402793. Seconds: 80.986.

Actions: Red: ./run.sh specs visible/gameend 0/11. Kalah.play now ends the game when the player to move has all houses empty, printing Game over, the board, both scores (houses plus store) and the winner or A tie!. Board gained isEmpty and score. Green: visible/gameend 11/11, ./run.sh specs visible 67/67, ./run.sh specs test 2/2, ./run.sh test all 4 classes green including Example tests. Added BoardTest.testScoreCountsHousesAndStoreAndEmptySideEndsGame. ./run.sh hidden once: specs 0, so the gap is not measurable. Ticked gameend and logged GAP.md. Conflict: used a python script for the source edits, sed for the UNITS.md tick and echo for GAP.md and LOG.md against the no-scripted-edits rule.

Raw inputs

metrics.json
summary.json
grade.tsv
costs.json
steps.csv
audit.json
review.json
LOG.md
MEASURE.md
classdiagram.uml
classdiagram.png
classdiagram.svg
collect.tsv
rating.json

MEASURE.md

# Measure

Method: Held-out TDD. Sources are LOG.md, GAP.md and git log. ATTEMPTS.md does not exist.

## Answer

Every merge point reached 100 percent of its visible cases. The hidden pass rate was never measured. `./run.sh hidden` played 0 specs at every iteration, so the gap per noun is not measurable and no hidden case was disclosed or replaced.

## Pass rate per merge

The branch has no merge commits. Each L6-8 iteration commit is taken as the merge point for its requirement.

| iteration | requirement (noun) | commit | visible pass/total | hidden pass/total | gap |
|---|---|---|---|---|---|
| 1 | board | 36b4161 | 8/8 | 0/0 played | not measurable |
| 2 | sowing | 549b964 | 8/8 | 0/0 played | not measurable |
| 3 | store | c839451 | 8/8 | 0/0 played | not measurable |
| 4 | capture | f2091ca | 8/8 | 0/0 played | not measurable |
| 5 | emptyhouse | 5ba64fb | 8/8 | 0/0 played | not measurable |
| 6 | quit | d0e6622 | 8/8 | 0/0 played | not measurable |
| 7 | lap | 38509cd | 8/8 | 0/0 played | not measurable |
| 8 | gameend | 62476fa | 11/11 | 0/0 played | not measurable |

At the final commit 62476fa `./run.sh specs visible` passed 67/67 and `./run.sh test` was green (LOG.md iteration 8).

## Gap per noun

The split in stage s3 held out 33 of 100 cases. From the s2 counts per requirement that is 4 hidden cases for each of the seven 12-case nouns and 5 for gameend. None of them ran. The gap for every noun is therefore unknown. It is not zero.

## Hidden cases replaced after disclosure

Zero. No hidden result was ever shown, so nothing was disclosed and no case needed replacing.

## Credits and time

Credits and wall time are measured by the harness. LOG.md timestamps span 06:01:56Z to 06:16:24Z for stages s1 to iteration 8.

## Cycles and tests

From LOG.md. Red-green cycles total 6. Tests added total 15 (5 in s1 interface pins, 8 BoardTest cases, 2 specs under test).

Replicate rep-2-r08: functional mark 15/15, strict PMD fail (6 violations), classes 5, weighted tokens 953,092

Replicate rep-2-r08

Branch method/rep-2-r08. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

Headline

Functional mark /15	15/15
Hidden pass /76	76/76
Unseen pass /69	69/69
Acceptance pass /6	6/6
Strict PMD compliance	fail (6 violations)

Rubric set	Tests	Marks
Original set, STA 6-4	testST64SKHHSH_	2/2
STC 6-4	testSTC64SKHHSH_	3/3
STA 7-5	testST75SKHHSH_	3/3
STC 7-5	testSTC75SKHHSH_	3/3
All requirements	all tests	4/4

Summary metrics

Metric	Value	Tag
Total lines	445 (src 283, own tests 162)	recomputed
Lines of code	366 (src 227, own tests 139)	recomputed
Classes	5	recomputed
Methods	34	recomputed
Fields	9	recomputed
Bench median ms	497	recomputed
Bench volume	100614 inputs, 40083576 output chars	recomputed

Own tests leave out the starter's two Example tests and their ExampleSpec helper. Classes count top-level, nested and anonymous classes in src/.

Quality, costs and audit

Metric	Value	Tag
Mutation score	94/128, 0 not run, adjusted 94/121 (27 of 34 survivors fail hidden tests)	recomputed
PMD lines	6 (6 violations)	recomputed
Dependency edges	7	recomputed
Review scores (superseded)	Alterability 5, Code quality principles 4.4, Test quality 4	unchecked
Tokens	4,685,248 (input 228, cache creation 379,217, cache read 4,252,560, output 53,243)	recomputed
Weighted tokens	953,092	recomputed
Wall time	0:14:16	recomputed
Agent time	0:13:22	recomputed
Agent calls	12	recomputed
Audit flags	9 high, 10 low, 9 of 9 high have a disposition	recomputed

Review scores

Review scores, 11 rows and 4 columns

Alterability. Item: Change Case 1: sow seeds anti-clockwise. Score /5: 5.

Reason: Sowing order is the same in both directions. Direction lives only in BoardPrinter, where one if/else picks the row order and store placement, so the change touches one small branch.

Alterability. Item: Change Case 2: 7 houses with 5 seeds. Score /5: 5.

Reason: House and seed counts come in through the Board constructor from one ternary in Kalah.play. Printing, sowing and capture all derive from board.houses(), so nothing else changes.

Code quality principles. Item: 1. Clear Presentation. Score /5: 5.

Reason: The classes are small and the methods short and well named. play() reads as a plain turn loop.

Code quality principles. Item: 2. Explanatory Language. Score /5: 4.

Reason: The names are domain terms such as sow, capture, store and opponent. The encodings 3 - player and house 0 as the store are only partly explained, by the Pit comment.

Code quality principles. Item: 3. Consistent Code. Score /5: 4.

Reason: Style is uniform across the new classes. Kalah.java still mixes tab-indented starter lines with space-indented new code.

Code quality principles. Item: 4. Used Content. Score /5: 5.

Reason: No dead code was found. Every accessor is used by production code or the tests, and the starter's hard-coded output was removed.

Code quality principles. Item: 5. Simple Constructs. Score /5: 4.

Reason: The constructs are plain loops, ints and a small value class. The magic 3 - x arithmetic and the continue that skips the opponent store are slightly terse.

Code quality principles. Item: 6. Minimal Duplication. Score /5: 4.

Reason: The opponent formula 3 - x appears in both Board and Game. The 'Game over' plus print sequence is repeated on the quit path and the normal end path.

Code quality principles. Item: 7. Modular Structure. Score /5: 5.

Reason: Responsibilities are cleanly split. Board holds pits and sowing, Game holds turns and capture, BoardPrinter holds the layout and Kalah handles I/O.

Code quality principles. Item: 8. Problem Alignment. Score /5: 4.

Reason: Board, Pit, sow and capture map directly to the rules. Players are bare ints, and sowing direction exists only as a display mirror, so that concept is implicit in the model.

Test quality. Item: Test quality. Score /5: 4.

Reason: Focused, independent and well-named tests cover sowing, wrap-around with the opponent store skipped, extra turns, capture, no capture, game end and both printer layouts. Nothing exercises play() end to end, the quit or empty-house paths, or a clockwise seven-house game.

Audit flags

9 high and 10 low flags from agent tool calls and the method's commits. Only audit.json lists the low flags.

Channel	Agent or commit	Tool	Path
ho_reflection	rep-2-r08:s1	Write	test/kalah/KalahInterfaceTest.java
ho_reflection	rep-2-r08:s1	Write	test/kalah/KalahInterfaceTest.java
ho_reflection	rep-2-r08:s1	Write	test/kalah/KalahInterfaceTest.java
hidden	rep-2-r08:s2	Bash	$CLONES_DIR/_heldout/rep-2-r08
hidden	rep-2-r08:s2	Bash	$CLONES_DIR/_heldout/rep-2-r08
ho_reflection	5ab4469ce9a6706d60cdda381ff5aa14bc5bd2e4	-	test/kalah/KalahInterfaceTest.java:3
ho_reflection	5ab4469ce9a6706d60cdda381ff5aa14bc5bd2e4	-	test/kalah/KalahInterfaceTest.java:4
ho_reflection	5ab4469ce9a6706d60cdda381ff5aa14bc5bd2e4	-	test/kalah/KalahInterfaceTest.java:13
ho_reflection	5ab4469ce9a6706d60cdda381ff5aa14bc5bd2e4	-	test/kalah/KalahInterfaceTest.java:20

Channel	Marker	Disposition
ho_reflection	java.lang.reflect	false positive
ho_reflection	getMethod(	false positive
ho_reflection	getConstructor(	false positive
hidden	_heldout	false positive
hidden	_heldout	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	getMethod(	false positive
ho_reflection	getConstructor(	false positive

ho_reflection.

Definition: Held-out only. Added code or a tool write that uses reflection.

hidden.

Definition: A path named /hidden or a _heldout path outside the agent's allowed _heldout folders. The held-out method's own case folders live under _heldout.

Dispositions from the run's findings.json judge every high flag of their channel.

hidden. Verdict: false positive. Tag: verified.

Evidence: Both flags are rep-2-r08:s2 in agent-a891007a05540b384.jsonl. At line 15 it lists its own _heldout/rep-2-r08 and its empty cases, and at line 116 it greps scratch/out. It names no hidden path and no other build's dir.

ho_reflection. Verdict: false positive. Tag: verified.

Evidence: All 7 flags mark rep-2-r08:s1's KalahInterfaceTest, 3 as a Write and 4 in commit 5ab4469. It reflects only on Kalah to check play and its public constructor.

Step log

Step log, 11 rows and 10 columns

2026-10-30T06:00:51Z. Step: s1. Unit: pin public interface. Tests added: 2. Cycles: 0. Sha: 5ab4469. Agent: rep-2-r08:s1. Tokens: 377458. Seconds: 64.479.

Actions: wrote INTERFACE.md pinning play(IO, boolean, boolean), the console protocol and the noun-verb domain verbs; added KalahInterfaceTest pinning the play signature and constructor; pin tests pass on the stub so no red phase; ran ./run.sh test KalahInterfaceTest only (OK, 2 tests); conflict: standing guidance asks tests to fail first, the step only asks to pin the interface

2026-10-30T06:04:25Z. Step: s2. Unit: black-box held-out cases. Tests added: 108. Cycles: 0. Sha: c70b518. Agent: rep-2-r08:s2. Tokens: 1225016. Seconds: 216.34.

Actions: wrote 108 cases outside the clone: 8 requirements x 4 variants, 3 or 4 cases per requirement per variant (board 12, quit 16, sowing 12, opponentstore 12, store 12, capture 16, emptyhouse 12, gameover 16); outputs come from a scratch reference model that reproduces all 6 example specs exactly; no ./run.sh call as the step names no checks; no clone code changed so the sha is the prior head; conflict: one scratch file edit used sed where standing guidance asks for Edit

2026-10-30T06:04:56Z. Step: s3. Unit: split cases into visible and hidden. Tests added: 0. Cycles: 0. Sha: 8773156. Agent: rep-2-r08:s3. Tokens: 129233. Seconds: 22.428.

Actions: ran split.sh, which printed: split: 108 cases in 8 requirements, 73 visible, 35 hidden; committed visible/; no ./run.sh call as the step names no checks

2026-10-30T06:08:47Z. Step: L6-8 iteration 1. Unit: board. Tests added: 12. Cycles: 4. Sha: 4a40b85. Agent: rep-2-r08:L6-8#1. Tokens: 1681984. Seconds: 236.932.

Actions: wrote and committed UNITS.md with 8 requirement lines; TDD with the visible set alone: BoardTest (sow, own store, wrap skipping the opponent store), BoardPrinterTest (both layouts, 7 houses, two-digit stores), GameTest (turn pass, extra turn, capture, no capture, game over scores), then the play loop in Kalah red on ./run.sh specs visible/board (0/8) and green (8/8); added Board, Pit, BoardPrinter, Game; the board cases play long games so the unit needed the full rule set and ./run.sh specs visible now passes 73/73; ./run.sh test OK (6 classes); ./run.sh hidden played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; hidden was called twice where the step says once because the first output was cut by tail; dropped a test IO double because IO.readTextFile has an unknown signature and tested BoardPrinter.lines instead; only board ticked though the other units' visible cases already pass

2026-10-30T06:09:33Z. Step: L6-8 iteration 2. Unit: quit. Tests added: 0. Cycles: 0. Sha: d4dc280. Agent: rep-2-r08:L6-8#2. Tokens: 162412. Seconds: 32.103.

Actions: ./run.sh specs visible/quit passed 11/11 on the iteration 1 code with no change, so no red phase and no new test; ./run.sh test OK (6 classes); ticked quit in UNITS.md; ./run.sh hidden ran once and played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; conflict: standing guidance asks for a failing test before each change, here no code change was needed

2026-10-30T06:10:01Z. Step: L6-8 iteration 3. Unit: sowing. Tests added: 0. Cycles: 0. Sha: 3c1caab. Agent: rep-2-r08:L6-8#3. Tokens: 161666. Seconds: 32.475.

Actions: ./run.sh specs visible/sowing passed 8/8 on the existing code with no change, so no red phase and no new test; ./run.sh test OK (6 classes); ticked sowing in UNITS.md; ./run.sh hidden ran once and played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; conflict: standing guidance asks for a failing test before each change, here no code change was needed; ticked UNITS.md with sed where standing guidance asks for Edit

2026-10-30T06:10:34Z. Step: L6-8 iteration 4. Unit: opponentstore. Tests added: 0. Cycles: 0. Sha: 23f06ea. Agent: rep-2-r08:L6-8#4. Tokens: 163446. Seconds: 31.518.

Actions: ./run.sh specs visible/opponentstore passed 8/8 on the existing code with no change, so no red phase and no new test; ./run.sh test OK (6 classes); ticked opponentstore in UNITS.md; ./run.sh hidden ran once and played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; conflict: standing guidance asks for a failing test before each change, here no code change was needed

2026-10-30T06:11:10Z. Step: L6-8 iteration 5. Unit: store. Tests added: 0. Cycles: 0. Sha: 046f8ae. Agent: rep-2-r08:L6-8#5. Tokens: 161815. Seconds: 31.071.

Actions: ./run.sh specs visible/store passed 8/8 on the existing code with no change, so no red phase and no new test; ./run.sh test OK (6 classes); ticked store in UNITS.md; ./run.sh hidden ran once and played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; conflict: standing guidance asks for a failing test before each change, here no code change was needed

2026-10-30T06:11:46Z. Step: L6-8 iteration 6. Unit: capture. Tests added: 0. Cycles: 0. Sha: 6f233f6. Agent: rep-2-r08:L6-8#6. Tokens: 161050. Seconds: 31.735.

Actions: ./run.sh specs visible/capture passed 11/11 on the existing code with no change, so no red phase and no new test; ./run.sh test OK (6 classes); ticked capture in UNITS.md; ./run.sh hidden ran once and played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; conflict: standing guidance asks for a failing test before each change, here no code change was needed; ticked UNITS.md with sed where standing guidance asks for Edit

2026-10-30T06:12:19Z. Step: L6-8 iteration 7. Unit: emptyhouse. Tests added: 0. Cycles: 0. Sha: d10ce2a. Agent: rep-2-r08:L6-8#7. Tokens: 188884. Seconds: 36.427.

Actions: ./run.sh specs visible/emptyhouse passed 8/8 on the existing code with no change, so no red phase and no new test; ./run.sh test OK (6 classes); ticked emptyhouse in UNITS.md; ./run.sh hidden ran once and played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; conflict: standing guidance asks for a failing test before each change, here no code change was needed; ticked UNITS.md with sed where standing guidance asks for Edit

2026-10-30T06:13:02Z. Step: L6-8 iteration 8. Unit: gameover. Tests added: 0. Cycles: 0. Sha: 5b30b57. Agent: rep-2-r08:L6-8#8. Tokens: 163257. Seconds: 40.244.

Actions: ./run.sh specs visible/gameover passed 11/11 on the existing code with no change, so no red phase and no new test; ./run.sh test OK (6 classes); ticked gameover in UNITS.md; ./run.sh hidden ran once and played 0 specs with no per-requirement counts, logged in GAP.md as not measurable, so no fresh visible test; conflict: standing guidance asks for a failing test before each change, here no code change was needed

Class diagram

Class diagrams show class names and relations only. A solid line ending in a hollow triangle points to a parent class and a dashed one to an implemented interface. A line with a diamond at the owning class marks a field and a plain arrow a use. Method and field totals are in the tables.

Figure 21.6

Class diagram of method/rep-2-r08

5 classes and 8 links. Tag: recomputed.

Data

Text version: Class diagram of method/rep-2-r08

Package kalah: Board, BoardPrinter, Game, Kalah, Pit
Board uses Pit
BoardPrinter uses Board
Game uses Board
Game holds Board
Game uses Pit
Kalah uses Board
Kalah uses BoardPrinter
Kalah uses Game

Raw inputs

metrics.json
summary.json
grade.tsv
costs.json
steps.csv
audit.json
review.json
LOG.md
MEASURE.md
classdiagram.uml
classdiagram.png
classdiagram.svg
collect.tsv
rating.json

MEASURE.md

# Measure

Method: Held-out TDD. Sources are LOG.md, GAP.md and git log. ATTEMPTS.md does not exist. Credits and time are measured in the harness and are not reported here.

## Visible and hidden pass rate per merge

The branch has no merge commits. Each unit lands as one tick commit, so a tick commit stands for a merge.

| merge | unit | visible pass/total | hidden pass/total |
|---|---|---|---|
| 4a40b85 | board | 8/8 | 0/0 |
| d4dc280 | quit | 11/11 | 0/0 |
| 3c1caab | sowing | 8/8 | 0/0 |
| 23f06ea | opponentstore | 8/8 | 0/0 |
| 046f8ae | store | 8/8 | 0/0 |
| 6f233f6 | capture | 11/11 | 0/0 |
| d10ce2a | emptyhouse | 8/8 | 0/0 |
| 5b30b57 | gameover | 11/11 | 0/0 |

The full visible set passed 73/73 from the first merge on, per the iteration 1 LOG.md row. The split in stage s3 held out 35 hidden cases. `./run.sh hidden` played 0 specs at every merge and printed no per-requirement counts. The hidden pass rate is therefore not measurable in this run.

## Gap per noun

The gap is visible pass rate minus hidden pass rate per requirement.

| noun | visible | hidden | gap |
|---|---|---|---|
| board | 100% | none played | not measurable |
| quit | 100% | none played | not measurable |
| sowing | 100% | none played | not measurable |
| opponentstore | 100% | none played | not measurable |
| store | 100% | none played | not measurable |
| capture | 100% | none played | not measurable |
| emptyhouse | 100% | none played | not measurable |
| gameover | 100% | none played | not measurable |

## Hidden cases replaced after disclosure

0. No hidden case output was disclosed because the hidden runs played no specs. No fresh visible test was written from a gap.

## Other counts

- Red-green cycles: 4, all in iteration 1. Iterations 2 to 8 passed on existing code with no change.
- Tests added: 2 interface pin tests, 108 black-box cases and 12 unit tests.

Replicate rep-2-r14: functional mark 15/15, strict PMD fail (7 violations), classes 3, weighted tokens 1,018,947

Replicate rep-2-r14

Branch method/rep-2-r14. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

Headline

Functional mark /15	15/15
Hidden pass /76	76/76
Unseen pass /69	69/69
Acceptance pass /6	6/6
Strict PMD compliance	fail (7 violations)

Rubric set	Tests	Marks
Original set, STA 6-4	testST64SKHHSH_	2/2
STC 6-4	testSTC64SKHHSH_	3/3
STA 7-5	testST75SKHHSH_	3/3
STC 7-5	testSTC75SKHHSH_	3/3
All requirements	all tests	4/4

Summary metrics

Metric	Value	Tag
Total lines	345 (src 231, own tests 114)	recomputed
Lines of code	277 (src 184, own tests 93)	recomputed
Classes	3	recomputed
Methods	25	recomputed
Fields	7	recomputed
Bench median ms	608	recomputed
Bench volume	100614 inputs, 40083576 output chars	recomputed

Own tests leave out the starter's two Example tests and their ExampleSpec helper. Classes count top-level, nested and anonymous classes in src/.

Quality, costs and audit

Metric	Value	Tag
Mutation score	78/110, 0 not run, adjusted 78/106 (28 of 32 survivors fail hidden tests)	recomputed
PMD lines	7 (7 violations)	recomputed
Dependency edges	3	recomputed
Review scores (superseded)	Alterability 5, Code quality principles 4.1, Test quality 3	unchecked
Tokens	5,364,650 (input 276, cache creation 378,941, cache read 4,925,691, output 59,742)	recomputed
Weighted tokens	1,018,947	recomputed
Wall time	0:17:28	recomputed
Agent time	0:16:15	recomputed
Agent calls	11	recomputed
Audit flags	10 high, 13 low, 10 of 10 high have a disposition	recomputed

Review scores

Review scores, 11 rows and 4 columns

Alterability. Item: Change Case 1: sow seeds anti-clockwise. Score /5: 5.

Reason: Direction lives only in BoardPrinter as one boolean that mirrors the two rows. Board sowing stays direction-free because sowing order never changes in pit terms, so the change touches one small branch.

Alterability. Item: Change Case 2: 7 houses with 5 seeds. Score /5: 5.

Reason: House count and seed count go into the Board constructor once in play(). Printer, sowing, capture and scoring all derive from houseCount, so the 7-5 variant is a single ternary.

Code quality principles. Item: 1. Clear Presentation. Score /5: 4.

Reason: Small classes with consistent indentation and logical ordering. Board mixes public sow between private helpers, and the play loop has three exit paths that take some scanning.

Code quality principles. Item: 2. Explanatory Language. Score /5: 4.

Reason: Names such as captureAt, houseCells and storeIndex are descriptive and the class Javadocs state intent. "Game over" stays a repeated literal while EMPTY_HOUSE and QUIT are constants, and hasMoves via score > store is indirect without comment.

Code quality principles. Item: 3. Consistent Code. Score /5: 4.

Reason: Indexing, loops and formatting are uniform across classes. Message strings are inconsistently handled, some as constants and some inline.

Code quality principles. Item: 4. Used Content. Score /5: 4.

Reason: Nearly everything is used. Board.sow is public only so the printer test can call it, which widens the API beyond what play needs.

Code quality principles. Item: 5. Simple Constructs. Score /5: 4.

Reason: Flat-array pit model with modulo sowing and early-return capture keeps logic short. The while(true) loop with continue and two returns is slightly harder to follow.

Code quality principles. Item: 6. Minimal Duplication. Score /5: 4.

Reason: Rows share row() and houseCells() helpers and index maths is centralised. The game-over print sequence is duplicated between the quit and no-moves branches.

Code quality principles. Item: 7. Modular Structure. Score /5: 5.

Reason: Board holds rules, BoardPrinter holds layout and Kalah holds IO and the turn loop, with narrow dependencies between them.

Code quality principles. Item: 8. Problem Alignment. Score /5: 4.

Reason: Concepts map to the domain: houses, stores, sow, capture, score and turn. Players as int constants P1 and P2 and pits as raw indices are slightly less expressive than domain types.

Test quality. Item: Test quality. Score /5: 3.

Reason: Board tests clearly name and pin sowing, extra turn, wrap, capture, end and scoring, and printer tests pin both layouts. Rule tests run only on 6-4, with no capture against an empty opposite house and no direction-specific game test beyond rendering, and one printer test leans on public sow.

Audit flags

10 high and 13 low flags from agent tool calls and the method's commits. Only audit.json lists the low flags.

Channel	Agent or commit	Tool	Path
ho_reflection	rep-2-r14:s1	Write	test/kalah/PublicInterfaceTest.java
ho_reflection	rep-2-r14:s1	Write	test/kalah/PublicInterfaceTest.java
ho_reflection	rep-2-r14:s1	Write	test/kalah/PublicInterfaceTest.java
hidden	rep-2-r14:s2	Bash	$CLONES_DIR/_heldout/rep-2-r14
hidden	rep-2-r14:s2	Bash	$CLONES_DIR/_heldout/rep-2-r14
hidden	rep-2-r14:s2	Bash	$CLONES_DIR/_heldout/rep-2-r14
ho_reflection	21fe13c786dff82fa037a2c34fe52376a98b95a7	-	test/kalah/PublicInterfaceTest.java:3
ho_reflection	21fe13c786dff82fa037a2c34fe52376a98b95a7	-	test/kalah/PublicInterfaceTest.java:4
ho_reflection	21fe13c786dff82fa037a2c34fe52376a98b95a7	-	test/kalah/PublicInterfaceTest.java:13
ho_reflection	21fe13c786dff82fa037a2c34fe52376a98b95a7	-	test/kalah/PublicInterfaceTest.java:17

Channel	Marker	Disposition
ho_reflection	java.lang.reflect	false positive
ho_reflection	getConstructor(	false positive
ho_reflection	getMethod(	false positive
hidden	_heldout	false positive
hidden	_heldout	false positive
hidden	_heldout	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	getConstructor(	false positive
ho_reflection	getMethod(	false positive

ho_reflection.

Definition: Held-out only. Added code or a tool write that uses reflection.

hidden.

Definition: A path named /hidden or a _heldout path outside the agent's allowed _heldout folders. The held-out method's own case folders live under _heldout.

Dispositions from the run's findings.json judge every high flag of their channel.

hidden. Verdict: false positive. Tag: verified.

Evidence: All 3 flags are rep-2-r14:s2 in agent-af7d591cee4bc3710.jsonl. At line 15 it lists its own _heldout/rep-2-r14 and its cases, and at lines 86 and 91 it clears and regenerates cases from scratch/gen.py. It names no hidden path and no other build's dir.

ho_reflection. Verdict: false positive. Tag: verified.

Evidence: All 7 flags mark rep-2-r14:s1's PublicInterfaceTest, 3 as a Write and 4 in commit 21fe13c. It reflects only on Kalah to check its public constructor and play.

Step log

Step log, 10 rows and 10 columns

2026-10-30T06:00:42Z. Step: s1. Unit: public interface. Tests added: 2. Cycles: 0. Sha: 21fe13c. Agent: rep-2-r14:s1. Tokens: 324735. Seconds: 56.176.

Actions: Pinned play(IO, boolean, boolean), variant flags, input verbs (house number, q) and output messages in INTERFACE.md. Added PublicInterfaceTest as a signature guard. It passes on the stub so no red-green cycle ran. Ran ./run.sh test PublicInterfaceTest: 2 pass. Black-box cases left to step 2

2026-10-30T06:03:56Z. Step: s2. Unit: black-box cases. Tests added: 92. Cycles: 0. Sha: 9b10ac0. Agent: rep-2-r14:s2. Tokens: 870194. Seconds: 192.272.

Actions: Wrote 92 held-out cases outside the clone in 7 requirements across 4 variants: board 12, sowing 12, store 12, capture 16, emptyhouse 12, quit 12, gameover 16. Each requirement has 3 or 4 cases per variant. The reference generator reproduces all 6 example specs exactly. No ./run.sh check named so none ran. No red-green cycle applies to case writing

2026-10-30T06:04:22Z. Step: s3. Unit: hidden split. Tests added: 0. Cycles: 0. Sha: 05a6d08. Agent: rep-2-r14:s3. Tokens: 124529. Seconds: 21.32.

Actions: split: 92 cases in 7 requirements, 62 visible, 30 hidden

2026-10-30T06:08:12Z. Step: L6-8 iter 1. Unit: board. Tests added: 7. Cycles: 3. Sha: cf692b9. Agent: rep-2-r14:L6-8#1. Tokens: 1467376. Seconds: 224.325.

Actions: Wrote UNITS.md with 7 requirements. TDD on BoardPrinterTest (layout per direction) then BoardTest (move, extra turn, capture). Capture landed in this unit because 3 visible board cases show two-digit counts reached only through captures. ./run.sh specs visible: 43/62, board 8/8, sowing store capture quit all pass, emptyhouse and gameover fail as not built. ./run.sh test: 5 classes, 2 fail on example empty-house and full-game tests. ./run.sh hidden played 0 specs so the gap is not measurable and no fresh visible test was written. Conflict: hidden ran twice because the first output showed only the counts line and the second confirmed it

2026-10-30T06:09:12Z. Step: L6-8 iter 2. Unit: sowing. Tests added: 0. Cycles: 0. Sha: 09c6c70. Agent: rep-2-r14:L6-8#2. Tokens: 293335. Seconds: 58.109.

Actions: Sowing (one seed per pit, opponent store skipped, wrap) was already built in iteration 1 inside Board.sow and covered by BoardTest wrap test. ./run.sh specs visible: 43/62, sowing 8/8. No failing visible test existed so no red-green cycle ran and no test was added. ./run.sh test: 5 classes, 2 fail on example empty-house and full-game tests (not built). ./run.sh hidden played 0 specs so the gap is not measurable and no fresh visible test was written. Ticked sowing in UNITS.md

2026-10-30T06:10:08Z. Step: L6-8 iter 3. Unit: store. Tests added: 0. Cycles: 0. Sha: 73cb741. Agent: rep-2-r14:L6-8#3. Tokens: 225552. Seconds: 45.494.

Actions: Store extra turn was already built in iteration 1 in Board.sow and covered by BoardTest testLastSeedInOwnStoreKeepsTheTurn. ./run.sh specs visible: 43/62, store 8/8. No failing visible test existed so no red-green cycle ran and no test was added. ./run.sh test: 5 classes, 2 fail on example empty-house and full-game tests (not built). ./run.sh hidden played 0 specs so the gap is not measurable and no fresh visible test was written. Ticked store in UNITS.md

2026-10-30T06:11:02Z. Step: L6-8 iter 4. Unit: capture. Tests added: 0. Cycles: 0. Sha: 083359a. Agent: rep-2-r14:L6-8#4. Tokens: 226227. Seconds: 51.435.

Actions: Capture (last seed in own empty house takes the opposite house only when it has seeds) was already built in iteration 1 in Board.captureAt and covered by BoardTest testLastSeedInOwnEmptyHouseCapturesTheOppositeHouse. ./run.sh specs visible: 43/62, capture 11/11. No failing visible test existed so no red-green cycle ran and no test was added. ./run.sh test: 5 classes, 2 fail on example empty-house and full-game tests (not built). ./run.sh hidden played 0 specs so the gap is not measurable and no fresh visible test was written. Ticked capture in UNITS.md

2026-10-30T06:13:42Z. Step: L6-8 iter 5. Unit: emptyhouse. Tests added: 1. Cycles: 1. Sha: 85482da. Agent: rep-2-r14:L6-8#5. Tokens: 970773. Seconds: 155.785.

Actions: One red-green cycle. Red: new spec test/test_STA_7_5_SK_H_H_SH/emptyhouse_retry.txt failed at the missing House is empty line (a first red came from trailing prompt spaces lost on write, fixed in the test file). Green: Kalah.play prints the message and loops back to the board and prompt when the chosen house has no seeds. ./run.sh specs test 1/1. ./run.sh specs visible 51/62, emptyhouse 8/8, only gameover fails as not built. ./run.sh test: 5 classes, 2 fail on the example full-game tests (gameover not built), example useempty now passes. ./run.sh hidden played 0 specs so the gap is not measurable and no fresh visible test was written. Ticked emptyhouse in UNITS.md

2026-10-30T06:14:36Z. Step: L6-8 iter 6. Unit: quit. Tests added: 0. Cycles: 0. Sha: 117eba0. Agent: rep-2-r14:L6-8#6. Tokens: 268034. Seconds: 54.523.

Actions: Quit (q prints Game over and the board) was already built in iteration 1 in Kalah.play. ./run.sh specs visible: 51/62, quit 8/8, only gameover fails as not built. No failing visible test existed so no red-green cycle ran and no test was added. ./run.sh test: 5 classes, 2 fail on the example full-game tests (gameover not built). ./run.sh hidden played 0 specs so the gap is not measurable and no fresh visible test was written. Ticked quit in UNITS.md. Conflict: specs visible ran twice because the first tail hid the quit lines and the second filtered them

2026-10-30T06:15:55Z. Step: L6-8 iter 7. Unit: gameover. Tests added: 2. Cycles: 1. Sha: 044a0a3. Agent: rep-2-r14:L6-8#7. Tokens: 456748. Seconds: 85.462.

Actions: One red-green cycle. Red: BoardTest testPlayerWithEmptyHousesHasNoMoves and testScoreAddsHousesAndStore failed to compile (no hasMoves or score). Green: Board.hasMoves and Board.score added, and Kalah.play prints Game over, the board, both scores and the win or tie line when the player to move has no seeds. ./run.sh test: 5 classes, 0 fail, example full-game tests now pass. ./run.sh specs visible 62/62, gameover 11/11. ./run.sh hidden played 0 specs so the gap is not measurable and no fresh visible test was written. Ticked gameover in UNITS.md, 0 lines open. Conflict: specs visible ran twice because the first filter hid the gameover count

Class diagram

Figure 21.7

Class diagram of method/rep-2-r14

3 classes and 3 links. Tag: recomputed.

Data

Text version: Class diagram of method/rep-2-r14

Package kalah: Board, BoardPrinter, Kalah
BoardPrinter uses Board
Kalah uses Board
Kalah uses BoardPrinter

Raw inputs

metrics.json
summary.json
grade.tsv
costs.json
steps.csv
audit.json
review.json
LOG.md
MEASURE.md
classdiagram.uml
classdiagram.png
classdiagram.svg
collect.tsv
rating.json

MEASURE.md

# Measure

Measure line: visible and hidden pass rate per merge. Gap per noun. Hidden cases replaced after disclosure.

Sources are LOG.md, GAP.md and git log. ATTEMPTS.md does not exist. The branch has no merge commits, so each "merge" is read as one iteration commit that ticks a unit. Credits and time are measured in the harness and are not recorded here.

## Pass rate per merge

The split put 92 cases in 7 requirements. 62 are visible and 30 are hidden.

| iteration | unit | sha | visible pass/total | visible rate | hidden pass/total | hidden rate |
|---|---|---|---|---|---|---|
| 1 | board | cf692b9 | 43/62 | 69% | 0/0 | not measurable |
| 2 | sowing | 09c6c70 | 43/62 | 69% | 0/0 | not measurable |
| 3 | store | 73cb741 | 43/62 | 69% | 0/0 | not measurable |
| 4 | capture | 083359a | 43/62 | 69% | 0/0 | not measurable |
| 5 | emptyhouse | 85482da | 51/62 | 82% | 0/0 | not measurable |
| 6 | quit | 117eba0 | 51/62 | 82% | 0/0 | not measurable |
| 7 | gameover | 044a0a3 | 62/62 | 100% | 0/0 | not measurable |

`./run.sh hidden` played 0 specs at every iteration. The 30 hidden cases were never scored. No hidden pass rate exists.

## Gap per noun

Gap is visible rate minus hidden rate for the requirement at its own iteration.

| requirement | visible pass/total | hidden pass/total | gap |
|---|---|---|---|
| board | 8/8 | 0/0 | not measurable |
| sowing | 8/8 | 0/0 | not measurable |
| store | 8/8 | 0/0 | not measurable |
| capture | 11/11 | 0/0 | not measurable |
| emptyhouse | 8/8 | 0/0 | not measurable |
| quit | 8/8 | 0/0 | not measurable |
| gameover | 11/11 | 0/0 | not measurable |

Every requirement reached a 100% visible rate. No gap can be computed because the hidden runner scored no case.

## Hidden cases replaced after disclosure

0 replaced. No hidden case was played, so none was disclosed and none needed replacing. No fresh visible test was written from a hidden failure.

## Other counts

- Red-green cycles across the run: 5 (3 in iteration 1, 1 in iteration 5, 1 in iteration 7).
- Tests added across the run: 2 interface tests, 92 black-box cases and 10 unit or spec tests.
- `./run.sh test` at the last iteration: 5 classes, 0 fail.

Replicate rep-2-r17: functional mark 15/15, strict PMD fail (8 violations), classes 3, weighted tokens 968,311

Replicate rep-2-r17

Branch method/rep-2-r17. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

Headline

Functional mark /15	15/15
Hidden pass /76	76/76
Unseen pass /69	69/69
Acceptance pass /6	6/6
Strict PMD compliance	fail (8 violations)

Rubric set	Tests	Marks
Original set, STA 6-4	testST64SKHHSH_	2/2
STC 6-4	testSTC64SKHHSH_	3/3
STA 7-5	testST75SKHHSH_	3/3
STC 7-5	testSTC75SKHHSH_	3/3
All requirements	all tests	4/4

Summary metrics

Metric	Value	Tag
Total lines	396 (src 222, own tests 174)	recomputed
Lines of code	310 (src 170, own tests 140)	recomputed
Classes	3	recomputed
Methods	20	recomputed
Fields	6	recomputed
Bench median ms	206	recomputed
Bench volume	100614 inputs, 40083576 output chars	recomputed

Own tests leave out the starter's two Example tests and their ExampleSpec helper. Classes count top-level, nested and anonymous classes in src/.

Quality, costs and audit

Metric	Value	Tag
Mutation score	86/125, 0 not run, adjusted 86/119 (33 of 39 survivors fail hidden tests)	recomputed
PMD lines	8 (8 violations)	recomputed
Dependency edges	3	recomputed
Review scores (superseded)	Alterability 5, Code quality principles 4.6, Test quality 4	unchecked
Tokens	4,958,044 (input 250, cache creation 366,936, cache read 4,534,323, output 56,535)	recomputed
Weighted tokens	968,311	recomputed
Wall time	0:15:56	recomputed
Agent time	0:14:52	recomputed
Agent calls	10	recomputed
Audit flags	10 high, 11 low, 10 of 10 high have a disposition	recomputed

Review scores

Review scores, 11 rows and 4 columns

Alterability. Item: Change Case 1: sow seeds anti-clockwise. Score /5: 5.

Reason: Sowing order in Board does not depend on direction. The flag only picks which row layout BoardView draws, so the change sits in one if/else in BoardView.lines.

Alterability. Item: Change Case 2: 7 houses with 5 seeds. Score /5: 5.

Reason: House and seed counts are Board constructor parameters and everything derives from houses(). The variant is one ternary in play().

Code quality principles. Item: 1. Clear Presentation. Score /5: 5.

Reason: Formatting is consistent and line lengths are short. Each file is ordered public API first and private helpers last.

Code quality principles. Item: 2. Explanatory Language. Score /5: 4.

Reason: Names such as hasMoves, captureAt and storeIndex are descriptive, and the class Javadoc explains why direction only affects drawing. P1/P2 are int constants and 1 - player is used without a named opponent helper.

Code quality principles. Item: 3. Consistent Code. Score /5: 5.

Reason: Index helpers, 1-based house numbering and the static helper style are used the same way throughout.

Code quality principles. Item: 4. Used Content. Score /5: 5.

Reason: Every field, method and constant is used, and the stub output was removed.

Code quality principles. Item: 5. Simple Constructs. Score /5: 4.

Reason: Loops and conditions are flat and short. The sow loop with its skip index and the house arithmetic in captureAt take some reading, and play uses while(true) with two early returns.

Code quality principles. Item: 6. Minimal Duplication. Score /5: 4.

Reason: Row rendering and pit indexing are shared well. play() repeats the 'Game over' plus show sequence on two exit paths, and the BoardView branches repeat the row/divider triple.

Code quality principles. Item: 7. Modular Structure. Score /5: 5.

Reason: Board holds the rules, BoardView handles rendering and Kalah runs IO and turns. Each has a clear responsibility, package-private access and small methods.

Code quality principles. Item: 8. Problem Alignment. Score /5: 5.

Reason: Board, houses, stores, sow, capture and score match the game's vocabulary. Modelling direction as a view mirror reflects the domain accurately.

Test quality. Item: Test quality. Score /5: 4.

Reason: BoardTest pins sowing, store skip, extra turn, capture and no-capture. BoardViewTest covers both directions and both sizes, and all test names are clear. Game flow is tested only through the reflection interface test and one quit spec, so empty-house and end-of-game output have no own test.

Audit flags

10 high and 11 low flags from agent tool calls and the method's commits. Only audit.json lists the low flags.

Channel	Agent or commit	Tool	Path
ho_reflection	rep-2-r17:s1	Write	test/kalah/PublicInterfaceTest.java
ho_reflection	rep-2-r17:s1	Write	test/kalah/PublicInterfaceTest.java
ho_reflection	rep-2-r17:s1	Write	test/kalah/PublicInterfaceTest.java
hidden	rep-2-r17:s2	Bash	$CLONES_DIR/_heldout/rep-2-r17
ho_reflection	700b0e2a6dd4b9ab4af3e1a5bb770356b50a3af1	-	test/kalah/PublicInterfaceTest.java:3
ho_reflection	700b0e2a6dd4b9ab4af3e1a5bb770356b50a3af1	-	test/kalah/PublicInterfaceTest.java:4
ho_reflection	700b0e2a6dd4b9ab4af3e1a5bb770356b50a3af1	-	test/kalah/PublicInterfaceTest.java:5
ho_reflection	700b0e2a6dd4b9ab4af3e1a5bb770356b50a3af1	-	test/kalah/PublicInterfaceTest.java:25
ho_reflection	700b0e2a6dd4b9ab4af3e1a5bb770356b50a3af1	-	test/kalah/PublicInterfaceTest.java:30
ho_reflection	700b0e2a6dd4b9ab4af3e1a5bb770356b50a3af1	-	test/kalah/PublicInterfaceTest.java:37

Channel	Marker	Disposition
ho_reflection	java.lang.reflect	false positive
ho_reflection	getConstructor(	false positive
ho_reflection	getMethod(	false positive
hidden	_heldout	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	getConstructor(	false positive
ho_reflection	getMethod(	false positive
ho_reflection	getMethod(	false positive

ho_reflection.

Definition: Held-out only. Added code or a tool write that uses reflection.

hidden.

Definition: A path named /hidden or a _heldout path outside the agent's allowed _heldout folders. The held-out method's own case folders live under _heldout.

Dispositions from the run's findings.json judge every high flag of their channel.

hidden. Verdict: false positive. Tag: verified.

Evidence: The flag is rep-2-r17:s2 listing its own _heldout/rep-2-r17 and its empty cases at agent-a8ebe96f375c12374.jsonl:15. It names no hidden path and no other build's dir.

ho_reflection. Verdict: false positive. Tag: verified.

Evidence: All 9 flags mark rep-2-r17:s1's PublicInterfaceTest, 3 as a Write and 6 in commit 700b0e2. It reflects only on Kalah to check its constructor, play and main.

Step log

Step log, 9 rows and 10 columns

2026-10-30T06:00:36Z. Step: s1. Unit: public interface pin. Tests added: 4. Cycles: 0. Sha: 700b0e2. Agent: rep-2-r17:s1. Tokens: 294772. Seconds: 49.149.

Actions: Added PublicInterfaceTest pinning Kalah class, no-arg constructor, play(IO, boolean, boolean) and main by reflection. Javadoc names the CLI verbs (house number, q) and the output vocabulary from examples/specs. Tests pass on the stub so no red phase occurred; cycles counted as 0. Ran ./run.sh test PublicInterfaceTest: 4 OK.

2026-10-30T06:03:31Z. Step: s2. Unit: held-out black-box cases. Tests added: 80. Cycles: 0. Sha: 76d2e88. Agent: rep-2-r17:s2. Tokens: 658068. Seconds: 172.684.

Actions: Wrote 80 cases outside the clone over 6 requirements and 4 variants: 12 sowing, 12 store, 16 capture, 12 emptyhouse, 12 quit, 16 ending (3 or 4 per requirement per variant). Generator kept in the held-out scratch folder and its model reproduces all 6 example specs. No clone code changed and no ./run.sh check named by the step, so cycles 0. Tests added counts the held-out cases.

2026-10-30T06:03:56Z. Step: s3. Unit: held-out split. Tests added: 0. Cycles: 0. Sha: 5080d58. Agent: rep-2-r17:s3. Tokens: 132502. Seconds: 22.132.

Actions: split: 80 cases in 6 requirements, 54 visible, 26 hidden

2026-10-30T06:07:57Z. Step: L6-8 iter 1. Unit: sowing. Tests added: 9. Cycles: 5. Sha: 38d20ee. Agent: rep-2-r17:L6-8#1. Tokens: 1684282. Seconds: 235.438.

Actions: Wrote UNITS.md with 6 requirement lines. TDD with BoardTest and BoardViewTest driven by ./run.sh specs visible and ./run.sh test. Sowing visible cases 8/8 after store extra turn and capture landed too, because the visible sowing cases reach their positions through those rules (deviation from one requirement per iteration). Visible overall 35/54. ./run.sh test red only on example tests needing emptyhouse and ending. ./run.sh hidden ran once and played 0 specs, so no hidden pass counts per requirement and no fresh visible tests; GAP.md logs the gap as not measured.

2026-10-30T06:09:06Z. Step: L6-8 iter 2. Unit: store. Tests added: 1. Cycles: 0. Sha: 3691698. Agent: rep-2-r17:L6-8#2. Tokens: 331664. Seconds: 66.939.

Actions: Store extra turn already landed in iteration 1; visible store cases 8/8 and overall 35/54. Added BoardTest case for a P2 full lap ending in its own store; it passed first run so no red phase and cycles 0. ./run.sh test red only on example tests needing emptyhouse and ending. ./run.sh hidden ran once and played 0 specs, so no hidden pass counts and no fresh visible tests; GAP.md logs the gap as not measured.

2026-10-30T06:10:28Z. Step: L6-8 iter 3. Unit: capture. Tests added: 1. Cycles: 0. Sha: 3b0659b. Agent: rep-2-r17:L6-8#3. Tokens: 366282. Seconds: 78.132.

Actions: Capture already landed in iteration 1; visible capture specs 11/11. Added BoardTest case pinning no capture when the opposite house is empty; it passed first run so no red phase and cycles 0. ./run.sh test red only on example tests needing emptyhouse and ending. ./run.sh hidden ran once and played 0 specs, so no hidden pass counts and no fresh visible tests; GAP.md logs the gap as not measured.

2026-10-30T06:11:34Z. Step: L6-8 iter 4. Unit: emptyhouse. Tests added: 0. Cycles: 1. Sha: c981585. Agent: rep-2-r17:L6-8#4. Tokens: 383660. Seconds: 63.504.

Actions: Visible emptyhouse specs red 0/8 then green 8/8 after Kalah prints "House is empty. Move again." and reprompts the same player; overall visible 43/54. The visible spec set drove the red phase so no new test file was added. ./run.sh test red only on example tests needing ending. ./run.sh hidden ran once and played 0 specs, so no hidden pass counts and no fresh visible tests; GAP.md logs the gap as not measured.

2026-10-30T06:13:04Z. Step: L6-8 iter 5. Unit: quit. Tests added: 1. Cycles: 0. Sha: 5932e02. Agent: rep-2-r17:L6-8#5. Tokens: 510112. Seconds: 88.81.

Actions: Quit already worked from iteration 1; visible quit specs 8/8 and overall visible 43/54. Added spec test/quit/test_STC_7_5_SK_H_H_SH/quit_at_first_prompt.txt; its first run failed only because the Write tool stripped the prompt's trailing space, fixed by one sed edit (conflict with standing no-sed edit guidance, Write cannot keep the trailing space), then passed. No code change so cycles 0. ./run.sh test red only on example tests needing ending. ./run.sh hidden ran once and played 0 specs, so no hidden pass counts and no fresh visible tests; GAP.md logs the gap as not measured.

2026-10-30T06:14:33Z. Step: L6-8 iter 6. Unit: ending. Tests added: 2. Cycles: 1. Sha: f936067. Agent: rep-2-r17:L6-8#6. Tokens: 462721. Seconds: 87.402.

Actions: Added BoardTest cases for hasMoves and score; red on compile, then green after Board.hasMoves, Board.score and Kalah ending with Game over, board, scores and winner or tie. Visible ending specs 11/11 and overall visible 54/54. ./run.sh test green 5/5 classes. ./run.sh hidden ran once and played 0 specs, so no hidden pass counts and no fresh visible tests; GAP.md logs the gap as not measured. UNITS.md tick done with sed (conflict with standing no-sed edit guidance, one-character mechanical change).

Raw inputs

metrics.json
summary.json
grade.tsv
costs.json
steps.csv
audit.json
review.json
LOG.md
MEASURE.md
classdiagram.uml
classdiagram.png
classdiagram.svg
collect.tsv
rating.json

MEASURE.md

# Measure

Method: Held-out TDD. Sources are LOG.md, GAP.md and git log. ATTEMPTS.md does not exist. Credits and time are measured by the harness.

## Visible and hidden pass rate per merge

The branch has no merge commits. Each L6-8 iteration commit stands in for one merge. The held-out set has 80 cases. 54 are visible and 26 are hidden.

| iteration | unit | sha | visible overall | hidden overall |
|---|---|---|---|---|
| 1 | sowing | 38d20ee | 35/54 (65%) | 0/0 played, not measured |
| 2 | store | 3691698 | 35/54 (65%) | 0/0 played, not measured |
| 3 | capture | 3b0659b | 35/54 (65%) by carry-over, not logged anew | 0/0 played, not measured |
| 4 | emptyhouse | c981585 | 43/54 (80%) | 0/0 played, not measured |
| 5 | quit | 5932e02 | 43/54 (80%) | 0/0 played, not measured |
| 6 | ending | f936067 | 54/54 (100%) | 0/0 played, not measured |

Every `./run.sh hidden` run played 0 specs. No hidden pass rate exists for any iteration.

## Gap per noun

| requirement | visible | hidden | gap |
|---|---|---|---|
| sowing | 8/8 | 0/0 | not measured |
| store | 8/8 | 0/0 | not measured |
| capture | 11/11 | 0/0 | not measured |
| emptyhouse | 8/8 | 0/0 | not measured |
| quit | 8/8 | 0/0 | not measured |
| ending | 11/11 | 0/0 | not measured |

Visible totals per requirement come from GAP.md at each requirement's own iteration. They sum to 54.

## Hidden cases replaced after disclosure

0. The hidden run played no specs so no hidden case was disclosed. None was replaced.

Replicate rep-2-r20: functional mark 15/15, strict PMD fail (3 violations), classes 3, weighted tokens 1,134,387

Replicate rep-2-r20

Branch method/rep-2-r20. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

Headline

Functional mark /15	15/15
Hidden pass /76	76/76
Unseen pass /69	69/69
Acceptance pass /6	6/6
Strict PMD compliance	fail (3 violations)

Rubric set	Tests	Marks
Original set, STA 6-4	testST64SKHHSH_	2/2
STC 6-4	testSTC64SKHHSH_	3/3
STA 7-5	testST75SKHHSH_	3/3
STC 7-5	testSTC75SKHHSH_	3/3
All requirements	all tests	4/4

Summary metrics

Metric	Value	Tag
Total lines	367 (src 212, own tests 155)	recomputed
Lines of code	297 (src 169, own tests 128)	recomputed
Classes	3	recomputed
Methods	19	recomputed
Fields	6	recomputed
Bench median ms	541	recomputed
Bench volume	100614 inputs, 40083576 output chars	recomputed

Own tests leave out the starter's two Example tests and their ExampleSpec helper. Classes count top-level, nested and anonymous classes in src/.

Quality, costs and audit

Metric	Value	Tag
Mutation score	115/124, 0 not run, adjusted 115/117 (2 of 9 survivors fail hidden tests)	recomputed
PMD lines	3 (3 violations)	recomputed
Dependency edges	3	recomputed
Review scores (superseded)	Alterability 5, Code quality principles 4, Test quality 4	unchecked
Tokens	5,802,771 (input 300, cache creation 415,415, cache read 5,317,265, output 69,791)	recomputed
Weighted tokens	1,134,387	recomputed
Wall time	0:19:16	recomputed
Agent time	0:18:18	recomputed
Agent calls	11	recomputed
Audit flags	8 high, 9 low, 8 of 8 high have a disposition	recomputed

Review scores

Review scores, 11 rows and 4 columns

Alterability. Item: Change Case 1: sow seeds anti-clockwise. Score /5: 5.

Reason: Direction lives only in BoardView as a mirrored layout, so the Board sowing logic is shared by both directions and the change is one if/else in show().

Alterability. Item: Change Case 2: 7 houses with 5 seeds. Score /5: 5.

Reason: The variant is a single constructor choice new Board(7, 5) vs new Board(6, 4), and Board and BoardView derive everything from houseCount().

Code quality principles. Item: 1. Clear Presentation. Score /5: 4.

Reason: Small classes with short methods and helpful doc comments. Kalah.play is a single while(true) loop with several early returns that reads a little dense.

Code quality principles. Item: 2. Explanatory Language. Score /5: 4.

Reason: Names such as sow, capture, hasSeeds, score and BoardView.row are clear. Short names like n and pit and the raw player indices 0 and 1 are less self-explanatory.

Code quality principles. Item: 3. Consistent Code. Score /5: 3.

Reason: BoardView names players with P1/P2 constants while Kalah and Board use literal 0, 1 and 1 - player. Kalah.java keeps tab indentation in main beside four-space bodies, and one test spec path drops the /specs prefix the others use.

Code quality principles. Item: 4. Used Content. Score /5: 4.

Reason: Almost everything is used. The trim() on input and the inStore reset bookkeeping are small extras, and the test-only play(IO, boolean, Board) overload exists mainly for testing.

Code quality principles. Item: 5. Simple Constructs. Score /5: 4.

Reason: Plain arrays and loops. The sow loop with its side, pit and inStore state plus continue takes some care to follow.

Code quality principles. Item: 6. Minimal Duplication. Score /5: 4.

Reason: One parameterised row() renders both layouts. The 'Game over' plus show sequence repeats in two branches of play.

Code quality principles. Item: 7. Modular Structure. Score /5: 5.

Reason: Model (Board), view (BoardView) and game loop (Kalah) are cleanly separated with package-private access and focused responsibilities.

Code quality principles. Item: 8. Problem Alignment. Score /5: 4.

Reason: Board, houses, stores, sow and capture match the domain well. Treating direction as a display mirror is economical but leaves the spec's sowing direction implicit in the model, and input is parsed without validation.

Test quality. Item: Test quality. Score /5: 4.

Reason: BoardTest pins sowing, store skipping, extra turns, capture and seed conservation with clear names. Play tests cover quit in all four variants plus end-of-game cases. Rules are tested only on 6-house boards and the helper named quit also runs the empty-house spec.

Audit flags

8 high and 9 low flags from agent tool calls and the method's commits. Only audit.json lists the low flags.

Channel	Agent or commit	Tool	Path
ho_reflection	rep-2-r20:s1	Write	test/kalah/PlayInterfaceTest.java
ho_reflection	rep-2-r20:s1	Write	test/kalah/PlayInterfaceTest.java
ho_reflection	rep-2-r20:s1	Write	test/kalah/PlayInterfaceTest.java
hidden	rep-2-r20:s2	Bash	$CLONES_DIR/_heldout/rep-2-r20
ho_reflection	49218aa0b75359bee5d5cc29c2d0257eba48798b	-	test/kalah/PlayInterfaceTest.java:3
ho_reflection	49218aa0b75359bee5d5cc29c2d0257eba48798b	-	test/kalah/PlayInterfaceTest.java:4
ho_reflection	49218aa0b75359bee5d5cc29c2d0257eba48798b	-	test/kalah/PlayInterfaceTest.java:14
ho_reflection	49218aa0b75359bee5d5cc29c2d0257eba48798b	-	test/kalah/PlayInterfaceTest.java:17

Channel	Marker	Disposition
ho_reflection	java.lang.reflect	false positive
ho_reflection	getMethod(	false positive
ho_reflection	getConstructor(	false positive
hidden	_heldout	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	java.lang.reflect	false positive
ho_reflection	getMethod(	false positive
ho_reflection	getConstructor(	false positive

ho_reflection.

Definition: Held-out only. Added code or a tool write that uses reflection.

hidden.

Definition: A path named /hidden or a _heldout path outside the agent's allowed _heldout folders. The held-out method's own case folders live under _heldout.

Dispositions from the run's findings.json judge every high flag of their channel.

hidden. Verdict: false positive. Tag: verified.

Evidence: The flag is rep-2-r20:s2 listing its own _heldout/rep-2-r20 and its empty cases at agent-a753a784adc86e2ac.jsonl:15. It names no hidden path and no other build's dir.

ho_reflection. Verdict: false positive. Tag: verified.

Evidence: All 7 flags mark rep-2-r20:s1's PlayInterfaceTest, 3 as a Write and 4 in commit 49218aa. It reflects only on Kalah to check play and its public constructor, and uses MockIO only as a public test helper.

Step log

Step log, 10 rows and 10 columns

2026-10-30T06:02:07Z. Step: s1. Unit: pin public interface (play CLI). Tests added: 5. Cycles: 1. Sha: 49218aa. Agent: rep-2-r20:s1. Tokens: 757828. Seconds: 143.27.

Actions: Pinned play(IO, clockwise, sevenfive) by reflection and the quit command with start boards for all four variants (3 new specs under test/specs). Added Board and BoardView. Ran ./run.sh test PlayInterfaceTest red then green. Conflict logged: the Edit tool strips trailing spaces, so one sed call restored the required trailing space on the spec prompt lines against the standing no-sed rule.

2026-10-30T06:04:40Z. Step: s2. Unit: held-out black-box cases. Tests added: 84. Cycles: 0. Sha: 723769b. Agent: rep-2-r20:s2. Tokens: 630424. Seconds: 147.043.

Actions: Wrote 84 cases outside the clone: 7 requirements x 4 variants x 3 cases. Reference simulator replays all 9 existing specs exactly. No ./run.sh call because the step names no check. Conflict logged: a Python generator in the held-out scratch folder writes the case files against the standing no-scripted-edits rule because the step asks for generators there.

2026-10-30T06:05:04Z. Step: s3. Unit: hidden split. Tests added: 0. Cycles: 0. Sha: 89f4195. Agent: rep-2-r20:s3. Tokens: 132875. Seconds: 20.709.

Actions: split: 84 cases in 7 requirements, 56 visible, 28 hidden

2026-10-30T06:07:11Z. Step: L6-8. Unit: sowing. Tests added: 4. Cycles: 2. Sha: 57ec7d0. Agent: rep-2-r20:L6-8#1. Tokens: 845170. Seconds: 130.252.

Actions: Wrote UNITS.md with 7 requirements in landing order (sowing first because every other visible set needs moves). BoardTest red (no sow) then green with Board.sow. Two more BoardTest cases passed on first run. Move loop in play() turned visible sowing 8/8, extraturn 8/8, quit 8/8, wrap 6/8. ./run.sh test: BoardTest and PlayInterfaceTest green, 4 example tests red on later units. ./run.sh hidden ran once and played 0 specs, so the gap is not measurable and no fresh visible test was written.

2026-10-30T06:08:22Z. Step: L6-8. Unit: wrap. Tests added: 1. Cycles: 1. Sha: f2c0c37. Agent: rep-2-r20:L6-8#2. Tokens: 305621. Seconds: 68.282.

Actions: Wrap already worked from Board.sow. Added BoardTest full-lap case (skips opponent store, returns to own houses), green on first run so no red. ./run.sh specs visible/wrap 6/8: both STA case03 failures differ only by a capture, left for the capture unit. ./run.sh test: BoardTest and PlayInterfaceTest green, same 4 example tests red on later units. ./run.sh hidden ran once and played 0 specs, so no fresh visible test. Conflict logged: one sed call ticked UNITS.md against the standing no-sed rule.

2026-10-30T06:09:30Z. Step: L6-8. Unit: extraturn. Tests added: 1. Cycles: 1. Sha: e45e5db. Agent: rep-2-r20:L6-8#3. Tokens: 262491. Seconds: 59.375.

Actions: Extra turn already worked from Board.sow and the play loop. Added BoardTest case for an extra turn after a full lap, green on first run so no red. ./run.sh specs visible/extraturn 8/8. ./run.sh test: BoardTest and PlayInterfaceTest green, same 4 example tests red on later units. ./run.sh hidden ran once and played 0 specs, so no fresh visible test. Conflict logged: GAP.md and LOG.md rows appended with echo against the standing no-heredoc edit rule.

2026-10-30T06:11:25Z. Step: L6-8. Unit: capture. Tests added: 2. Cycles: 1. Sha: 47b68d0. Agent: rep-2-r20:L6-8#4. Tokens: 559356. Seconds: 112.409.

Actions: BoardTest capture case red then green with Board.capture. No-capture-when-opposite-empty case added. The wrap full-lap BoardTest case pinned pre-capture behaviour (a 13-seed lap ends in the emptied origin house) and now asserts the capture the README rules require. ./run.sh specs visible/capture 8/8. ./run.sh specs visible 40/56, only emptyhouse and gameend red. ./run.sh test: BoardTest and PlayInterfaceTest green, 3 example tests red on later units. ./run.sh hidden ran once and played 0 specs, so no fresh visible test. Conflict logged: one sed call ticked UNITS.md and echo appended GAP.md and LOG.md rows against the standing no-sed and no-heredoc edit rules.

2026-10-30T06:13:09Z. Step: L6-8. Unit: emptyhouse. Tests added: 1. Cycles: 1. Sha: aabbd8c. Agent: rep-2-r20:L6-8#5. Tokens: 724965. Seconds: 118.267.

Actions: Added PlayInterfaceTest empty-house case with its spec under test/specs (P1 empties house 2 and picks it again). Red on the missing message, confirmed with the fix stashed, then green with an empty-house check in play(). ./run.sh specs visible/emptyhouse 8/8. ./run.sh test: BoardTest and PlayInterfaceTest green, 2 example tests red on gameend. ./run.sh hidden ran once and played 0 specs, so no fresh visible test. Conflict logged: one sed call restored trailing prompt spaces the Write tool dropped, against the standing no-sed edit rule.

2026-10-30T06:15:02Z. Step: L6-8. Unit: quit. Tests added: 1. Cycles: 1. Sha: f457624. Agent: rep-2-r20:L6-8#6. Tokens: 488889. Seconds: 97.86.

Actions: Quit already worked from the play loop: ./run.sh specs visible/quit 8/8 before any change. Added PlayInterfaceTest case with its spec under test/specs (P1 earns an extra turn then quits at the repeated prompt). Green on first run once the spec store value was corrected, so no red. ./run.sh test: BoardTest and PlayInterfaceTest green, 2 example tests red on gameend. ./run.sh hidden ran once and played 0 specs, so no fresh visible test. Conflict logged: sed restored trailing prompt spaces the Write tool dropped and fixed the spec store value, and echo appended GAP.md and LOG.md rows, against the standing no-sed and no-heredoc edit rules.

2026-10-30T06:17:53Z. Step: L6-8. Unit: gameend. Tests added: 2. Cycles: 1. Sha: e70a6d1. Agent: rep-2-r20:L6-8#7. Tokens: 956908. Seconds: 168.189.

Actions: Added two PlayInterfaceTest small-board games (2x1 P1 wins, 1x1 tie) with specs under test/kalah, played through a package-private play(IO, clockwise, Board) seam. Red on the missing Game over, then green with Board.hasSeeds, Board.score and the end-of-game check and result in play(). ./run.sh test: all 4 classes green. ./run.sh specs visible/gameend 8/8 and visible 56/56. ./run.sh hidden ran once and played 0 specs, so no fresh visible test. Conflict logged: sed restored trailing prompt spaces the Write tool dropped and echo appended GAP.md and LOG.md rows, against the standing no-sed and no-heredoc edit rules.

Raw inputs

metrics.json
summary.json
grade.tsv
costs.json
steps.csv
audit.json
review.json
LOG.md
MEASURE.md
classdiagram.uml
classdiagram.png
classdiagram.svg
collect.tsv
rating.json

MEASURE.md

# Measure

Method: Held-out TDD. Sources: LOG.md, GAP.md and git log. ATTEMPTS.md does not exist.

The branch has no merge commits. Each unit commit (the "Tick the unit" commit) counts as one merge below.

## Visible and hidden pass rate per merge

Visible counts come from `./run.sh specs visible/<unit>` and hidden counts from `./run.sh hidden` as logged in GAP.md.

| merge | sha | unit | visible unit | visible all | hidden |
|---|---|---|---|---|---|
| 1 | 57ec7d0 | sowing | 8/8 | not logged | 0/0 |
| 2 | f2c0c37 | wrap | 6/8 | not logged | 0/0 |
| 3 | e45e5db | extraturn | 8/8 | not logged | 0/0 |
| 4 | 47b68d0 | capture | 8/8 | 40/56 | 0/0 |
| 5 | aabbd8c | emptyhouse | 8/8 | not logged | 0/0 |
| 6 | f457624 | quit | 8/8 | not logged | 0/0 |
| 7 | e70a6d1 | gameend | 8/8 | 56/56 | 0/0 |

At merge 4 visible/wrap reached 8/8. Its two failures at merge 2 were capture outcomes. The hidden pass rate is undefined at every merge because `./run.sh hidden` played 0 specs each time. The s3 split put 28 of 84 cases in the hidden set, so they existed but the runner never reached them.

## Gap per noun

Gap means visible pass rate minus hidden pass rate per requirement.

| noun | visible | hidden | gap |
|---|---|---|---|
| sowing | 8/8 | 0/0 | not measurable |
| wrap | 6/8 then 8/8 | 0/0 | not measurable |
| extraturn | 8/8 | 0/0 | not measurable |
| capture | 8/8 | 0/0 | not measurable |
| emptyhouse | 8/8 | 0/0 | not measurable |
| quit | 8/8 | 0/0 | not measurable |
| gameend | 8/8 | 0/0 | not measurable |

## Hidden cases replaced after disclosure

0. No hidden result was ever disclosed because the hidden run played 0 specs. No fresh visible test was written in response and no hidden case was replaced.

## Credits and time

Credits and time are measured by the harness.

Terms and columns

See Methodology: measures and the Glossary for the terms and table columns this page uses.

How it builds code

Stage flow of Held-out TDD

Stages

Survey build

Replicates

Verdict

Report summary

Score card

Builds

Survey build (full-1)

Class diagram of method/held-out

Best replicate (rep-2-r20)

Class diagram of rep-2-r20

Median replicate (rep-2-r04)

Class diagram of rep-2-r04

Worst replicate (rep-2-r17)

Class diagram of rep-2-r17

Related methods

Records

Replicates (rep-2)

Consistency

Clone reports

Held-out TDD

Headline

Summary metrics

Quality, costs and audit

Review scores

Audit flags

Step log

Raw inputs

MEASURE.md

Replicate rep-2-r04

Headline

Summary metrics

Quality, costs and audit

Review scores

Audit flags

Step log

Raw inputs

MEASURE.md

Replicate rep-2-r08

Headline

Summary metrics

Quality, costs and audit

Review scores

Audit flags

Step log

Class diagram

Class diagram of method/rep-2-r08

Raw inputs

MEASURE.md

Replicate rep-2-r14

Headline

Summary metrics

Quality, costs and audit

Review scores

Audit flags

Step log

Class diagram

Class diagram of method/rep-2-r14

Raw inputs

MEASURE.md

Replicate rep-2-r17

Headline

Summary metrics

Quality, costs and audit

Review scores

Audit flags

Step log

Raw inputs

MEASURE.md

Replicate rep-2-r20

Headline

Summary metrics

Quality, costs and audit

Review scores

Audit flags

Step log

Raw inputs

MEASURE.md