Experimental design and approach

Study question

Classic greedy TDD writes the smallest failing test and then the smallest change that passes it. Do TDD variants that plan the class structure first, change the step size, try several starts or change how the tests check the code lead an AI coding agent to a better Kalah solution?

Run ended 2026-10-16. Model Claude Opus 5.5 (claude-opus-5-5). Runs: full-1 survey, tcp-1-list2 list reliability, rep-1 planned contrasts, rep-2 planned contrasts. 11 survey methods. Claims carry a tag. Recomputed means redone from source data. Verified means checked against a primary source. Unchecked means written or judged by an agent and not checked.

Task and variants

The agent builds the course's Kalah assignment. The brief below is docs/SPEC.md at tdd-experiment c3dd988 verified.

Excerpt of the Assignment 2 brief ("Making Changes"), adapted for this experiment. The game rules and the output format are in README.md. Example specs are in examples/.

Task

Implement Kalah so that it supports all four variants below, and keep the implementation alterable. The variant is chosen by the two boolean parameters of play().

Restrictions

A significant amount of the assessment is based on an automatic marking system. If you ignore any of the following restrictions, your submission will not be marked.

• No Java 9 (or later) features, including JDK. Java 8 is allowed.
• No reference to any other files (e.g. use of configuration files).
• No use of third-party libraries.
• Do not use System.exit() under any circumstances.

The play() method

A test suite tests all variants together through this method:

    /**
     * Play a game of Kalah.
     * @param io Where to get input from and direct output to
     * @param clockwise If true, then sowing of seeds should be done in a clockwise direction,
     * otherwise if false sowing should be in an anti-clockwise direction.
     * @param sevenfive If true, then the game should have 7 houses per player, and start with 5 seeds in each house,
     * otherwise if false then it should have 6 houses per player starting with 4 seeds per house.
     */
    public void play(IO io, boolean clockwise, boolean sevenfive) {
      ....
    }

The Kalah.java stub runs the anti-clockwise 6-4 variant because of the parameters its main() passes to play(). To play another variant, change those values:

  public static void main(String[] args) {
      new Kalah().play(new MockIO(), false, false); // Change values of boolean parameters as needed
  }

So to change to the clockwise 6-4 variant, change the first parameter to true.

Variants

Your implementation must allow all 4 variants: 6 houses with 4 seeds both clockwise and anti-clockwise, and 7 houses with 5 seeds both clockwise and anti-clockwise. The specific variant being tested is given by how clockwise and sevenfive are set. The test cases are the same scenarios for each variant (4 sets of test cases). examples/EXAMPLES.md lists a sample of the test specifications.

The four variants that play() selects verified.

Change cases

The implementation must allow all combinations of these two change cases.

Change Case 1: Sow seeds in an anti-clockwise direction

The game can be played with seeds sown in an anti-clockwise direction, the opposite of the clockwise sowing described in README.md.

Change Case 2: 7 houses with 5 seeds

The game can be played with 7 houses per player, each house starting with 5 seeds.

Harness pipeline

Every run goes through the same harness. Agents do the building, reviewing and rating. Harness scripts do the rest.

collect.sh runs collectOne for each clone and collectList for a list-only run. A step marked yes is skipped when fetchBack fails recomputed.

Runs and arms

An arm is a design or a test-list order. A run builds one or more arms under fixed caps.

A build is one final clone of a method or arm. Start clones and rebuilds are not builds. Multi-start TDD runs each start in a clone of its own, up to the starts cap, so pilot-1 makes 3 builds in up to 6 clones recomputed, pilot-2 makes 3 builds in up to 6 clones recomputed and full-1 makes 11 builds in 14 clones verified. The runs make 62 builds, the counts in Figure 2.2, and 5 rebuilds recomputed. Figure 2.2 rings the rebuilt replicates.

Every run of the study recomputed. Every run used model opus at medium effort with caps of 15 cycles per call and 3 starts. Methods doc is the short sha256 of the methods doc each run was built under. The methods text quoted on the study pages is doc 12f026f6.

• tcp-1-list. Writes the test list of 19 clauses and 12 candidates with lister tcp-1-list-lister, seed 20261020.
• tcp-1. Builds the tcp-1-list list in 5 orders.
• tcp-1-list2. Rates the tcp-1-list test list again with lister tcp-1-list2-lister, seed 20261020.
• rep-2/rebuild. Rebuilds r04 r08 r14 r17 r20 of rep-2 at a later harness. Analysis and validity covers why.

Replicate protocol

rep-1 and rep-2 test planned contrasts between arms. Both follow one protocol, and the build refuses to run if their shapes differ.

Study stage and course context

Both parts below come from the full-1 report that these study pages are built from. "This page" in them means that report.

Study stage

Each run of the study and its state. Complete means its folder holds its comparison page. Listed means it holds the test list and orders of a list-only run. Pending means it holds its launch arguments in args.json and neither page.

Course context

The COMPSCI 701 course items this study bears on.