Agentic software engineering · October 2026

A subagent crew for coding agents

Development and evaluation of a five-role delegation pipeline for Claude Code. The primary evidence is a controlled effort comparison study on tasks drawn from the history of a real team-built mobile app.

Summary

The crew at high effort shipped the fewest defective changes of six setups and sits on the correctness frontier. It costs less than max effort solo and less than ultracode. It costs about three times a solo agent at the same effort, so on cost alone a solo agent at high effort is the better buy.

0 of 18shipped diffs with a leftover defect for high crew, the lowest of six setups recomputed

0.10failure or error rate per run for high crew. High solo 0.18, max solo 0.24 recomputed

$10.94 vs $15.88mean API cost per task, high crew against high ultracode solo recomputed

3.34×cost of high crew over high solo, 11 tasks, Holm p 0.003 recomputed

The correctness edge is a direction. It is not yet a significant result. The smallest pairwise Fisher exact p on defect counts is 0.105 and the grading rule was set after the results were seen. recomputed

Frontier graphs

Each point is one setup on the 11 tasks complete in every setup. A frontier line joins the setups that no other setup beats on both axes at once. A point off the line is dominated.

Correctness against cost

Failure or error rate counts runs not resolved plus resolved runs whose shipped diff has a leftover defect, divided by all runs. The line runs through medium solo, high solo and high crew. High crew holds the lowest rate on offer at 0.10. No setup with a lower rate exists at any cost. recomputed

Quality-weighted effect against cost

Effect is the resolve rate weighted by review quality at weight 0.30. The line runs from the origin through medium solo and high solo. High crew matches ultracode on effect at $4.94 less per task. Both sit below and right of high solo. recomputed

solo agent crew ultracode Ring: high crew

Read together the graphs show a trade. High solo buys the most effect per dollar. High crew buys the lowest failure or error rate and pays about $7 more per task for it. Ultracode costs 1.3 times high crew with no gain on either axis. That last gap is not significant at Holm p 0.107. recomputed

The crew

A main session talks to the developer and delegates each code change to five subagents in a fixed order. Each subagent runs in its own context with its own model and tool permissions. Only the implementer may write files. The reviewer reads the diff and cannot edit it. The main session keeps every outward step such as pushing, opening the pull request and deploy decisions. Merging stays with the developer.

Opus model Sonnet model main session light tier path failure path

The workflow is called RPD for Requirements, Plan and Deliver. Small diffs in one package take a light tier that skips the reviewer. Review fix rounds are capped at three before the developer decides. Every bug fix must ship with a regression test that CI runs.

Agent design framing

A common way to classify model use separates a one-shot prompt, a fixed workflow and an agent. The crew is a hybrid. Its stage order is a fixed workflow written in rules. Each stage is a full agent that runs the perceive, reason, act and observe loop with tools. Current agentic coding tools share the same building blocks this crew uses.

Agent concept	In the crew
Plan-then-execute	The architect writes a noun-verb plan. The implementer executes it in numbered groups.
Reflection	The reviewer critiques each diff. Must-fix findings go back to the implementer for at most three rounds.
Subagents	Five roles, each with its own context and model. The parent keeps control of every outward action.
Permissions	Allow, ask and deny per role. The reviewer and explorer cannot edit. A push that deploys needs explicit approval.
Instruction file	A shared rules file such as AGENTS.md or CLAUDE.md read at every session start. Instructions, not memory.
Tests as ground truth	The verifier runs local checks pinned to the CI toolchain. CI must start before the turn ends.
Delegation has a budget	Each subagent re-reads context and tokens multiply. The study below measures that cost directly.

Ultracode is the comparison setup. It lets a single agent write an orchestration script at run time that fans work out to many subagents. In these terms the agent builds its own fixed workflow per task. The crew fixes the workflow once in advance.

A common rule of thumb says an agent earns its cost when the steps are unknown in advance. The study asks the same question one level up. Does a pipeline of agents earn its cost over one agent?

Study design

The effort comparison study is a controlled benchmark with a pre-registered analysis plan. Every setup used the same model, Claude Opus 5.5, on the same tasks in a sandbox with no network access. Setups differ only in reasoning effort and in whether the crew or ultracode is switched on.

Six setups

Arm	Setup	Role in the study
A	high effort, solo	Registered family. Same effort as the crew
B	max effort, solo	Registered family. Primary comparison with C
C	high effort, crew	Registered family. The crew under test
D	medium effort, solo	Secondary family
E	medium effort, crew	Secondary family
F	high effort, ultracode solo	Extension stage, one repetition per task

Each crew comparison pairs the crew with the solo setup one effort level up. The crew spends more compute per task, so the fair question is whether the same money spent on more solo effort does better. The crew configuration is a snapshot pinned to one commit. The solo setups run the same rules with the crew sections removed. verified

Tasks from a real project history

The source project is a location-based mobile app. It is a Flutter app in the MVVM pattern with a Node back end on PostgreSQL and PostGIS, built by a five-member student team. Its git history supplied realistic tasks. The study read it through a one-time static import and never touched the live repository.

Legacy stratum

Merged pull requests replayed from their base commit

Merged PRs in the import

214

Closing issue, 50 to 400 lines, tests

Flutter candidates

Extracted

Compile at base

Used after leak check

14 of 17 extracted tests call API names the issue text cannot reveal. A solver could not pass them without seeing the answer. verified

Entry stratum

Fresh tasks written from real issues

Pre-registered tasks

Kept after calibration

Complete in every setup

Each task has a sanitized issue text and stubs at the entry points it names.
Hidden fail-to-pass tests decide resolution. Each task has 1 to 14.
Six come from the author's own issues, four from teammates' issues and three span the full stack.

The entry stratum carries the analysis. The legacy stratum is too small to test and is only described. verified

Controls and measures

Each run starts in a fresh sandbox with no network egress. Runs that broke the sandbox are left out. verified
Resolution is the hidden tests passing on the agent's committed diff. Unresolved runs count as failures. verified
Defects come from a blinded review of each shipped diff after the runs ended. verified
Cost is API-equivalent USD per task. Time is wall minutes per task without API retry waits. verified
A budget guard started no new run while either usage window stood at or above 60%. verified

Analysis plan

Friedman test across the registered family, then paired Wilcoxon tests with Holm correction. verified
Hodges-Lehmann shifts with a bootstrap over tasks for confidence intervals. Cochran's Q for resolution. verified
Cost ratios are geometric means of per-task ratios. verified
Sample size was fixed in advance with no interim look. With 12 tasks the paired design detects an effect of about dz 0.81. verified

Stage full ran 84 runs across A, B and C. Stage full-med ran 56 runs across D and E. Stage full-ultra ran 14 runs of F. API-equivalent spend was about $1,170 in total. verified

Field evidence

Before the controlled study the crew ran on real work. Logs from those sessions give two observational windows. They show cost and process trends and cannot isolate the crew's effect.

Measure	Window 1	Window 2
Scope	one app project	six project groups
Agent runs	737	1,575
Total cost, USD	≈ 745	≈ 2,140
Must-fix defects caught by the reviewer	40 at $2.61 each	not recomputed
Median cost per product issue	$12.42 → $3.96	no significant change
Median time to pull request	43 → 15 min	no significant change

Window 1 gains came alongside a shift to smaller issues, so they mix the crew's tuning with task size. A shorter architect plan is the one change with a clean measured effect. Cost per 100 lines of code rose. Source is the crew architecture report built from session logs. unchecked

Critical questions

Does the crew raise correctness?

direction only

High crew shipped 0 defective diffs of 18 and had the lowest failure or error rate. No pairwise difference reaches significance. One extra defect could reorder the setups. recomputed

Is the crew cheaper than raising solo effort?

supported

This is the pre-registered primary comparison. High crew costs 0.77 times max solo at Holm p 0.003 and resolves 18 of 20 runs against 17 of 21. recomputed

Does the crew pay for itself against a solo agent at the same effort?

answered no on cost

High crew costs 3.34 times high solo and takes about 3.6 times as long. High solo resolves a larger share and has the best effect per dollar. The crew buys fewer leftover defects for that price. recomputed

Is the crew better value than ultracode?

open

High crew costs $4.94 less per task at equal effect and a lower failure or error rate. The cost ratio is not significant at Holm p 0.107 with one ultracode repetition per task. recomputed

Does the crew help at medium effort?

answered no

Medium crew costs more than high solo and resolves a smaller share of runs. Its failure or error rate is the highest of six setups. recomputed

Do the results generalise beyond this benchmark?

open

The tasks are small entry tasks from one project. The rule of thumb predicts agents earn their cost when steps are unknown in advance. Larger tasks with unclear steps may favour the crew more. This study does not test that. unchecked

Limits

Every figure on this page uses one basis. That is gate-passing runs on the 11 tasks complete in every setup. Task 356 leaves because one setup has no valid run there. One max solo run fails the reconcile gate. Three runs with a sandbox violation are left out. recomputed
The error rate grade and the ultracode pairing rule were set after results were seen. Treat both as exploratory. verified
Ultracode ran one repetition per task due to cost. The other setups ran two. verified
A defect count of 0 is a lower bound. The rule of three puts the upper 95% bound for high crew near 0.17. recomputed
One model, one project and 11 analysed tasks. The same person designed the crew and the benchmark. Hidden tests and blinded defect review reduce that bias but do not remove it. verified

Data

Entry stratum, 11 tasks, gate-passing runs. Lower is better for every column except resolved.

Setup	Runs	Resolved	Defective diffs	Failure or error	Effect	USD per task	Min per task
medium solo (D)	22	19	4 of 19	0.32	0.86	3.08	9.1
high solo (A)	22	21	3 of 21	0.18	0.95	3.81	10.9
medium crew (E)	21	16	2 of 16	0.33	0.77	8.93	29.7
high crew (C)	20	18	0 of 18	0.10	0.91	10.94	38.9
max solo (B)	21	17	1 of 17	0.24	0.82	14.34	47.6
high ultracode solo (F)	11	10	1 of 10	0.18	0.91	15.88	36.2

Rows sorted by cost. Effect is the quality-weighted resolve rate at weight 0.30. Paired cost ratios use exact Wilcoxon tests with Holm correction within each family of three pairs. Ultracode pairs use repetition 0 of each arm. recomputed

Sources

Effort study repository at commit 1ef5534. Files results/full/dashboard/grades.csv, frontier.csv and ratios.csv, plus docs/plan.md and docs/handoff.md.
Crew architecture report, built from Claude Code session logs for the two field windows.
Agent design concepts follow standard material on agentic AI and agentic coding tools.
Tags. recomputed redone from the study data. verified read in a primary study file. unchecked taken from a secondary report.