Quelris

Blog

Same checklist, three agent designs.

I ran one review checklist as domain subagents, as one big prompt and as skills loaded on demand. Here is how each did, how that lines up with Anthropic's advice to fold subagents into one loop, and where my data disagrees.

13 min read

  • AI agents
  • Architecture

On October 22, Anthropic's Applied AI team published a write-up on when to fold subagents into the main agent loop. Their diagnosis is that many enterprise agents copy the org chart: one subagent per team, with an orchestrator on top. That design drops detail at every handoff, piles up overlapping subagents, collects conflicting prompt rules and adds latency. Their fix is one loop with one context, where skills and tools load only when the model needs them. They keep subagents for parallel work, verification and measurable context bloat, and fold the rest in one at a time against an end-to-end eval. Across several enterprise deployments, a single agent with skills often beat both the one-big-prompt design and the subagent design on quality, and often cost less and ran faster.

Most of my work over the past month ran through agents: implementing, designing, reviewing, handling tickets. I also happen to have run one checklist, in a security review tool I'm building, through exactly those three designs:

  1. five domain subagents, each with its slice of the checklist,
  2. the whole checklist pasted into one big prompt,
  3. the checklist split into skills that one session loads on demand.

This post summarizes their write-up, gives my read of it, and then puts my numbers next to it. My results agree with half of their finding.

What Anthropic found

Their write-up covers agents built on Claude Platform. The common enterprise version: each capability belongs to a different team, each team builds its own agent, and a central platform team puts an orchestrator on top. The agent ends up as a copy of the org chart. The designs they review often stack an intent classifier, a fixed reasoning pipeline, user-segment buckets and separate branches for suggestions under a multi-agent orchestrator, when one clear system prompt, four or five thin data tools and firm output guardrails would cover most of it.

They list five ways that version breaks:

  1. Context management. Details get dropped as context passes between agents.
  2. Sprawl. Hundreds of overlapping subagents with no enforced standards and no owner.
  3. The Frankenprompt. Every team adds a sentence to the system prompt, and the rules conflict.
  4. Local evals. Each team's evals are tuned to its own subtask, nobody maintains the whole set, and leadership decides on watered-down local scores.
  5. Latency. One customer that compared its multi-agent harness with a single loop saw roughly 2x faster runtime with the single loop.
The company: a platform team above four teams, orders, shipping, billing and returns, each owning one capability.The companyEach team owns one capability.Platform teamOrdersShippingBillingReturnsOne team per capability.The agent it builds: an orchestrator above four subagents, orders, shipping, billing and returns, one per team, each with its own prompt. It has the same shape as the company.The agent it buildsThe same shape, one level down.OrchestratorordersshippingbillingreturnsOne subagent per team, each with its own prompt.
Figure 1. The company and the agent it builds end up with the same shape.
A handoff: the subagent's context holds six details, but the summary it sends to the orchestrator carries two of them.A handoffDetail drops at every pass.subagent contextsummaryorchestrator receives2 of 6 detailsThe Frankenprompt: one system prompt with a rule from each team. The orders rule, answer in one short paragraph, and the billing rule, list every option first, pull against each other.The FrankenpromptEvery team adds a sentence.system promptordersAnswer in one short paragraph.shippingQuote the carrier word for word.billingList every option first.returnsAsk before quoting policy.platformUse a friendly tone.Two teams' rules pull against each other.
Figure 2. Two of the five: a handoff drops detail, and rules from different teams collide in one prompt.

What they suggest instead

Their alternative is progressive disclosure. The agent sees short descriptions of its skills and tools and loads the full version only when it needs one. They give three reasons: shared context lets the agent connect results from earlier steps, short descriptions keep the system prompt small (Anthropic cut Claude Code's system prompt for its advanced models by over 80%, partly this way), and current models can carry far more in one loop than most teams expect.

The loop itself is plain. One model and one context window run an open-ended tool loop, and the harness sets turn and cost budgets and gates writes. Each turn the model emits a batch of tool calls, the harness runs them in parallel, and the raw results come back to the same model. The model writes its plan into the context on the first turn and revises it as results arrive. Procedures live in skills. Hard limits such as validation and approval gates live in tools.

Orchestrator and specialists: an orchestrator calls three specialists, orders, shipping and billing. Each one runs its own model loop with its own prompt.Orchestrator and specialistsEach specialist runs its own loop.OrchestratorordersshippingbillingA model loop and a prompt per specialist.One agent loop: one model with one context, inside a harness that sets turn and cost budgets. It has loaded the shipping steps skill and the order lookup tool; the returns policy skill and the approval gate tool stay unloaded until needed.One agent loopSkills and tools load on demand.harness: turn and cost budgetsOne model, one contextskillstoolsshipping stepsreturns policyorder lookupapproval gate
Figure 3. Specialists each run their own loop. The single loop keeps one context and loads skills and tools when it needs them.

Where subagents still earn their place

Anthropic keeps subagents for verification, parallel work and modular permissioning, and lists three signals for keeping one: context bloat you can measure in a single loop, parallel tasks that need fresh context windows you throw away afterward, and a single agent that keeps picking the wrong tool or has conflicting instructions. When you do split, divide by context. Give each subagent a piece of work that needs little from the rest of the system and hands back a compact result.

My own daily setup over the past few weeks fits those cases:

  • a search agent on a small model with its edit tools removed and my instructions file left out,
  • a reviewer that reads and runs checks, also without edit tools,
  • an implementer for scoped changes,
  • a designer that has to screenshot its own result at desktop and phone widths.

Both read-only agents can still run shell commands, so the boundary is a tool list, not a sandbox. I haven't run these through an eval either. I keep them because they give a fresh context or a narrower tool set, not because they scored better. The rules around them matter as much: no delegating tiny or tightly coupled edits, never two subagents on the same files, one-shot with a concrete scope, no resuming one after its cache has expired, and subagents never commit.

Split by problem type: a message passes from orders to shipping to billing and gets shorter at every hop.Split by problem typeA game of telephone.ordersshippingbillingthe messageShorter after every hop.Split by context: the main loop sends two self-contained jobs to subagents with fresh context windows, running in parallel, and each sends back a compact result.Split by contextSelf-contained jobs, run in parallel.Main loopjob Afresh contextjob Bfresh contextLittle goes in, a compact result comes back.
Figure 4. Splitting by problem type passes the message along and it shrinks. Splitting by context sends out self-contained jobs.

Fold in stages

Anthropic's order: start with the subagents that get the most use, and merge specialists that chain on each other's output or are often called together. Before rewriting anything, have the main agent call the remaining subagents as tools, so it keeps the conversation and you get a baseline. Then turn one subagent at a time into skills and MCP tools, and compare the eval with and without it.

  1. Build a small end-to-end eval first.
  2. Fold in one subagent at a time, starting with the ones used most.
  3. Run each task more than once and check which tasks flipped, along with the overall score.
  4. Keep it as a subagent only if the folded version scores worse.

They also published a migration kit that chains their Claude API skills into a guided workflow.

Each pass: build a small end-to-end eval, fold in the most-used subagent, run each task more than once, check which tasks flipped, then repeat with the next one.Each passOne subagent, measured end to end.1Build a small end-to-end eval2Fold in the most-used subagent3Run each task more than once4Check which tasks flippednextoneKeep or revert: compare the folded version with the subagent on the end-to-end eval. If it scores the same or better, keep it folded; if it scores worse, keep the subagent. Then move on to the next most-used one.Keep or revertKeep a subagent only if folding scores worse.Folded version vs. the subagentsame or betterworsekeep it foldedkeep the subagentThen move on to the next most-used one.
Figure 5. One subagent per pass, measured against the same end-to-end eval.

Who owns what

Teams don't have to give up ownership. They move from owning agents to owning skills and tools, and one small central team owns the system prompt and the end-to-end eval. Group tools by user intent, because past 30 to 50 tool definitions in the system prompt tool choice tends to degrade. Reference material such as policies and specs goes behind a lookup tool or into a skill's reference files, not pasted into prompts by code.

Teams own agents: each of the four teams, orders, shipping, billing and returns, owns a whole agent with its own prompt and evals, and nobody owns the whole.Teams own agentsPrompts and evals multiply.ordersteamagentpromptevalsshippingteamagentpromptevalsbillingteamagentpromptevalsreturnsteamagentpromptevalsFour prompts, four eval sets, no owner for the whole.Teams own skills and tools: one central team owns the system prompt and the end-to-end eval of a single agent loop. The orders and billing teams each own a tool, and the shipping and returns teams each own a skill, that the loop uses.Teams own skills and toolsOne central team owns the prompt and the eval.Central teamsystem prompt · end-to-end evalAgent looporderstoolshippingskillbillingtoolreturnsskill
Figure 6. Ownership moves from whole agents to the skills and tools inside one loop.

My read of the write-up

Three things stand out to me, before any of my own data.

Routing doesn't go away, it moves

An orchestrator that picks a subagent and a model that picks a skill are doing the same job. Progressive disclosure moves that decision out of code you can log and into a choice the model makes inside one context. Their own eval advice shows it: once a subagent is folded in, they switch from tracking routing accuracy to checking whether the agent loaded the right skill. The failure is the same, only quieter. When a skill doesn't load, nothing errors. The agent answers without it.

Most of it is about ownership

Of the five failure modes, sprawl, the Frankenprompt and local evals are organizational problems, and so is the fix: teams own skills and tools, and one small team owns the system prompt and the end-to-end eval. A company that folds its subagents into one loop but keeps per-team prompt sections and per-team evals will rebuild the same problems inside one agent. The architecture change only holds if the ownership change comes with it.

The method will outlast the advice

Their closing admits that guidance shifts as models change, and their capability argument rests on current models that always think and check their own work. The durable part is the migration method: an end-to-end eval first, one change at a time, repeated runs, and a look at which tasks flipped. That applies whichever design wins on your task, which is why the rest of this post leans on it.

How I measured

The targets were Next.js and Supabase apps. Four were the development set: three public repositories built with AI coding tools, plus one of my own. Every design and the baseline ran on the same model (Sonnet) with read-only tools. The baseline was a one-sentence prompt: "Review this app for security problems before I launch it."

Judging changed partway through, which matters for comparing numbers:

  • First comparison. One run per method. A reviewer agent merged both methods' claims, checked each against the code, and did a short sweep of its own. That produced a list of 41 real issues.
  • Later comparisons. The 41-issue list was frozen, and one AI mapper matched each run's claims against it without knowing which design made them. Repeats became the norm.

No human checked the issue list, so read every recall figure below as recall against that list, not against every bug in the apps. Effort, time and token budgets weren't matched across designs.

Design 1: five domain subagents

Auth, database, secrets, input handling and abuse each got their own auditor and checklist, running side by side. In the first comparison:

  • One-sentence prompt: 31 of 41 real issues (76%), 6 false claims
  • Five domain subagents: 20 of 41 (49%), 3 false claims

Most of the gap was low-severity hygiene. On medium and above it was closer, 10 against 8, and the single critical issue went to the plain prompt.

  1. Each one saw only its slice. The plain prompt found real problems between the checklists: an injectable CI workflow, over-broad repository permissions, an open redirect, missing Origin checks, a race on a balance. No auditor owned those, so nobody looked.
  2. They passed checks they hadn't verified. The auth auditor passed a check while users could likely give themselves privileged account fields, the only critical issue in the set. The check's wording didn't cover writes to individual columns, so that pass was as much a checklist gap as a skipped verification. Either way, a wrong pass is worse than no answer, because the report tells the owner the problem is handled.
  3. They were shallow. Auditors finished in 20 to 70 seconds on apps of 5,000 to 75,000 lines.

The two methods weren't redundant, though. Together they found 39 of the 41, and the subagents caught 8 the plain prompt missed.

A second version added a sixth auditor with an open brief, required every pass to list what it examined, added the missing checks and raised effort, all at once. Recall went to 59%, or 51% if issues mentioned only inside the note of a passed check don't count. The auditors still wrote "pass" over gaps they had described in their own notes.

In Anthropic's terms, this was a one-person version of splitting by problem type. My auditors ran side by side with no handoffs between them, so I didn't hit their first failure mode. I did hit the fourth: each auditor passed its own fixture of checks, and none of them owned the issues between the checklists.

Design 2: one big prompt

Next I pasted the whole checklist, with playbooks for each area, into a single prompt of about 220 KB. One session, one context, everything loaded from the start. Scored against the frozen list:

  • One-sentence prompt, 4 runs: 63%
  • One big prompt, first version, 8 runs: 76%
  • One big prompt after tuning on the same four apps, 4 runs: 90%

The plain prompt's 63% here is the same prompt as the 76% above, re-scored under the stricter setup.

The 90% is inflated by tuning on the same apps, so I froze that version and ran it once on a held-out set of eight apps it had never seen, three runs each:

  • One-sentence prompt: recall 55%, critical and high recall 84%, precision 53%, slowest runs (p95) about 4 minutes
  • One big prompt: recall 70%, critical and high recall 88%, precision 52%, slowest runs (p95) about 7 minutes

The big prompt found more, at about the same precision, and it still made six wrong passes across those runs. Both cost about $1.10 a run on average.

Design 3: skills loaded on demand

Then I moved the playbooks into a skill. One session sees a short index and loads the playbook it needs, which is close to what Anthropic describes. I changed the review method at the same time, to a more focused first pass, so this isn't a clean comparison of where the instructions live. Over 12 runs on the development set:

  • recall was 61%, below the big prompt and about level with the plain prompt's 63%,
  • both critical and high issues in the set (there are only two) were found in every run,
  • 75 of the 77 claims decided so far were real, with 29 still pending,
  • the 95th-percentile run took about six minutes and cost about $1.50.

That version missed the gate I had written down in advance, recall at least the plain prompt's 76% plus 10 points, by 25 points. The stop rule says a miss of 20 points or more after three repeats means redesign, not more tuning, so tuning ended there.

When I classified the misses, almost none needed a new playbook. Most had a playbook that existed and wasn't applied: the run marked a check "manual" when the repository itself showed the gap. The next versions added a bounded second pass over files the first pass never opened, made those checks fail when the gap is visible, and split findings that need different fixes. In a preflight, one run per app, that reached 85%, one point under the gate. Splitting findings also changes how many get credited, and single runs sit inside the run-to-run noise below, so that number isn't a result yet.

Build the eval before the architecture

Anthropic's migration starts with a task-completion eval: one shared end-to-end set owned by a central team, plus smaller sets each team ships with its skills. Ten golden examples are enough to start. Grade the final answer against a rubric, track routing accuracy while an orchestrator still exists, and switch to checking skill and tool selection once a subagent is folded in.

What helped most in mine:

  • the list of real issues, built from the first runs, was frozen and hashed, so later runs couldn't quietly redefine "real",
  • a held-out set of eight apps was kept out of tuning and used once, on a frozen version,
  • the gates were written down before the skill versions ran: recall, critical and high recall, false positive rate, wrong passes, and 95th-percentile time and cost,
  • the stop rule above, which ended the 61% version.

The one-sentence prompt turned out to be the most useful row in the whole eval. Without it, the five subagents would have looked fine.

Run it more than once

In their sample support agent, folding one subagent into another cut latency but broke a process that had been passing. The main agent now held the order lookup tool and answered by itself, without asking the shipping specialist for the delivery date. The break showed up in only one run out of two.

My numbers say the same thing. Across three runs of the same version on the same app, recall had a standard deviation of 7.5 to 20 points depending on the app. My first comparison was single runs, and so is the 85% preflight. A single run can make a change look like a win or a loss when it is neither.

A simplified eval result after folding one subagent in. Latency is lower than before. Of four tasks, the delivery date task passed in run 1 and failed in run 2; the others passed both times.Eval · after folding one subagentLatencybeforeafter1TaskRun 1Run 2Order statuspasspassDelivery datepassfailChange addresspasspassRefund rulespasspass2
  1. Latency went down, so the change looked like a win.
  2. The delivery date task failed in one of two runs. A single run could have missed it.
Figure 7. An eval after folding one subagent in (simplified; task names are illustrations).

Where my data disagrees

My results back their first point: splitting a review by domain left gaps that no subagent owned. They don't back the second, at least for this kind of task.

1. Skills didn't beat one big prompt

The one big prompt beat the plain prompt on the development set and on the held-out set. The skill version came in about level with the plain prompt and below the big prompt, until I added a forced coverage pass. The method change in that version muddies the comparison, but the miss pattern points somewhere specific. On-demand loading works when the agent knows what it needs. A review is about the places it doesn't expect a problem, and the playbook it doesn't reach for is the one that would have caught it. For coverage tasks like reviews and audits, I'd keep a one-big-prompt version in the eval and expect the skill version to have to beat it.

2. Breadth tasks may want parallel subagents

The plain prompt and the five subagents together found 39 of 41 issues, far more than either alone. In some benchmark runs the plain prompt also spun up parallel subagents on its own to cover more ground. Anthropic already keeps subagents for parallel work. For review work, I'd lean on that exception more than the write-up does: a plain open-ended reviewer next to a checklist reviewer, each in a fresh context, looks better in my data than any single design.

3. Long contexts are where cost piles up

Anthropic's first signal for keeping a subagent is measurable context bloat, and I'd weight it more. In my own Claude Code logs, when I replayed 83 sessions with handoffs at 120k tokens, the six sessions that went past 500k produced about 86% of the savings. That doesn't compare architectures, and prompt caching makes re-reading a long context cheap. But it suggests that long single sessions are where the cost goes, and a loop that keeps everything in one window produces more of them.

What my data can't tell you

My runs used Sonnet. Their capability argument is about Opus 5.5-class models, which always think and check their own work, so my numbers may say as much about the model as about the architecture. My evidence is also thin: four development apps, eight held-out apps, AI judging and some single runs. Their numbers come from customer cases with no published eval design. Neither settles it. Your own eval, with a plain-prompt baseline in it, does.

Source: Building agents: When to fold subagents into your main agent loop, Anthropic.