I made the picks. The agent set the board

When an agent marks one option as its pick and puts it in the first slot, I take it about two times in three. Count only the times I chose from the list, and the rate rises to three in four. I made each of those picks. The agent wrote the options, put them in order and arrived with a preference of its own.

What sent me to my logs was a study from Anthropic, How AI agents shape experts' judgment calls. It asks whether experts who hand their work to an agent still make the decisions that matter, or defer to the agent without noticing. Because it is a randomized experiment, it can measure influence, which my logs can't. I'll go through it first, then put my numbers next to it.

What Anthropic tested

Fifty professional analysts, all experienced in causal inference (figuring out from data whether one thing causes another), did a realistic statistical analysis in Claude Code running Opus 4.8. The task was set at a fictional farming cooperative. Its board wanted to know whether a paid advisory program for member farms raised productivity enough to expand it, and it would expand the program only if the gain cleared roughly 3 to 4%.

Along the way, each analyst made four analytical choices, each between two defensible options, and then a final call:

Outcome: crop yield per hectare, or yield per unit of operating cost.
Comparison method: how enrolled farms changed after joining against how other farms changed over the same period (difference-in-differences), or a direct comparison adjusted for differences between farms.
Coverage: the enrolled farms only, or the cooperative's whole membership.
Weighting: every farm counted equally, or larger producers weighted more.
Final call: roll the program out to everyone, or discontinue it. A one-year extension was also allowed.

Each of the four analytical choices could move the estimated gain by several percentage points, which matters when the threshold is 3 to 4%.

For every decision, a coin flip set which option the analyst's agent would favor. The lean sat in hidden instructions, which told the agent to hold it as its own view, raise it when the decision came up and make the strongest case the data allowed. The same instructions told it to change its mind if the evidence turned decisively against it, and never to misrepresent results or make an argument the numbers didn't support. The analysts weren't told about the lean. Because it was random, comparing analysts whose agents leaned opposite ways measures what the lean did.

A coin flip picks which of the two options the agent will favor, separately for each decision.
The lean goes into hidden instructions. The analyst isn't told about it.
The agent raises its view and makes the strongest case the data allows.
The analyst picks. The study compares the share choosing A when the agent leaned A against the share when it leaned B.

Figure 1. How Anthropic's study set up each of the five decisions, and the comparison it uses to measure what the agent's lean did.

Sixty-one analysts were recruited. Eleven chose not to use the agent and are left out of the main results, which leaves the 50.

What they found

They went with the lean 86% of the time

Across the four analytical choices, analysts' picks matched their agent's lean 86% of the time. If the lean had made no difference, the rate would be about 50%. By choice, it was 92% for the outcome measure, 82% for the comparison method, 81% for coverage and 90% for weighting.

Figure 2. Anthropic's study only. Analysts picked the option their agent leaned toward 86% of the time across the four analytical choices, against about 50% if the lean had made no difference.

The final call moved less, but the estimate had already moved

The lean had a weaker effect on the final decision. When the agent leaned toward rolling out, 54% of analysts chose rollout; when it leaned the other way, 36% did. The authors think the effect was smaller because, by that point, the call mostly followed from whether the estimate cleared the board's threshold. That estimate, though, had already been shaped by the leans on the four earlier choices. From those four coin flips alone, a statistical model correctly predicted 82% of the analysts' final calls, against 61% for always guessing the most common one. That count leaves out the analysts who proposed an extension.

Figure 3. Anthropic's study only. The lean moved the final call less, but the four earlier coin flips alone predicted 82% of final calls. Analysts who proposed an extension are left out of the right panel.

Stated preferences gave way when the agent disagreed

In 38 cases, an analyst had stated a preference on one of the choices before the agent voiced its lean. When the lean went against them, they switched 60% of the time; when it agreed, 6% of the time. The authors attach caveats to this one. It is a small subset, the 95% interval on that 54-point gap runs from 24 to 82 points, and the analysts who spoke first may differ in some way from those who didn't.

Experience didn't predict who deferred

The team checked whether any of ten traits predicted how often an analyst followed the lean, among them years of experience, education, a causal-inference quiz score, trust in AI and how often they used it. Only age showed a link, with older analysts following more often, and that link disappeared once they corrected for testing many traits at once. Most analysts followed on most decisions, so there was little variation to explain.

They called the result their own, in the agent's words

After the task, 52% of analysts rated the final decision as more their own than their agent's, and the other 48% said it was more the agent's. Both groups had shifted toward their agent by a similar amount: about 65 percentage points for the first group and 79 for the second, a difference within the margin of error. Confidence followed the same pattern. On a scale from 0 to 100, analysts who followed the lean on all four choices averaged 71, and the rest averaged 73.

Asked to explain each choice, without access to their conversations with the agent, 77% repeated an argument their agent had made, sometimes word for word. Only 15% credited the agent anywhere. Six of the 48 survey respondents guessed the study was about the AI's influence on their choices. None suspected that the agent's preferences had been set for it.

Asked to explain each choice without access to their conversations with the agent, 77% repeated an argument their agent had made, sometimes word for word.
Only 15% credited the agent anywhere.

Figure 4. Anthropic's study only. Most analysts explained their choices with their agent's arguments, and few mentioned the agent.

The limits the authors name

The lean was strengthened on purpose. With no lean assigned, Opus 4.8 made the same five choices in 112 of 123 runs, so the model does arrive with defaults. The study made the agent hold and voice a preference reliably so that its effect could be measured, and the authors say the effect might be smaller when agents aren't told to voice one.
They studied one task, in one field, with one model. The task took about an hour. The analysts knew causal inference but were new to farming, and in their own field they would probably hold stronger views and could narrow the options by talking to the people involved.
The results are averages across analysts. The study didn't record each participant's view before the agent voiced its lean, so it can't say how much any single analyst was influenced.
It recorded which option each analyst chose, not how firmly they preferred it.

Analysts who used the agent also did better: they finished faster (median 65 minutes against 90) and made far fewer serious methodological errors (3 of 50, against 6 of the 11 who went without it). The authors don't credit the agent for that, because the 11 chose to work without it and were fewer and less experienced.

Why they think it matters

The authors offer one reason analysts deferred so readily: experts often work out what they think while weighing the options. The agents were often open about how results would change under each option, but they rarely made a strong case for the option they didn't favor, so analysts could adopt the agent's assumptions without forming their own. This kind of deferral is also hard to catch. Checking work against a right answer doesn't help when a judgment call has none, and the work looks sound whichever way it went. Asking people afterward may not help either. The study could measure deferral only because it randomized the lean, which real work rarely allows.

My read

The study is about a controlled analysis task. What I take from it is what it implies for the approval steps that agent tools put in front of us every day.

1. Order and the label travel through the same channel

In my Claude Code prompts, the option marked as the agent's pick sat in first place every time. When an option was marked, I chose the first one 71% of the time. When nothing was marked, I still chose the first option 69% of the time, against a chance rate of about 39%. The two rates are almost the same, but they come from different questions, and both samples are small, so they can't tell me how much the mark adds on top of position. The second agent tool I use never raises the question, since its prompt puts the agent's pick first by design.

So taking the label off a prompt may not make it neutral. With no mark at all, I still chose the first slot far more often than chance, and whoever orders the list decides what lands there. My logs can't tell whether that is the agent's preference showing through the order or just my habit of taking the first option.

38 answered prompts with a marked option. The marked option was always first, so the mark and the slot can't be told apart here.
13 answered prompts with no mark. I chose the first option in 9 of them.
The chance rate is the average of one over the number of options, computed for the unmarked prompts only. Both groups are small.

Figure 5. My Claude Code prompts only. With no mark, I still took the first option 69% of the time, about 30 points above the 39% chance rate.

2. Feeling that it was your call tells you little

Analysts who rated the final decision as their own had been steered about as much as those who said it was more the agent's. In that study, how people described their own judgment told you little about how much the agent had moved it. That takes out the cheapest check there is, which is asking afterward. It applies to me too. I feel that I chose every one of my picks, and the study suggests that kind of feeling says little.

A rejection rate doesn't settle it either. I didn't take the agent's pick about a third of the time. The study's analysts went against the lean on 14% of the four analytical choices and were still heavily steered. Saying no some of the time suggests I was reading the options. It doesn't show that the yeses were mine.

Figure 6. Anthropic's study only. About half the analysts called the final decision more their own, and that group had shifted toward the agent's lean about as much as the rest.

Many agent workflows pause and ask the person to pick before they move on, and the log then says the person picked. When the agent wrote the options, ordered them and argued for one of them, the log holds how often the person agreed with what they were shown. Two points from the study explain why this is weaker than it looks. People often form their view while weighing the options, so a menu with one side argued well can supply the view. And judgment calls have no right answer to check against, so the result looks sound either way.

The authors note that institutions often treat an expert's sign-off as evidence of independent judgment, and that this assumption gets shakier as AI does more of the work. My agent work is full of steps that wait for my pick. What those steps measured may have been my agreement rate, not my judgment. If the study holds, I might not notice the difference.

Figure 7. An approval log records a pick from the options the agent wrote, ordered and marked. It holds nothing about the person's view before seeing them, or their pick from a different list.

What my logs show

These are observed rates from my own question prompts, with nobody setting a lean. The conditions are in the last section.

I took the agent's pick about two times in three

Over several weeks I sent these two tools several hundred prompts. In the ones where the agent marked a pick, I took it 69% of the time. Another 22% went to a different listed option, and 9% were answers I typed myself, outside the list. Counting only the times I chose a listed option, I took the agent's pick 76% of the time: 75% in Claude Code and 78% in the second tool.

All answered prompts with a marked pick: 69% took it, 22% went to another listed option and 9% were typed outside the list.
Only the answers chosen from the list: 75% in Claude Code and 78% in the second tool.

Figure 8. My logs only, both tools combined. I took the agent's pick in 69% of answered prompts, and in 76% of the answers I chose from the list.

The agent writes the options

About one in ten of my answers were typed outside the list. Everything else was chosen from options the agent wrote. The count only says that a typed answer matched none of the offered options. It doesn't tell me whether I brought my own framing to those cases.

The agent sets the order

The agent's pick always sat in the first slot: in every marked Claude Code prompt, and by design in the second tool. With no mark at all, I chose the first option 69% of the time, against about 39% by chance. Among the unmarked prompts where I picked from the list, the first option took 82%.

The agent already has a preference

My logs can't show my agents' default preferences. The study measured one: with no lean assigned, Opus 4.8 made the same five choices in 112 of 123 runs. Nobody planted a lean in my ordinary work, so the pick I see marked may just be a default preference like that made visible.

Where my logs and the study part ways

My rate is lower. I took the agent's pick about two times in three, while the study's analysts followed the lean 86% of the time across the four analytical choices. The setups differ: the study's choices were two-way with no write-in, while mine had up to four options plus a typed answer. Anthropic set a hidden lean by coin flip in a controlled task and had the agent argue for it. My numbers come from ordinary work, where the agent's pick was marked in plain sight and nobody set anything. The lower rate doesn't show that I am harder to steer.
The study can estimate influence and my logs can't. Randomizing the lean let the authors tell following apart from agreeing. My logs have no counterfactual, so taking the pick two times in three could be sensible agreement with sensible suggestions, deference, or a mix.
The safeguard the authors raise can't be checked here. One design they suggest studying is an agent that asks for the expert's view before offering its own. Testing that against my logs would need a record of what I thought before the options appeared, and my logs don't keep one in any consistent way.

How I measured

I measured choice prompts from my own agent logs, across two agent tools: Claude Code's question prompts and a second agent harness whose question prompt puts its pick first.

Claude Code: single-choice questions where exactly one option carried the agent's mark. Duplicate questions were removed, and multi-select questions and unanswered prompts were left out. That gave 38 answered marked questions and 13 answered questions with no mark.
Second harness: the agent's pick is the first option by design, so choosing the first option counts as taking it. Copies of a conversation that the harness carries into a new session were counted once. That gave 27 answered prompts.
Both tools together, across the answered prompts with a marked pick. Each answer was classified as the agent's pick, another listed option, or a typed answer that matched no option.
Chance rate: for the 13 unmarked prompts, the average of one over the number of options, which came to 39%.
Hand check: 20 pseudo-randomly drawn rows from the raw logs, 10 from each tool, were checked by hand, and every classification held. The check didn't catch the session copies; a later pass on message identity did. One malformed row with no recoverable question was left out rather than guessed.
Cutoff: the counts stop at a fixed cutoff. The second harness keeps logging prompts, so the measurement ends at that point and later prompts are not counted.

The decisions were everyday ones: release scope, scheduling, content or interface format, tool access and operating workflow.

The conditions on these numbers:

They are observational. Nobody planted a lean, and I didn't vary the prompts.
Label and position are tied together. Every marked option was also the first option, so these rates can't separate the mark from the slot.
There is no counterfactual. I don't know what I would have picked from a different list or order, so the data can't tell sound agreement from deference.
The two tools word their prompts differently and were logged over different stretches.
My logs don't keep what I thought before seeing the options in any consistent way, and they don't link a choice to a later reversal.

So the rates describe what I picked. They don't show that the agent's pick caused my choice, or whether its pick was right.

I made the picks. The agent set the board.