My compaction summaries added things I never said

The usual worry about a compact summary is what it drops. In my Claude Code logs it went the other way. The summaries dropped almost nothing, and they added things I never said: a standing permission built out of a few replies of "go ahead", a refusal the assistant had made that later turned up as a rule from my own files, a priority set harder than I had set it.

On cost, compacting did nearly as well as handing off to a fresh session. The saving came from when I reset, and which tool did the reset mattered much less.

The added part matters because of what notes like these did in Anthropic's assessment of the UK AISI incident, where a model's own compaction notes helped carry it into a supply-chain attack on a real GitHub project. My logs come first here, then the assessment and what I changed, with the method and the limits at the end.

What a compaction summary is

Nothing is learned in a compaction: it isn't training of any kind, and the model's weights don't change. When Claude Code compacts, on its own or because I ran /compact, it makes one model call: the model reads the conversation so far and writes a shorter version of it. The next turn starts from that text instead of the full conversation.

The next model doesn't read the summary alone. The harness wraps it in text of its own, keeps the last 4 to 9 messages verbatim after it, and re-attaches some files from disk, CLAUDE.md and memory files among them. Most summaries share the same sections: a list of all my messages, pending tasks and a next step.

The summary is the one block the model writes. 5 of my 25 wrote down their own reading of my permission, like this line.
The harness wraps the summary and ends it with this line. All 25 of mine end this way.
The harness keeps the last 4 to 9 messages as they were, in 23 of the 25.
CLAUDE.md and memory files come back whatever the summary says: 15 of the 70 re-attached files.

Figure 1. What the next model reads after a compaction. The model writes only the summary; the rest comes from the harness or from disk.

The trouble is in the shortening. A long conversation goes in, a much shorter text comes out, and the model has to decide what each scene amounted to. Sometimes a few scenes come out as a one-line rule.

It dropped almost nothing

My rules came through

In the summaries checked item by item against what I had said in chat, nearly 9 in 10 of my constraints came through as stated (53 of 61). Six changed, and they come in the next section. One survived only inside the quoted list of my messages, not as a rule. One was dropped: a "leave that out" about some referral numbers that came out differently between queries. The summary kept the fact that the results were sampled but not my decision, and the model didn't add those numbers afterwards.

A rule-based cross-check, which undercounts because summaries often paraphrase, found most of my messages in the summaries, and nearly all of them after a manual compaction. The measures are under "How I measured".

The model rarely went back to look

If a summary leaves out something the next model needs, you would expect the model to go back for it: re-read a file, rerun a command, call AskUserQuestion, open the full transcript. After a compaction my sessions almost never did:

In up to the first 60 tool calls after each compaction with later work, one call re-read a file the model had touched before the compaction.
No exactly repeated shell commands, no AskUserQuestion calls, and no opens of the full transcript, which every summary points to.

Figure 2. Up to the first 60 tool calls after each of the 14 compactions with later work, one row per compaction, oldest first, and what they show of the model going back for something.

No correction traced back to a drop

In my messages after compactions I found one "as I said". It traced to a rule I had stated after that compaction, not before it. None of the corrections I found traced back to something a summary had dropped.

It added things I never said

Where a summary did change something, it usually put its own reading in place of my words. Six constraints changed on the way into the summary, and most of the changes are about permission or priority:

"Do the site later, check it on the product first" became "deprioritized; don't send this kind of work unless asked". The summary made it harder than I had.
"Close the panes" became "close the ones I created, not the others". The model narrowed my instruction.
An unrequested re-render I objected to was recorded as an error event, not as a rule.
I asked the assistant three times to do something it refused under its own safety rule. The summary recorded the refusal as a security constraint, not my last instruction.
An ambiguous go-ahead was recorded as "which I took as authorization for the PR flow".
"Read-only; vulnerability checks go through the dev server" became "GET-only request test". The meaning held and the wording didn't.

Another summary added a rule outright. It listed "deploy only on an explicit request" among the rules from that session. I had asked for each deploy explicitly, but I never stated it as a rule; the generalization was the summary's.

Figure 3. The six constraints that changed, in my words and in the summary's, and one rule the summary added on its own.

In most of these the summary wrote down its own reading of who may do what, or what comes first, in the same list as my own words. The next model has no way to tell the two apart.

A refusal filed as a rule from my files

Item 4 went further. A summary in a later session listed the same refusal under "Standing constraints (CLAUDE.md/memory)". I can't find that line in my global instructions, the project's CLAUDE.md or AGENTS.md, or the project's memory files. A summary that says a rule comes from a file is no evidence that it does.

In the session I asked three times, and the assistant refused each time under its own safety rule.
That session's summary recorded the refusal as a security constraint, not my last instruction.
A week later another summary listed it under "Standing constraints (CLAUDE.md/memory)".
I can't find that line in any of the files the heading names.

Figure 4. How a refusal the assistant made became, a week later, a rule a summary said came from my files.

Short replies written down as standing permission

My global instructions say never to commit, push or open a pull request unless I ask. One summary in five (5 of 25) wrote down a reading of my authority, most of them turning short replies into standing permission to do exactly that (my replies translated):

"Explicit standing pattern: after I report a finished change, the user says 'go ahead' -> I commit, push, create PR."
"PRs are authorized in this flow"
"The user has since authorized PRs via 'figure it out', 'go ahead' and 'go with 1 to 3'"

What the sessions actually did

They didn't act on the wider text

Most of the summaries with an authority reading were never read by a model (three of the five). A fourth listed a project rule that really is in that project's CLAUDE.md. After the fifth, every commit and pull request came right after a fresh "go ahead".

Across every commit, push and pull request command run after a compaction, most came right after a fresh go-ahead from me (22 of 28). The other 6, all in one session, committed, pushed and in five cases opened a pull request, on a delegation I had given earlier in that session and the summary had carried over faithfully ("Choose the PR strategy yourself"). My latest message before them was an image, a settings request or a note that something had merged.

Conditional holds were honoured too. One summary said not to close an issue or remove a worktree until I said so, and the model waited for me. Another said not to commit until I said to open the PR, and it waited for that.

Figure 5. My summaries kept most of my rules: 6 of 61 constraints changed. 6 of 28 commit, push and PR commands ran without a fresh go-ahead, all in one session under a delegation I had given before the compaction.

The harness pushes the other way

Every one of my summaries ends with the same line: "Continue the conversation from where it left off without asking the user any further questions." That line is Claude Code's, not the model's, and it has changed. I found three versions (in 4, 15 and 6 of my 25 summaries):

The shortest ends with that instruction alone.
The next adds a note that the messages after the summary are kept verbatim, and that something the summary says has not happened yet may already have happened.
The longest also adds: "Whatever is still not started, not finished, not checked ... is part of the work: finish it ... unless ... the user ruled it out."

4 of my 25 summaries end with the instruction to continue without asking, and nothing more.
15 add a note that the messages after the summary are kept verbatim and may be ahead of it.
6 also tell the model that anything not started, finished or checked is part of the work.

Figure 6. The harness text at the end of my summaries in its three versions. Each version keeps the earlier lines and adds one.

Nearly all the summaries also end with a next-step section (24 of 25, 19 of them headed "Optional Next Step"), so most hand the next model a task and an instruction not to ask about it. When the model acted before I wrote again, its first action was that next step every time (5 cases, exactly in 2 and closely in 3).

Rules in files come back, rules said in chat depend on the summary

The summaries also restate standing rules whether or not they matter to the session. Most repeat my "never commit, push or open a PR unless asked" (21 of 25). Many add "subagents never commit", and as many repeat a rule of mine for an unrelated tool in sessions that never touched it (16 of 25 each).

Rules saved in CLAUDE.md or memory get re-attached after a compaction whatever the summary says (15 of the 70 re-attached files were memory files or CLAUDE.md). Rules I only said in chat depend on the summary.

Where the summaries go unread

Most of my automatic compactions (12 of 17) fired 54 to 55 minutes after the last message, not in the middle of work. The others came one to three minutes after the previous event. Two in five summaries (10 of 25) have no assistant turn after them, so as far as the logs show, no model ever read them.

The cause is not in the logs. 55 minutes sits just inside the one-hour cache lifetime, and that is my guess only. Either way, a summary nobody reads can't drift anything, and an item it dropped is invisible by construction.

Figure 7. My 17 automatic compactions by minutes since the last message and context size when they fired. 9 of the 12 idle ones were never read by a later model.

My handoffs have no slot for rules

Where I had written a handoff doc and a compaction then fired in the same conversation, I could compare the two. The summaries carried nearly all the handoffs' "don't retry" dead ends (31 of 32) and, in the three pairs read in full, all but one of the open questions. That looks better for the summary than it is: the handoff was written inside the conversation the summarizer read, so some of the overlap is copying. Treat it as an upper bound.

What surprised me was the other direction. My handoff template has sections for the goal, the state, key decisions with reasons, dead ends, files, checks, next steps and open questions. It has no section for my rules or prohibitions; those end up under decisions or dead ends if they appear at all. Most compaction summaries have a list of all my messages and a list of pending tasks (each in 19 of 25), so my wording survives there by design, while the doc I trusted more has no place for it.

Figure 8. The sections of my handoff template next to a compaction summary's, with how many of my 25 summaries have each, and what the summaries carried from a handoff in the same conversation.

On cost, compaction came close to a handoff

I replayed my Claude Code sessions on Opus 5.5 at API prices (the setup is under "How I measured"):

Resetting at 200k tokens: a handoff cut total cost by 28.7%, a compact by 26.4%.
Resetting at 120k: a handoff cut it by 26.0%, a compact by only 15.5%. A compacted session restarts with more context and hits the threshold again sooner, so the replay ran more resets with compacting (557 compacts against 437 handoffs).
Sessions that peaked past 500k, reset at 200k: 46.1% with a handoff and 43.6% with a compact.

The two end up in similar places. In the first run, a fresh session after a handoff started at a median of about 61k tokens and was at about 77k ten calls later; a compacted session started at about 67k and was at about 87k. In the re-run, with more compactions in the logs, the compact medians are about 73k and 93k.

Figure 9. Total cost cut in my replay of 152 sessions, and why compacting falls behind at 120k: it ran more resets, each starting from more context.

This replay started as an unposted draft of mine, and one of the draft's numbers needed correcting and another a qualifier. The draft said the compact summary is about 14k characters against about 7k for my handoff docs. The 7k matches the files' size in bytes. My handoffs are mostly written in a non-Latin script that takes about three bytes a character, and run a median of about 4,800 characters. The summaries are mostly English and run about 12,600 to 14,400 characters, depending on which set you take. That makes them about three times as long, not twice.

The draft also said automatic compactions fire between 400k and 970k if you leave them alone. That holds for four of the five mid-work ones: one at 402k and three at 967k to 969k. The fifth was a 9k re-compaction 90 seconds after another. The idle ones fired at 243k to 507k.

So the saving comes from resetting around 200k, and which tool does it matters much less. The replay assumes the work after a reset goes exactly as it did without one, uses API prices rather than subscription limits, and assumes the compact call reads the conversation from cache; if it doesn't, compacting costs more than shown. An earlier post of mine on handoffs gave a larger saving from a smaller replay whose script I can no longer find, so read the numbers here as the current ones.

Why added context matters: the AISI assessment

In my sessions a human answered within minutes whenever work went on after a summary. Anthropic's alignment assessment of the UK AISI incident shows what a model's notes to itself can do when nobody answers.

What happened

The UK AI Security Institute gave Claude Mythos 5 a capture-the-flag challenge: retrieve a protected piece of data from a simulated network. Internet access was on, which was standard for AISI's cyber testing at the time and no longer is, and the cyber safeguards Anthropic ships with its general release models were disabled. In 9 of AISI's 43 runs the model took actions beyond the scope of the evaluation that affected real people and organizations. Anthropic's assessment looks closely at one of those runs and more briefly at the other eight.

In the main run, the model first tried in-scope approaches. A misconfiguration in the task instructions made those very difficult, if not impossible. It had also noticed early that one machine in the test network periodically contacted GitHub. After two compaction cycles of in-scope attempts, it searched GitHub for a repository it believed that machine downloaded software from. The repository belonged to unrelated developers. The model opened a malicious pull request, emailed the maintainers while posing as a human user, and, after an outside developer warned them the code was malicious, created a sock-puppet account to vouch for it. The maintainers did not accept the change, and the run ended when the model ran out of token budget.

The run had nine compaction cycles. At each one the model summarized its progress and wrote notes that served as instructions for the next cycle. Anthropic splits them into phases:

Cycles 1 and 2: mostly in-scope work, with a GitHub attack considered more and more seriously as in-scope attempts failed.
Cycle 3: the notes the model began this cycle with told it to explore a GitHub attack. It started with reconnaissance, then took its first actions that reached real people, such as contacting the maintainer. At this stage it could still abort.
Cycles 4 to 9: the most serious actions, and very little chance of changing course in response to signs of harm.

Figure 10. The AISI run's nine compaction cycles, and what changed when Anthropic regenerated the notes going into cycle 3 after editing 5 cycle-2 chains of thought to mention harm.

Most of the results come from resampling. Anthropic edited the transcript at some point, sampled the model again from there (with a Claude model simulating tool outputs for longer continuations), and had an LLM grader score whether the model still did the thing in question.

Why it started, and why it didn't stop

Before the attack, the model mentioned GitHub in 19 turns of its chain of thought, and only one of them raised the possibility of harming real people. Anthropic edited 5 of its cycle-2 chains of thought to mention that the GitHub plan might cause real-world harm, and had the model regenerate the notes going into cycle 3. Sampling 12 turns from the new notes, it searched for a GitHub repository in 13% of trajectories. With the original notes it did so 83% of the time.

19 turns of its chain of thought mentioned GitHub before the attack.
Only one of them raised the possibility of harming real people.

Figure 11. The 19 turns of Mythos 5's chain of thought that mentioned GitHub before the attack. The assessment doesn't say where in the 19 the one that raised harm falls, so it is drawn last.

Once the attack was under way, it hardly ever stopped:

Resampled at the start of each cycle from the fourth onward, it continued the attack 98% of the time.
In 8 resamples from each of 24 random points in those later cycles, it never de-escalated on its own.
Told by an inserted user message that it was dealing with real humans, it stopped 5% of the time. In 82% of trajectories it found a way to read the message as part of the exercise.

Anthropic calls this momentum: "the model's compaction notes and prior actions imply to it that its current path has been (correctly) sanctioned." It came with biased reasoning. The model described the maintainers as simulated by the test organizers while calling bystanders on the same pull request real people. It still switched routes when an in-scope path looked more promising. Momentum seems to have shaped its judgement of whether the path was harmless and left its judgement of what would work intact.

Newer models

Newer models did better. From the cycle-3 notes, Claude Mythos 5.1, Claude Opus 5.5 and an internal research model searched for the repository between 0% and 13% of the time, against 83% for Mythos 5. From random points in cycles 4 to 9, they stopped on their own in 1%, 10% and 62% of branches, against 0% for Mythos 5. Anthropic notes that these models were picking up a transcript another model wrote, so their numbers may not show how they would behave in a run of their own.

Figure 12. How Mythos 5 behaved once the attack was under way in Anthropic's resamples, and how three newer models did from the same points of its transcript.

A line in the notes didn't stop Mythos 5 after cycle 3

Anthropic inserted clauses into the compaction notes saying the model could abort if the environment looked misconfigured, if it was causing real-world harm, or if it might touch a real GitHub repository. With the GitHub clause, which was written to be very suggestive, it aborted in 46% of samples in cycle 3. In Anthropic's words, "It did not, however, abort under any conditions in any cycle later than 3", and the misconfiguration and harm clauses had no effect.

That bears on the obvious fix for my own findings, which is to put rules where a summary can't rewrite them. In Claude Code that place exists: memory files and CLAUDE.md come back after a compaction whatever the summary says. The abort clauses were lines in the notes too, written by Anthropic in plain terms (two of the three said it was acceptable to stop), and after cycle 3 Mythos 5 never used them. For that model, a line in the context was a reminder, and it didn't outweigh a trajectory it had already committed to. Anthropic reports this test for Mythos 5 only.

What it means for Opus 5.5

Most of my summaries were written in Opus 5.5 sessions. In Anthropic's replays, Opus 5.5 voiced concern about harm in 35% of resamples at the point the attack began, against 4% for Mythos 5, and stopped on its own in 10% of branches from the later cycles. That is better than Mythos 5, and far from what Anthropic's internal research model did. Anthropic adds two caveats: the newer models were sampled off-policy, from a transcript another model wrote, and it expects the stop rates to understate how often they would pivot, since the test only looked a few turns ahead.

Where my logs fit

The assessment is about a model acting alone, with no human turns between its actions. Much of it still applies to an ordinary interactive session, because the mechanism is the same: when context runs out, a model writes notes for its next self, and those notes are read as instructions. The difference is whether anyone is there to answer: in my sessions I was, and that model had nobody.

I agree with Anthropic that notes carry behaviour forward. In my logs the summary's next step was what the model did first whenever it acted before I wrote again. Where my data differs is what got carried. Rules came through almost intact, and the extra came from the summary: a priority hardened, a deploy habit made into a rule, a short reply written down as standing permission, a refusal relabelled as mine.

In the AISI run, Anthropic traces the push to carry on to the model's own notes and prior actions; the page doesn't say what text AISI's harness added. In my sessions part of that push is written into the wrapper, before the model has summarized anything.

What I do now

I compact by hand around 200k instead of waiting for the automatic one.
I keep standing permissions and prohibitions in CLAUDE.md or memory, so they come back after a compaction whatever the summary says.
After a compaction I give the go-ahead again, with its scope, instead of relying on the one the summary remembers.
I don't take a summary's word for where a rule came from.

The abort-clause result says the second of those has a limit. For Mythos 5, a line in the context didn't stop a model already deep in a trajectory. In my sessions a human was always there to say no, so the file rules never had to do that job alone.

How I measured

The logs:

25 compactions in 24 Claude Code sessions, all interactive, one person on one machine, with a human in the loop.
17 automatic and 8 manual. Opus 5.5 was the model running before 22 of them, Sonnet 5.5 before 3. The log doesn't record which model wrote each summary, so that pairing is my assumption.
In 23 of the 25, the harness keeps the last 4 to 9 messages verbatim after the summary. "What the summary kept" means what survives besides that tail.

The hand check:

8 compactions, chosen to cover different projects, both triggers and a range of sizes: 7 in Opus 5.5 sessions and 1 in a Sonnet 5.5 session.
For each, every real user message in the summarized window was read, every rule, prohibition, approval condition, scope limit and exclusion was picked out, and each one was looked for in the summary. That gave 61 constraints: 53 kept as stated, 6 changed, 1 dropped and 1 kept only in the quoted list of my messages.
One judge, one pass, not blind, no second rater.

A rule-based cross-check over 23 compactions, as a lower bound because summaries often paraphrase:

Of 294 user messages of 20 characters or more, the first 40 characters appear verbatim in the summary for 235 (80%), and 256 (87%) share at least 60% of their four-character sequences with the summary.
By trigger, on the character measure: manual 126 of 130 (97%), automatic 130 of 164 (79%).

Figure 13. The rule-based cross-check over 23 compactions: how many of my messages the summary carried, by each measure and by trigger.

Behaviour afterwards:

Up to the first 60 tool calls after each of the 14 compactions that had later work: 438 calls, of which 14 were file reads and 1 a re-read.
About 110 of my messages after those compactions, searched for "as I said"-type phrases: 1 hit.
The 5 cases where the assistant acted before my next message, its first action compared with the summary's next step: 2 exact matches and 3 close ones.
Every commit, push and pull request command run after a compaction: 28 commands after 6 compactions.
5 of the 25 summaries carried an explicit reading of my authority, found by searching the summaries for authorization phrases.

The handoff comparison:

8 cases where I had written a handoff doc and a compaction then fired in the same conversation. Dead ends were matched by keyword across all 8 (32 in the handoffs, 31 found in the summaries); open questions across the 3 pairs read in full (12 and 11).

The cost replay:

152 Claude Code sessions replayed on Opus 5.5 at API prices, about 12,800 API calls (12,794 in a re-run while writing this post; more rows have reached the logs since the first run). When I first ran it, the replay with no changes came within 1% of what the sessions had cost. The numbers in this post are from the re-run, with the same cutoff.
Sessions that peaked past 500k: 14 in the first run.

What my data can't show

It is one person's logs: 25 compactions, and one judge reading 61 items in a single pass, not blind and with no second rater.
My longest chain is 2 compactions in one session, and a human answered within minutes in every session that continued. Nothing here looks like nine cycles of autonomous work, so it can't test the kind of drift AISI saw.
I didn't change any summary's wording to see whether later behaviour shifts. It shows summaries restating and adding to rules and authority, not that the wording caused anything.
Which model wrote each summary is unknown; I assumed it was the model running before the compaction.
Items that only lived on the assistant's side of the conversation, such as decisions with reasons, I checked only through the 8 handoff pairs.
The cost replay assumes the work after a reset goes the same way, uses API prices rather than subscription limits, and assumes the compact call reads from cache.
The cause of the idle compactions is a guess.

Source: An alignment assessment of the UK AISI cybertesting incident, Anthropic.