Quelris

Blog

What an agent's all-pass report covered.

My pre-publish agent opened its report with "all pass" and noted further down that it never clicked a link. Both lines were true. A pass covers the assertions that ran, so coverage and "not run" belong next to the verdict.

7 min read

  • AI agents
  • Architecture

The first line of my pre-publish agent's report read: "all pass, nothing blocks publishing." Line 23 read: "I did not click the anchors or share links."

Both lines were true. The test suite, the strict build, the meta and Open Graph tags, the figure checks, and a layout assertion at desktop and phone width all passed. The agent did not hide anything either. The gap was in the report, written plainly, about halfway down and under a verdict that reads as if it covers the whole page.

Whether an agent can execute the app is no longer the interesting question. Mine already opens a browser, resizes the viewport, runs code against the DOM, and takes screenshots before a post goes out. The harder question is what a given pass is a claim about. A green result says the assertions that executed held. It says nothing about the ones that never ran.

Running the app does catch things a diff review can miss. Load a page in a phone-sized viewport and you can see a button that disappears at that width. The question I care about comes after that: once something runs, how do you tell what its pass covered?

The quoted pre-publish report drawn as 51 rows, each bar as long as the line it stands for. Line 1, the all-pass verdict, and line 23, saying the anchors and share links were not clicked, are marked in the accent color; the report ends at line 51.One report, 51 linesline 1: the verdictall pass, nothing blockspublishingline 23: the scopedid not click the anchorsor share linksline 51: end of report
  1. Line 1 opens with the verdict for the whole run.
  2. Line 23 is the only line about something the agent did not do.
Figure 1. The quoted report as 51 rows drawn to their real line lengths, with the verdict on line 1 and the scope line on line 23.

What "all pass" was made of

The report listed its checks one by one, each with a result and the command or URL behind it. Reading it as a list rather than a verdict, the passes fall into different kinds of evidence:

  • Commands with exit codes. The test suite passed with no failures, and the strict build finished and wrote the page, its Markdown copy and its preview image.
  • HTTP responses. The post, the blog index, the feed, the sitemap and the preview image all returned 200 from the local preview server.
  • Static reads of built HTML. Title, description, canonical URL, og:image and the card type were read out of the generated file and compared with what they should be.
  • DOM measurements in a real browser. At 1280 and 390 pixels wide, in light and dark, the page's scrollWidth equaled its clientWidth, so nothing overflowed sideways. Every figure rendered and nothing in the page returned an error status.
  • Markup checks on links. The post's one outbound link returned 200, the in-page anchors were described as fine, and the share links as well-formed.

Then came the one line that was not a pass: the agent did not click the anchors or the share links.

So the "Links: PASS" row rests on one live request and two descriptions of markup. The behavior a reader would actually hit, clicking an anchor and landing on the right heading, or opening a share link and getting the right prefilled text, was never exercised. Nothing I know of was broken. The row's label is just broader than its evidence.

Six rows from the report. Commands passed on exit codes, HTTP checks on status 200, built HTML on values read from the file, the browser DOM on measured widths, and links on described markup. The last row, clicks, was never exercised and is marked not run.What each pass rested oncheckevidenceresultcommandsexit codespassHTTPstatus 200passbuilt HTMLvalues read from filepassbrowser DOMwidths measuredpasslinksmarkup describedpassclicksnever exercisednot run
  1. Five kinds of evidence carried the same word: pass.
  2. Clicking an anchor or a share link was never exercised.
Figure 2. Every pass in the report, grouped by the evidence behind it, with the one check that never ran.

A layout assertion has a narrow job

The phone-width check is the clearest case of a pass that sounds bigger than it is. The agent set the viewport to 390 pixels and compared the document's scroll width with the viewport width. Equal widths mean no horizontal scroll. That is a useful assertion, and it held.

It also could not catch a button that disappears at phone width. A button set to display: none by a media query at small widths takes up no space. The page gets no wider, scrollWidth doesn't move, and the overflow assertion stays green. The same goes for a button that renders but sits under another element, or one whose click handler throws. A layout assertion answers whether the page fits. Whether the button is there, visible, reachable and does what it should are separate questions, and each needs its own check.

Illustration: a 390 pixel wide page with a visible Sign up button. The scroll width equals the viewport width, so the overflow assertion passes.Button visibleIllustration at a 390px viewport.Sign upscrollWidth 390 = viewport 390fits: passIllustration: the same page with the button hidden by a media query. The page gets no wider, the scroll width still equals the viewport width, and the overflow assertion still passes.Button hidden at 390pxSame page, same assertion result.scrollWidth 390 = viewport 390fits: pass
  1. A button hidden by a media query takes up no width, so the page doesn't get wider.
  2. Both pages measure the same and both pass.
Figure 3. An illustration of why an overflow assertion can't see a button hidden at phone width.

The agent did handle one thing the other way round, and handled it well. Two things looked wrong in its screenshots: a heading that seemed clipped, and a figure caption that seemed cut off. It checked both in the DOM. The heading was still mid-animation in the first capture and rendered fully on two lines a few seconds later. The caption's scrollHeight equaled its clientHeight, so nothing was clipped. It preferred a measurement over an impression, which is right for clipping. It also means the word "pass" in that report mostly meant DOM numbers, not a look at the page.

Trust checks you have seen go red

A green result carries more weight once you have seen the same check go red when it should.

I have one check where I've watched both happen: a two-account authorization replay I run against synthetic test apps. Each app starts from a clean database seeded with two accounts. Account A's record IDs are collected first. Then account B requests A's resources on a fixed list of approved routes, and the replay compares fields to see whether B got A's data.

On the vulnerable versions, every planned finding was confirmed: B received A's record with matching fields. On the patched version, none were. B was refused with a 404, or got records that weren't A's. The negative controls, such as a request for a user ID that doesn't exist, came back blocked as expected.

Three separate groups. Vulnerable versions: six planned findings, all six confirmed, meaning account B got account A's record. Patched version: two findings, none confirmed. Negative controls: two requests, both blocked as expected.Two-account replay resultsvulnerable versions6 of 6 confirmedpatched version0 of 2 confirmednegative controls2 of 2 blocked, as expectedB got A's recordblocked
  1. Vulnerable versions: every planned finding confirmed.
  2. Patched version: none confirmed. Negative controls: both blocked as expected.
Figure 4. Two-account replay results, with each population in its own group.

That spread is what makes the green on the patched version mean something. The check has returned red on code known to be exploitable and green on code known to be fixed. It is the same idea as mutation testing: if you inject a known fault and the suite stays green, the suite isn't catching that fault.

The replay also reports the way I want every check to report. Each result comes with its reason: "B was refused with 404", "B got 2 record(s), none of them A's", "B received A's id with 3 identical field(s)". A reader can see what the verdict rests on without opening the logs.

My read

1. Coverage before verdict

"All pass" is an aggregate, and aggregates hide their denominators. It does not say how many checks there were or what kind they were. And a check that never ran doesn't appear in the count at all. The lines that tell you the most are the ones that say what wasn't done. In my report that was line 23.

2. One word, different evidence

The same "pass" covered an exit code, an HTTP status, a DOM measurement and a description of markup. They don't carry equal weight. A 200 from a link says the URL responds. A well-formed share link says the URL is shaped correctly. Neither says what happens when someone clicks. A report that names the evidence type next to each pass makes the difference visible without rereading the method.

3. "Not run" is a status, not a sentence

Test runners already do this. A skipped test shows up as skipped, counted next to passed and failed, not hidden in a log. An agent's report deserves the same shape. Here is how the rows from my report would read:

Report rows with an explicit not-run status
layout: pass. scrollWidth equals clientWidth at 1280 and 390, light and dark.
meta: pass. Title, description, canonical and og:image read from built HTML.
links: pass. Outbound link returned 200; share URLs well-formed.
links: not run. Anchor clicks and share-link clicks.
A report shaped as one all-pass line followed by paragraphs of prose. The line saying links were not clicked is one sentence among the others.Verdict firstOne summary line, then prose.all passdid not clickThe gap is a sentence inside the prose.The same report as rows with a status column: layout pass, meta pass, links pass, and links not run, so the gap is in the same column as the passes.Status per rowEach check carries its own result.checkstatuslayoutpassmetapasslinkspasslinksnot runNot run sits in the status column.
  1. On the left, the gap is one sentence among many.
  2. On the right, not run sits in the same column as pass.
Figure 5. The same findings as a verdict with prose, and as rows with a status column.

4. Running the app is the start

Some problems only appear when the app runs, which is why my pre-publish checks include a 390-pixel viewport. Running the app makes a pass possible. It doesn't decide what that pass covered. A workflow that simulates a user moving through the app covers more than my overflow check did, and its report still needs to say which flows it walked and which it didn't.

Four questions about a button, from page fits the viewport to element is visible, element is reachable and click does the right thing. The overflow assertion answers only the first; each of the other three needs its own check.What a layout assertion can seepage fits the viewportoverflow assertionelement is visibleneeds its own checkelement is reachableneeds its own checkclick does the right thingneeds its own check
  1. Visibility, reachability and click behavior each need a check of their own.
Figure 6. Four questions about a button. An overflow assertion answers only the first.

How I measured

  • The report. Three pre-publish runs on one earlier post on this blog, all by the same agent. All three reports opened with "all pass". The one I quote is 51 lines long, and the not-clicked line is line 23. I read the agents' tool calls from the session transcripts and removed duplicate rows by call ID. The calls were viewport resizes, color-scheme emulation, page navigations, DOM evaluations, screenshots and short code runs. None clicked, tapped or pressed anything, and the code runs only resized, navigated, scrolled, measured and captured.
  • The layout assertion. In the quoted run, the code set the viewport size, then compared the document's scroll width with the window width. The report states the result as scrollWidth equal to clientWidth at 1280 and 390, light and dark.
  • The replay. Local synthetic apps, each started from a clean database and seeded with two test accounts. The routes come from a hand-written plan of approved read requests, one per finding. A result counts as confirmed when account B gets account A's record with matching fields. Response bodies are not stored, only the status, size, field-level equality and the reason. Across the runs there were six planned findings on vulnerable versions (all six confirmed), two on one patched version (none confirmed), and two negative controls (both blocked as expected).