Quelris

Blog

What skill usage can tell me.

My Claude Code logs show a narrow set of skills doing most of the loading, while two out of three user-level skills never ran. That tells me what to review, not which skills changed the work.

8 min read

  • AI agents
  • Architecture

Two out of every three skills I added at the user level had no recorded invocation over a month. That is a useful maintenance signal. It is not evidence that those skills failed, and it does not tell me which ones to remove.

The newer Claude Code telemetry described by Anthropic can break usage and estimated cost down by skill, plugin, agent, and MCP server. The same guidance separates tracking from proving: telemetry can show where work and spend went, while source control and tickets show what actually shipped. That distinction matters more than the dashboard.

My own logs had a clear usage pattern and an incomplete value story. A small group carried much of the observed use. Several skills were workflow commands I called deliberately. Other skills were reference material selected by the model. A load shows that something entered the session. It does not show that the task came out better.

A grid of 67 current user-level skills. Twenty-one filled squares represent skills with recorded use; 46 outlined squares form a review queue rather than a removal list.A broad library, narrow usecurrent user-level skillsrecorded usereview queue
  1. Filled squares show skills with at least one recorded invocation in the available window.
  2. Outlined squares are a review queue, not an automatic removal list.
Figure 1. Current user-level skills with recorded use beside the larger review queue with no recorded use.

Usage is a maintenance signal

The first useful question is simple: what has actually been loaded? In my current user-level library, two in three skills had no recorded invocation in the available window. Five observed skill names made up about two fifths of all recorded skill calls.

Those findings describe different populations, so they need to stay separate. The idle result concerns current user-level skills. The concentration result includes every observed name in the logs, including plugin, project, historical, and deleted entries. Combining them into one "long tail" would make the numbers sound cleaner than they are.

Distribution of 67 current user-level skills: 46 never used, 8 used once, 4 used two to five times, and 9 used six or more times. The never row is highlighted.Current user-level skillsOne library, 67 skills.never46once82–5 times46+ times9All observed skill names in the logs: the top five account for 102 of 244 invocations and every other observed name accounts for 142. This is distinct from the current user-level library.All observed namesA different population: 244 calls.top five102 of 244everything else142Includes current, plugin, project,historical, and deleted names.
  1. The left panel is 67 current user-level skills; the right panel is 244 observed invocations across every observed name.
  2. The highlighted rows show 46 never-used current skills and 102 calls from the top five observed names.
Figure 2. The current user-level library and all observed names have different denominators, so their usage patterns stay in separate panels.

The useful conclusion is narrower. Usage tells me where to start reviewing:

  • A skill with no recorded use may be obsolete, recently added, seasonal, or reserved for an unusual incident.
  • A skill that runs often may be a genuine force multiplier, or it may simply be attached to a routine the model repeats.
  • A crowded catalog gives the model more descriptions to choose between, but the local transcripts do not preserve the full historical list, so I cannot turn that into a measured bill.

An idle skill can still earn its place. Installing one is easy, and a rarely needed procedure may have no obvious moment when it announces that it is no longer useful. That is why zero use belongs in a review queue, not an automatic delete queue.

A review queue begins with no recorded use, then lists four possible paths: recently added, seasonal or rare, rare high-stakes use, and duplicate of another procedure. The final actions are watch next window, keep a named situation, keep, and merge or remove.Review paths for no recorded useno recorded userecently addedwatch next windowseasonal or rarekeep a named situationrare high-stakeskeepduplicate of anothermerge or remove
  1. No recorded use can mean recently added, seasonal or rare, rare high-stakes, or duplicate of another procedure.
  2. Only the duplicate path reaches merge or remove; the others keep context for another review.
Figure 3. Four review paths for a skill with no recorded use, from a recent addition to a duplicate procedure.

A call is not one kind of use

My most-used skills were not all doing the same job. Two of the three leaders were typed by hand almost every time: one moves work into a handoff, and one helps pick it back up. They are closer to shortcuts for workflow moments I already recognize than to expert reference material the model chooses for itself.

The model-selected loads gathered around design references and browser work. That is a different product. A hand-typed command is an explicit choice by the person at the keyboard. A model-selected skill is a routing decision inside the task. Counting both as "fired" is useful for inventory work only.

There is another split before any cost discussion. A typed command can be hidden from model invocation. When it is, the model does not see its listing and cannot call it on its own. In my setup, the two hand-typed workflow skills are configured that way. So is one idle skill.

That makes two measurements worth keeping apart:

  • For commands I type, measure whether the shortcut helps me resume, hand off, or finish a repeated workflow.
  • For skills the model can pick, measure selection, full-load use, and the task result after selection.

A single usage ranking flattens those into one list. It is still a useful list, but it is not a product decision.

Five anonymous high-use labels shown as stacked bars split between typed-by-hand and model-picked calls. A has 27 typed and 2 model-picked calls, B 10 and 13, C 22 and 0, D 6 and 9, and E 0 and 13. A and C are hidden from the model's list.A call has two sourcesFive high-use labels, kept anonymous.typed by handmodel-pickedA27 + 2hidden from the model's listB10 + 13C22 + 0hidden from the model's listD6 + 9E0 + 13
  1. A and C stay hidden from the model's list, so their typed calls are not model routing decisions.
  2. The accent segments are model-picked calls; the remaining segments are typed by hand.
Figure 4. Five anonymous high-use labels split into typed-by-hand and model-picked calls.

The dashboard stops before the outcome

Anthropic's sequence is practical: track usage and cost by skill, then connect them to finished work. Its examples put impact against a baseline and ask teams to count a defined unit where it lands. Tokens, lines changed, commits, and pull requests are useful telemetry, but they do not establish ROI by themselves.

That leaves a gap for a specific skill. A dashboard can show that a skill loaded, how often it loaded, and what the surrounding session cost. It cannot show whether that procedure changed the result. A call might have led to a stronger answer, a longer answer, a slower answer, or no meaningful difference at all.

The closest outcome check I have is an earlier review evaluation, described in Same checklist, three agent designs. The skill-loaded review design landed about level with the one-sentence baseline on recall. It did not isolate the effect of skills, because I changed the review method at the same time as moving the playbooks into skills. The result is a warning against reading "loaded" as "paid," not a verdict that skills made the review worse.

The miss pattern is still useful. Most misses had a relevant procedure available, but the run did not apply it. That is the failure a usage dashboard hides. It records the skill that entered context, not the procedure the model never selected when it mattered.

Two columns compare telemetry and an outcome evaluation. Telemetry records skill loaded, how often, and session cost, but leaves the question did it change the result empty and highlighted. An outcome evaluation records final result versus baseline, available but unapplied procedures, and review recall of 61 percent versus 63 percent, with method changed at the same time.What each measurement can recordskill loadedrecordedhow oftenrecordedsession costrecordeddid it changethe result?final resultvs baselineprocedureavailable, not appliedreview recall61% vs 63%method changedat the same time
  1. The highlighted telemetry cell is the question usage data leaves open: did it change the result?
  2. The earlier review comparison was about level with its baseline, but the method changed at the same time.
Figure 5. Telemetry records activity around a skill; an outcome evaluation adds the comparison that can answer whether the result changed.

My read: skills need their own unit of work

Usage and outcome are different layers of the system. The first keeps the library maintainable. The second decides whether a procedure is worth carrying.

For a skill that the model chooses, I would want an evaluation built around one task type:

  1. Define the finished result and how it will be judged.
  2. Run a baseline without the skill. For coverage work, that baseline should include a one-big-prompt version, because it exposes every procedure from the start.
  3. Run the same task with the skill available, more than once.
  4. Check selection as well as the final result. If the skill never loads, the procedure was unavailable in practice. If it loads and the result does not improve, the skill may be unnecessary, poorly scoped, or ignored after loading.
  5. Keep the result and the selection trace together. One without the other makes diagnosis hard.

An evaluation takes more effort than usage reporting, but it answers the question people usually mean when they ask whether a skill pays rent.

There is a practical middle ground before that eval exists. I can keep the model-visible library small enough to review, preserve rare high-stakes procedures, and treat typed workflow commands as commands. Then I can choose a few common task types and give their skills a baseline. That is slower than pruning by count, but it does not confuse absence of evidence with evidence of no value.

Five compact, numbered log-like rows: define result and its judgment rule; baseline without skill including a one-big-prompt coverage baseline; repeat runs with the skill available; check both selection trace and final result; and keep trace with result.A unit of work for evaluation1define resultset the judgment rule2baselinewithout skill; one-big-prompt for coverage3repeat runswith skill available4check trace + resultselection and final result5keep bothtrace with result
  1. The baseline includes a one-big-prompt version for coverage work, before the skill is made available.
  2. Selection and result stay together, because either one alone leaves the diagnosis incomplete.
Figure 6. A compact evaluation strip that keeps the baseline, repeated runs, selection trace, and final result in one unit of work.

How I measured

I parsed one developer's retained Claude Code transcripts over a month. The current inventory contained 67 user-level skills. The scan covered 807 top-level sessions, and I hand-checked 10 invocations for follow-through; a separate earlier review evaluation used 12 runs. The usage parser found 244 distinct skill invocations after deduplication.

The counts needed deduplication because copied and resumed transcript files replayed the same records. A first pass counted 266 invocations. Deduping model calls by tool-use ID and typed command calls by message ID reduced that to 244, an overcount of about 9% relative to the final count. Without that step, copied transcripts would look like independent use.

The inventory and logs have limits:

  • The 46 idle skills are 46 of 67 current user-level skills, not every plugin or project skill available to the setup.
  • The 102 of 244 concentration result is a different population: all observed skill names, including names that no longer exist in the current inventory.
  • The inventory is current state while the sessions are older records. Renamed and deleted skills can be absent from the current library.
  • The listing-size estimate is only a characters-divided-by-four proxy. The transcripts do not capture the complete historical listing, cache behavior, actual tokens, or a bill.
  • Ten hand-checked invocations showed whether a procedure was followed in the immediate transcript. They did not establish completion, time saved, causality, or quality.
  • The review comparison changed the review method as well as instruction placement. It is not a clean isolated test of skills.
Three transcript rows labelled original transcript, resumed copy, and subagent copy carry the same tool-use ID and collapse into one invocation. The first pass counted 266 invocations; identity dedupe counted 244, about 9 percent lower.Copied rows are not new callsoriginal transcripttool-use IDresumed copytool-use IDsubagent copytool-use ID1first pass266→deduped244about 9% overcount before identity dedupe
  1. Original, resumed, and subagent copies can carry the same tool-use ID and count as one invocation.
  2. The first pass found 266 rows; identity dedupe produced 244 invocations, about 9% lower.
Figure 7. Copied transcript rows can replay the same invocation, so stable identifiers collapse them before the usage count is reported.

What I do with the result

I use usage to maintain the catalog, not to declare winners. Skills with no recorded use go into a review queue. I keep a rare procedure when I can name the situation it is for. I keep the hand-typed workflow commands outside the model's choice list when they are really shortcuts for my own decisions.

For skills that are meant to improve a repeated task, I want an outcome measure before I call them valuable: the same task, a baseline, repeated runs, and a result that beats the baseline. Until then, usage and cost remain telemetry. They tell me where to look next.

Source: Manage costs and demonstrate ROI for Claude Code on Google Cloud, Anthropic.