The Tool That Was Never There: Two Ways an Agent Misses a Tool, Only One Emits an Error

Sixteen events, six names

I went back to a thirty-day session corpus expecting to confirm a count I had already written down. The count was 34. After I deduped by unique tool_use_id, it was 16.

That is the finding. In 1,019 Claude Code session files, 615,198,221 bytes, from 2026-08-04T18:32:11 to 2026-09-03T18:32:12, sixteen distinct tool-call events resolved to a tool that was not there. Six names. Write five times. Grep three. Read three. ToolSearch three. Glob once. Bash once.

Two scoping notes, because the failure surfaces are not the same and I am not going to average them. First: all sixteen of those events are structured errors. Every one carries is_error: true and arrives wrapped in <tool_use_error>. A loop keyed on tool errors will see all sixteen. Second: two of the sixteen carry a trailing clause saying the tool exists but is disabled in that context, which is a different state with a different fix. Post 63 is about those two.

The specimen I quote below is a third surface, and the one that worries me. There the model never emitted a bound call at all. It answered in prose about a tool it did not have. No tool_use_error exists for that turn, because no tool call was ever dispatched. That reply does not appear in the sixteen. It is not counted, and I am not going to count it, because it is a different kind of object.

I am not going to invent which sixteen they were, or dress the haiku reply as a stack trace it never produced. The corpus names a window, a dedup method, and a distribution. Window: those 1,019 files. Method: unique tool_use_id for tool_result blocks, message id for text blocks. Distribution: the six names above. Manifest at .planning/mine-30d-exact-manifest.json. Stats at .planning/mine-30d-exact-stats.json.

This is post 61. The subject is not a missing install on my machine. The subject is the shape of the miss. A runtime that lacks a name still produces a reply. The reply is fluent. The loop continues.

The reply that was not an exception

The cleanest specimen is a haiku session under withagents-forge. Model claude-haiku-4-5-20251001. File .claude/projects/-Users-nick-dev-withagents-forge/0e9b0f78-4468-4a78-951a-d4c00c4e5b42.jsonl, line 15. The agent was pointed at fv_probe. It did not throw. It wrote this:

I don't have access to an `fv_probe` tool. Looking at my available tools, I have:

- **Agent** — spawn subagents
- **SendMessage** — message other agents
- **TaskStop** — kill background tasks
- **Workflow** — run orchestration scripts

The `fv_probe` tool is not in my toolkit. I cannot call it to retrieve the secret phrase.

The pack records that the same response ends by offering to help connect the tool. I am not going to invent that closing sentence. The cited block stops at "I cannot call it to retrieve the secret phrase." The load-bearing fact is the genre. This is a helpful inventory. It lists Agent, SendMessage, TaskStop, Workflow. It names the missing tool. It declines the secret phrase. Then, per the pack, it offers to help wire the tool in.

That is not an exception. It reads as cooperation.

A supervisor that inspects this turn for a tool error finds nothing to inspect. There is no tool_result in it. The child answered, listed tools, and volunteered a next step. I did not observe what any parent did with that reply, and I am not going to claim one was fooled. The structural point stands on its own: an error-keyed check has no object here.

The other named specimens in the window are the counted kind, not this kind. Write at -Users-nick-Desktop-yt-transition-shorts-detector/1f17b872-…jsonl:6492. ToolSearch at -Users-nick-dev-blog-series/c2299611-…jsonl:3607. Bash at -Users-nick-proofpunk/c6a8d0db-…jsonl:20. Those are tool_result blocks with is_error: true. They are members of the sixteen. The haiku block is not, and that difference is the subject of this post.

Notice what the haiku did instead of dispatching. It enumerated the toolkit it did have. That list is true, as far as the corpus shows. Truth is not the same as dispatch. A description of Agent and Workflow is not a call to fv_probe. It is a paragraph standing where a call would have been. If you are a human reading the session, the paragraph is useful. If you are a supervisor keyed on tool errors, the paragraph is invisible, because there is no tool event under it to inspect.

Thirty-four was the replay

Raw byte-level regex on the same string produced 34 hits. Eighteen of those hits are the same events playing back. Transcripts in this corpus replay a tool use across lines. A grep that counts string occurrences counts the replay as new work.

The honest number is 16. Dedup key: unique tool_use_id on tool_result, unique message id on text blocks. That is the method the stats file uses for every other count in this post. Using 34 here and unique-id counts for Bash would mix two denominators. I will not do that.

Why the replay exists is not mysterious. A session jsonl is a conversation record, not a log of unique acts. The same tool_use_id can appear in the assistant message that issued the call, in the tool_result that answered it, in a later summary, in a resumed context. Byte-level search is the right instrument for proving a string exists and the wrong one for counting incidents.

The 18 extra hits are not 18 extra failures. They are 18 extra copies. Reporting 34 would have overstated the incident rate by more than two times, and would have made the scale comparison below a lie. Sixteen is the number to trust.

I want this correction in the body, not a footnote, because counting discipline is how this class stays honest. A scare number travels. A rate does not, unless the denominator is the same kind of object as the numerator. Events against events. Never raw bytes dressed as events.

This is the same family of instrument error as a vacuous pass, pointed at the counter instead of the gate. A count that cannot distinguish an event from its echo is a caption printed next to the corpus. It is not a measurement of the corpus. I already had 34 written down. The corpus did not owe me that number. The regex owed me that number. The regex was doing what regex does.

One in thirteen hundred

Same window. Same dedup method. The five largest categories of unique tool call: Bash 17,719. Read 1,776. Edit 755. Agent 321. Write 285. Those five sum to 20,856. Every tool name in the stats file, not just the top five, sums to 21,782.

Sixteen absent-tool events is 1 in 1,304 against the five-category subtotal, and 1 in 1,361 against all 21,782. I am quoting both denominators because a rate is only meaningful next to the set it was drawn from, and "1 in 1,300" is the five-category figure, not the all-calls figure. The difference does not change the argument. It changes whether the number is checkable. Rare enough that a dashboard of call volume will never surface it. Frequent enough that a thirty-day window of 1,019 files produced sixteen of them, across six names, without anyone treating the miss as a crash.

Write is the mode of the miss: 5 of 16. Grep, Read, and ToolSearch sit at 3 each. Glob once. Bash once. I am not going to invent a story about why Write leads. The pack does not say. Write also appears in the scale table at 285 unique calls, which only proves that Write was present for other events in the same window. Presence in the corpus is not presence on a given turn. An agent can have Write in one session and not in another. An agent can name Write and still emit a call the runtime does not bind. The sixteen events are the ones where the resolution was: not there.

Bash at 17,719 unique calls and Bash at 1 absent-tool event is the same name in two states. The name is not the proof of the mount. The resolution is. I will keep that sentence next to the distribution. A volume table that ranks Bash first can still hide a Bash the runtime declined to bind on one turn, because one event does not move 17,719.

One in 1,300 is not a reason to ignore the class. It is a reason the class stays invisible on a volume dashboard. Volume dashboards rank Bash, Read, Edit. They do not rank "called a name the runtime did not have." Sixteen events will not spike anything. But they will alert, if you are keyed on tool errors, because all sixteen are tool errors. That is the good news in this post, and I nearly wrote the opposite.

I am not claiming sixteen is the full population of every way an agent can fail to invoke a tool. The taxonomy inside the sixteen is: 14 unavailable, 2 withheld, where withheld means the result carried the clause saying the tool exists but is not enabled in that context. All sixteen are tool_result blocks with is_error: true. What the sixteen do not include is the third surface: a turn where no call was dispatched at all, so no tool_result exists to count. The fv_probe reply is that surface. It is uncounted here by construction, not by oversight.

Two surfaces, one of which a loop can see

I set out to write that a missing tool fails as prose. The corpus refused that thesis. Here is what it actually holds.

Surface one is the sixteen. A call is dispatched. The runtime declines to bind the name. A tool_result comes back with is_error: true, wrapped in <tool_use_error>. This is machine-detectable. A supervisor keyed on tool errors catches every one of the sixteen. There is nothing subtle here, and I am glad to be wrong about it.

Surface two is the fv_probe reply. No call was dispatched. The model wrote a paragraph about a tool it did not have, listed the four tools it did have, and offered to help connect the missing one. There is no tool_result for that turn, no is_error field, nothing for an error-keyed supervisor to catch. The turn looks like a normal assistant response, because that is exactly what it is.

The two surfaces need different detectors. Surface one needs error handling you already have. Surface two needs something else: a check that the turn you expected to contain a call actually contains one. I have drawn surface two in crimson because it is the one that passes a loop keyed on errors.

The left column is what every orchestrator I have written assumes, and on this corpus that assumption holds. Sixteen for sixteen. Tool missing, error raised, loop can halt.

The right column is the one I cannot detect with the same check. The child did not error, because the child did not call. It wrote. The write looks like work.

I do not have, from this corpus, a count of how many fv_probe-shaped turns exist. I found one and I am reporting one. Counting that class needs a different query than the one I ran: not "which tool_results carry an error" but "which turns were supposed to contain a call and do not." I have not written that query. I am not going to imply I have.

The offer to help is the sharp edge on surface two. A structured error is socially complete. Nobody retries a tool_use_error by being polite back. An offer is socially open. A parent that is itself an agent can accept the offer and spend the next turn trying to connect fv_probe, or spawn a sibling that also lacks it, or write a plan about mounting it. All of that looks like throughput. None of it is the probe.

What I will count next time

If I need an incident count out of session jsonl, I will count unique tool_use_id, not raw occurrences. The 34-to-16 correction is not a footnote. It is the difference between a scare and a rate. Sixteen in thirty days, against 17,719 unique Bash calls in the same files, is a rare miss, and every one of the sixteen carried machine-readable error markers. Thirty-four in the same window, counted the naive way, is a fake doubling plus a fake sense of precision.

If I need to know whether a child actually invoked a tool, I will not take a fluent reply as evidence of a call. The haiku listed four tools it did have. That list is not a call to fv_probe. It is a description of the absence of fv_probe. Description is not dispatch.

If I write a loop that retries on failure, error handling covers the sixteen already. What it does not cover is the turn with no call in it. That needs a different check: assert the turn contains a tool_use block when the plan said it would, and treat its absence as a failure even though nothing errored.

The three named files are the ones I can point at without guessing: Write at 1f17b872-…jsonl:6492, ToolSearch at c2299611-…jsonl:3607, Bash at c6a8d0db-…jsonl:20. The quoted body is line 15 of 0e9b0f78-4468-4a78-951a-d4c00c4e5b42.jsonl. The window is 2026-08-04T18:32:11 to 2026-09-03T18:32:12. Files: 1,019. Bytes: 615,198,221. Events: 16. Names: 6. Raw string hits I will not report as events: 34.

A checklist I will keep next to the next corpus pass is short.

Count unique ids, then report the unique number, then name the raw number as the wrong one. Sixteen, not 34. The raw figure is how you catch yourself.

Keep the denominator in the same units. 17,719 unique Bash calls belong next to 16 unique absences. They do not belong next to 34 string hits.

Treat a toolkit inventory as evidence of absence, not as evidence of work. Agent, SendMessage, TaskStop, Workflow were the haiku's available set. That set does not contain fv_probe. Listing it does not retrieve a secret phrase.

Require the call, not just the absence of an error. If a turn was supposed to invoke a tool, assert that a tool_use block is present in it. That check is the one that would have caught the fv_probe turn, and it is the check I did not have.

Cite the window. 1,019 files, 615,198,221 bytes, 2026-08-04T18:32:11 to 2026-09-03T18:32:12. A rate without a window is a caption.

What this corpus actually establishes is narrower than the post I meant to write, and more useful. Sixteen unavailable-tool events, over thirty days and 1,019 files, every one of them a tool_result carrying is_error: true inside <tool_use_error>. Fourteen unavailable, two withheld. Those are detectable with error handling most orchestrators already have, and I have no evidence any of them slipped past a supervisor.

Separately, one assistant turn answered in prose about a tool it did not have, and produced no tool event at all. I am not going to tell you that turn fooled a parent, because I did not measure that. What I can say is structural: there was nothing there for an error-keyed check to find. The remedy is not better error handling. It is requiring the expected tool_use block and treating its absence as a failure.

I set out to write that absent tools fail as fluent prose. Sixteen of sixteen say otherwise. The correction cost me the thesis and left me the better rule.