Distinguishing Model Errors From Tool Errors in Agent Traces

When an agent trace shows a failure, the visible symptom tells you almost nothing about where to intervene. A wrong final answer can originate in the model's reasoning, in a tool's execution, or in the harness that connects them, and each demands a different fix. The diagnostic work that matters is assigning fault side before anyone ships a patch.

Why reading a trace as a single failure hides the fix

Two engineers look at the same trace. One sees the model hallucinating a figure the tool never returned, and files a prompt revision. The other sees a tool silently returning a truncated payload after an HTTP 200, and files a schema guard. Neither patch will close the other's regression, and the incident will recur because only one diagnosis can be correct.

This is the structural problem with reading a trace as a single failure. Agent behavior is not produced by one component. It emerges from interactions among the model, the tool layer, the harness, memory, and the environment. A defect anywhere in that chain can surface as the same wrong output at the end. In traditional software, a failure is local and legible: a stack trace points to a line number, and the fix is bounded. In agents, any reasoning step can degrade silently, and each step conditions the next. A small early error compounds into a completely wrong outcome with no exception raised. Standard infrastructure monitoring, which tracks uptime, latency, and HTTP status codes, misses this class of failure entirely. The diagnostic signal lives in output quality measured against original intent, not in status codes. An agent can return 200 on every tool call and still produce a catastrophically wrong answer for the user.

How agent failures are structured across architectural layers

The first thing an engineer confronting an agent failure needs is a reason to believe the problem is tractable. Failures in agentic systems are structured rather than ad hoc, and they propagate across architectural boundaries in repeatable ways that can be systematically exploited for diagnosis. Shah et al.'s empirical taxonomy, built from thousands of faults in open-source agentic repositories, produced five architectural fault dimensions, thirteen symptom classes, and twelve root cause categories. Practitioners rated the taxonomy as highly representative of real-world failures, and a large majority reported it covered faults they had personally encountered.

The key finding from that work is that faults frequently traverse architectural boundaries. A defect in token management logic can propagate to authentication failures. A datetime-handling fault can surface as a scheduling anomaly. A state management defect can present as a memory symptom. The component where the failure surfaces is often not the component where it originated. Five categories is right because three collapses planner and tool errors into the same bucket, which destroys fix routing, but more granular taxonomies turn incident review into a labeling argument. The model-versus-tool distinction cuts across several of these categories, and it is where the real diagnostic work happens.

Why model errors and tool errors demand separate fixes

Attributing every agent failure to the model misses the cases that only a harness or tool integration fix can resolve. The converse is equally costly. Shah et al. found that failures in agentic AI show a distinctive hybrid profile: conventional software engineering faults combined with probabilistic LLM-driven behavior. These are not the same class of problem, and they do not respond to the same class of fix.

Model-side faults, which include bad decisions, hallucinated outputs, ignored results, and incorrect planning, are targets for post-training, system prompt revision, or planner-side evaluations. Harness-side and tool-side faults, which include malformed responses, schema drift, silent HTTP 200s, and missing error handling, point toward scaffolding fixes, schema guards, retry policies, and tool integration redesign. Environment faults reveal conditions that must be redesigned at the benchmark or test harness level. Neither model nor tool changes will resolve them. That requires knowing what model errors and tool errors look like as observable span artifacts.

What model errors look like inside a trace

Model errors leave distinctive fingerprints in the reasoning and planning spans of a trace, and they are separable from tool-layer artifacts if the right spans are scored. Tool-Skip, or TS, is when the model does not produce a valid executed tool call. The plan span names a tool, but no corresponding tool span fires. This is a planning or instruction-following failure, not a tool failure. Result-Ignore, or RI, occurs when the model calls the tool correctly and the tool returns a valid result, but the reasoning span that follows does not incorporate the returned data. The model reasons from its prior rather than the observed return. Output-Fabrication, or OF, occurs when the model calls the tool, receives a result, and then adds invented structured information not present in that return. This is hallucination on grounded context.

These three modes require different fixes even though all are model-side. Result-Ignore is fixed at the reasoning layer with a rubric that scores whether the model used the returned value, calibrated against held-out cases where the tool return contradicts what the model likely expected. Planning-layer model errors carry their own fingerprints. Agents without defined exit conditions oscillate between infinite retry loops that exhaust token budgets and paralysis where no output fires and no signal appears. Both states are observable in span telemetry, and both are model-side failures: the model fails to recognize a stopping condition.

What tool errors look like inside a trace

Tool errors produce a different class of trace artifact. The model's reasoning and planning spans may look entirely correct while the fault lives in the tool-output span or in the tool's response to a valid call. Four primary tool-side failure types make up the practical landscape. The tool-input span is well-formed from the model's perspective. The fault is in the schema contract.

Hallucinated tool invocations occur when the agent invokes a tool not in its registered set. The plan span names a function that does not exist at runtime. Silent bad returns are the most damaging failure mode because they produce no signal. The tool returns HTTP 200 with an empty payload, a truncated schema, or internally inconsistent data. No error surfaces for review. The model receives what looks like a valid response and proceeds on corrupted input. Missing error handling occurs when the tool returns a 500 and the agent fabricates a reasonable-sounding response on top of the failure. Error propagation amplifies the problem. In a multi-step agent, a malformed tool output at step two becomes the input to step three's reasoning. By the time the agent returns a final output, the original error may be several layers deep and invisible in the response. What began as a hallucinated output passed downstream becomes corrupted input for every subsequent reasoning step, compounding across layers before surfacing.

Walking the span tree to assign fault side

To assign fault side, you walk the span hierarchy in a defined order, score each span independently, and let the first bad span determine where the fault originated. The span hierarchy for agent debugging runs from root to agent to LLM to tool to retriever to embedding spans. The debugging procedure walks the span tree from input to plan to retrieval to tool args to tool output to LLM rewrite to final answer, scoring each span to find the first bad one.

Begin with the plan span. Does the emitted tool sequence match the executed sequence? A divergence here is model-side, a planning error, regardless of what the tool spans show. Then score each tool-input span. Does the argument shape match the tool's registered schema? A mismatch with a correctly formed plan span is tool-layer, either schema violation or catalog drift. Next, score each tool-output span. Does the return contain the expected payload? An HTTP 200 with empty or truncated data is a silent harness failure. A 500 with no error path is a missing-error-handling fault. Then score the reasoning span after each tool call. Does the model's next step incorporate the tool's return, or does it reason from prior? Incorporation failure with a valid tool-output span is model-side, a Result-Ignore. Incorporation of the return with added information not in the return is also model-side, an Output-Fabrication. Finally, check for loop signatures. Does the same tool with similar arguments fire repeatedly with no goal-state advance? The loop can be model-side if planning has no exit condition, but you should also check whether each iteration's tool-output span is returning bad data.

The critical disambiguation rule follows from this sequence. If the tool-output span contains valid, schema-conformant data and the reasoning span ignores or fabricates beyond it, the fault is model-side. If the tool-output span is malformed and the reasoning span faithfully reflects that malformed data, the fault is tool-side, even if the model's downstream behavior looks like hallucination. The fault side must be assigned to the originating agent's span, not the agent where the failure becomes visible.

Where the MCP tool layer introduces harness-side failures that mimic model behavior

The MCP ecosystem, the Model Context Protocol introduced by Anthropic in late 2024, has become a dominant tool-integration layer for agents. This ecosystem introduces a class of tool-layer failure where the initial tool execution succeeds and no error surfaces. These faults are structurally invisible to model-versus-tool diagnostics that rely on error signals, and they require proactive span validation rather than reactive error detection.

The silent-success failure class is the core challenge. The fault is in the content of the tool's return, not its form. Schema drift follows as a recurring harness-side failure pattern. A tool's response schema changes upstream, a field is renamed, a type changes, a field is removed, and the agent continues calling the tool successfully. The tool-output span is well-formed at the HTTP layer, but the reasoning span now operates on structurally different data than the model was trained to expect. The diagnostic implication is that span walking must include schema validation of tool-output content against the registered tool definition, not just HTTP status inspection. A 200 with a drifted schema is a harness-side fault that span-level tracing will only catch if the tool-output span is scored against the expected schema. Tools like Lemma treat failures as structured patterns rather than one-off anomalies, organizing recurring deviations into actionable issues across the architectural stack, an approach that aligns with Shah et al.'s finding that faults traverse boundaries in repeatable ways and can be systematically exploited for diagnosis.

Why aggregate task-success metrics hide fault-side regressions

Task success is a lagging indicator, and it obscures whether a regression is model-side or tool-side. Once fault side can be assigned on a single trace, the natural extension is tracking fault-side distributions over time to catch regressions before they reach users. Aggregate scoring hides that divergence.

The practical illustration is telling. Without fault-mode metrics, you cannot tell a model regression from a tool-layer change. Offline held-out sets catch regressions on known model-side failure cases, the TS, RI, and OF patterns on fixed inputs. Shah et al. find that failures in agentic systems follow consistent patterns that can be systematically exploited. So fault-mode metrics from the model-versus-tool taxonomy are stable enough to track across deploys as regression signals.

Routing each fault type to the right remediation surface

Once fault side is assigned, remediation is deterministic. The discipline of routing correctly is what converts incident review into compounding reliability improvement rather than theater. Model-side remediation targets track the failure modes precisely. Tool-Skip calls for a tighter system prompt with explicit termination conditions, a plan-versus-execute evaluator that scores divergence and gates the next request, and a cycle detector for repeated tool invocations with no goal-state advance. Result-Ignore calls for a reasoning rubric that scores whether the model incorporated the tool's return, calibrated against cases where the tool return contradicts the model's likely prior, the ToolFailBench parametric trap design. Output-Fabrication calls for hallucination detection on grounded context, scoring output spans for content with no lineage in upstream tool-output spans.

Tool-side remediation targets are equally specific. For silent bad returns, you validate tool-output spans against the expected schema and alert on 200s that carry empty or structurally inconsistent payloads. Missing error handling calls for structured error recovery paths for each tool. The absence of an error path is an architectural gap, not a model gap. For schema drift, you continuously compare tool-output span content against registered tool definitions, so you catch drift before the agent has worked on a changed schema for many production runs.

The workflow implication matters as much as the technical fix. Remediation should happen in the developer's existing workflow. A model-side finding goes to the team working on planner prompts. A tool-side finding goes to the team owning the integration. Engineers should not need to context-switch into a separate monitoring console to act on what they find. The compounding argument closes the loop. A named fault category with a clear repair surface becomes a regression test in CI. The next deploy has to clear it, and the incident does not recur silently.