OpenNovelty.org

AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.0Download Report PDF
LLM AgentAgentic SystemFailure Attribution
Abstract:

Large Language Model (LLM)-based agentic systems, often comprising multiple models, complex tool invocations, and orchestration protocols, substantially outperform monolithic agents. Yet this very sophistication amplifies their fragility, making them more prone to system failure. Pinpointing the specific agent or step responsible for an error within long execution traces defines the task of \textbf{agentic system failure attribution}. Current state-of-the-art reasoning LLMs, however, remain strikingly inadequate for this challenge, with accuracy generally below 1010\\%. To address this gap, we propose AgenTracer, the first automated framework for annotating failed multi-agent trajectories via counterfactual replay and programmed fault injection, producing the curated dataset TracerTraj. Leveraging this resource, we develop AgenTracer-8B, a lightweight failure tracer trained with multi-granular reinforcement learning, capable of efficiently diagnosing errors in verbose multi-agent interactions. On {Who&When} benchmark, AgenTracer-8B outperforms giant proprietary LLMs like Gemini-2.5-Pro and Claude-4-Sonnet by up 18.1818.18\\%, setting a new standard in LLM agentic failure attribution. More importantly, AgenTracer-8B delivers actionable feedback to off-the-shelf multi-agent systems like MetaGPT and MaAS with 4.8∼14.24.8\sim14.2\\% performance gains, empowering self-correcting and self-evolving agentic AI.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper introduces AgenTracer, an automated framework for annotating failed multi-agent trajectories through counterfactual replay and fault injection, alongside AgenTracer-8B, a lightweight failure tracer trained via multi-granular reinforcement learning. It resides in the 'Automated Failure Attribution Techniques' leaf, which contains only four papers total, including this work and three siblings. This represents a notably sparse research direction within the broader taxonomy of fifty papers, suggesting that automated failure attribution in multi-agent LLM systems remains an emerging and under-explored area compared to more crowded branches like domain-specific applications or general system design.

The taxonomy reveals that failure attribution methods form one branch among several interconnected research directions. Neighboring leaves include 'Failure Analysis and Characterization' (three papers focused on empirical failure pattern identification) and broader categories like 'Robustness and Reliability Enhancement' (covering anomaly detection and resilience testing). The paper's focus on automated counterfactual-based attribution distinguishes it from sibling works that may employ spectrum analysis or causal inference scaffolding. Its position bridges the gap between general system design frameworks and domain-specific applications, addressing a foundational diagnostic challenge that cuts across multiple application contexts.

Among twenty-eight candidates examined through limited semantic search, none clearly refute the three core contributions. The automated annotation pipeline examined ten candidates with zero refutable overlaps; the lightweight failure tracer similarly found no prior work among ten candidates; and the multi-granular reinforcement learning approach encountered no refutations across eight candidates. This absence of overlapping prior work within the examined scope suggests the specific combination of counterfactual replay, programmed fault injection, and multi-granular RL for failure attribution appears novel. However, the limited search scale means unexplored literature beyond these twenty-eight candidates could contain relevant precedents.

Based on the constrained literature search covering top-K semantic matches, the work appears to occupy a relatively uncontested niche within automated failure attribution. The sparse taxonomy leaf and zero refutations across contributions indicate novelty within the examined scope, though the analysis does not claim exhaustive coverage of all potentially relevant prior work in adjacent fields like software debugging or distributed systems fault localization.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
28
Contribution Candidate Papers Compared
0
Refutable Paper

Research Landscape Overview

Core task: Failure attribution in large language model based multi-agent systems. The field has evolved around several interconnected branches that address how to design, evaluate, and improve LLM-based multi-agent systems when things go wrong. The taxonomy reveals a structure spanning from foundational Failure Attribution Methods and Frameworks—which develop techniques to identify root causes of errors—to System Design and Platforms that provide the architectural scaffolding for multi-agent collaboration (e.g., MetaGPT[2], AgentScope[6]). Parallel branches focus on Robustness and Reliability Enhancement, Domain-Specific Applications (such as medical diagnosis or manufacturing), and Evaluation and Benchmarking efforts like AgentBench[7] that measure agent performance. Additional branches cover Agent Learning and Optimization, Visualization and Interpretability Tools for understanding agent behavior, Comparative Analysis of System Architectures, and Memory and Cognitive Architectures that underpin agent reasoning. Within this landscape, a particularly active line of work centers on automated techniques for diagnosing failures in multi-agent settings. AgenTracer[0] sits squarely in the Automated Failure Attribution Techniques cluster, emphasizing systematic tracing of errors back to specific agents or interaction patterns. This contrasts with neighboring approaches such as Automated Failure Attribution[3], which may employ different diagnostic heuristics, and Spectrum Analysis Attribution[14] or Causal Inference Scaffolding[15], which leverage statistical or causal reasoning to pinpoint failure sources. Meanwhile, works like Multi-Agent LLM Failures[1] and Multi-Agent Diagnostic[4] explore broader taxonomies of error modes and diagnostic strategies. A key open question across these branches is how to balance automation with interpretability: while automated attribution can scale to complex systems, human experts often need transparent explanations to trust and act on diagnostic results. AgenTracer[0] addresses this by providing traceable failure paths, positioning itself as a bridge between fully automated diagnosis and human-interpretable insights.

Claimed Contributions

AgenTracer automated annotation pipeline

The authors introduce AgenTracer, an automated framework that annotates failed multi-agent trajectories by using counterfactual replay to identify decisive error steps and programmatic fault injection to generate synthetic failures. This pipeline produces the TracerTraj dataset containing over 2,000 annotated trajectories across seven benchmarks.

10 retrieved papers
AgenTracer-8B lightweight failure tracer

The authors develop AgenTracer-8B, a specialized 8B-parameter model trained using multi-granular reinforcement learning that can accurately diagnose errors in multi-agent systems at both step-level and agent-level granularity, enabling automated debugging of agentic systems.

10 retrieved papers
Multi-granular reinforcement learning training approach

The authors propose a multi-granular reinforcement learning approach that combines agent-level and step-level rewards to train the failure tracer, enabling it to provide accurate attribution across different levels of granularity in complex multi-agent trajectories.

8 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

AgenTracer automated annotation pipeline

The authors introduce AgenTracer, an automated framework that annotates failed multi-agent trajectories by using counterfactual replay to identify decisive error steps and programmatic fault injection to generate synthetic failures. This pipeline produces the TracerTraj dataset containing over 2,000 annotated trajectories across seven benchmarks.

Contribution

AgenTracer-8B lightweight failure tracer

The authors develop AgenTracer-8B, a specialized 8B-parameter model trained using multi-granular reinforcement learning that can accurately diagnose errors in multi-agent systems at both step-level and agent-level granularity, enabling automated debugging of agentic systems.

Contribution

Multi-granular reinforcement learning training approach

The authors propose a multi-granular reinforcement learning approach that combines agent-level and step-level rewards to train the failure tracer, enabling it to provide accurate attribution across different levels of granularity in complex multi-agent trajectories.

Table of Contents