OpenNovelty.org

GhostEI-Bench: Do Mobile Agent Resilience to Environmental Injection in Dynamic On-Device Environments?

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.0Download Report PDF
Mobile AgentsEnvironmental InjectionBenchmarkGUI Agent Safety
Abstract:

Vision-Language Models (VLMs) are increasingly deployed as autonomous agents to navigate mobile Graphical User Interfaces (GUIs). However, their operation within dynamic on-device ecosystems, which include notifications, pop-ups, and inter-app interactions, exposes them to a unique and underexplored threat vector: environmental injection. Unlike traditional prompt-based attacks that manipulate textual instructions, environmental injection contaminates the agent's visual perception by inserting adversarial UI elements, such as deceptive overlays or spoofed notifications, directly into the GUI. This bypasses textual safeguards and can derail agent execution, leading to privacy leakage, financial loss, or irreversible device compromise.

To systematically evaluate this threat, we introduce GhostEI-Bench, the first benchmark dedicated to assessing mobile agents under environmental injection attacks within dynamic, executable environments. Moving beyond static image-based assessments, our benchmark injects adversarial events into realistic application workflows inside fully operational Android emulators, assessing agent performance across a range of critical risk scenarios. We also introduce a novel evaluation protocol where a judge LLM performs fine-grained failure analysis by reviewing the agent's action trajectory alongside the corresponding sequence of screenshots. This protocol identifies the precise point of failure, whether in perception, recognition, or reasoning.

Our comprehensive evaluation of state-of-the-art agents reveals their profound vulnerability to deceptive environmental cues. The results demonstrate that current models systematically fail to perceive and reason about manipulated UIs. GhostEI-Bench provides an essential framework for quantifying and mitigating this emerging threat, paving the way for the development of more robust and secure embodied agents.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper introduces GhostEI-Bench, a benchmark for evaluating mobile agents under environmental injection attacks within executable Android environments. It resides in the 'Benchmark and Evaluation Frameworks for Environmental Injection' leaf, which contains four papers total. This is a relatively sparse research direction within the broader taxonomy of 50 papers, suggesting that systematic evaluation frameworks for environmental injection remain underdeveloped. The work targets a specific gap: moving beyond static image-based assessments to dynamic, executable workflows where adversarial UI elements are injected into realistic application contexts.

The taxonomy reveals that environmental injection attacks on mobile and GUI agents form one major branch, with sibling leaves addressing security vulnerabilities and defense mechanisms. Neighboring branches cover false data injection in multi-agent systems and adversarial perturbations in reinforcement learning, which focus on sensor spoofing and state-space attacks rather than GUI-level manipulation. The scope note for this leaf explicitly excludes general robustness testing without environmental injection focus, positioning GhostEI-Bench within a narrow but critical niche: evaluating how agents perceive and respond to adversarial visual cues in mobile interfaces, distinct from prompt-based or communication-layer attacks.

Among 29 candidates examined, the analysis identified potential overlaps across all three contributions. The benchmark contribution examined 10 candidates with 1 refutable match, the evaluation protocol examined 10 with 2 refutable matches, and the threat model formalization examined 9 with 3 refutable matches. These statistics indicate that within the limited search scope, some prior work addresses related evaluation methodologies or threat characterizations. However, the relatively low refutation counts suggest that the specific combination of executable Android environments, dynamic injection, and fine-grained failure analysis may offer incremental novelty over existing static or web-focused benchmarks.

Given the sparse taxonomy leaf and limited search scope of 29 candidates, the work appears to occupy a moderately novel position within environmental injection evaluation. The analysis does not cover exhaustive literature beyond top-K semantic matches, so additional related work may exist in adjacent domains such as web agent security or mobile app testing. The contribution-level statistics suggest that while individual components have precedents, the integrated benchmark design targeting mobile GUI agents in executable environments may represent a meaningful step forward in a nascent research area.

Taxonomy

Core-task Taxonomy Papers
49
3
Claimed Contributions
27
Contribution Candidate Papers Compared
6
Refutable Paper

Research Landscape Overview

Core task: mobile agent robustness to environmental injection attacks. The field examines how autonomous agents—ranging from GUI-based assistants to multi-robot teams—withstand adversarial manipulations of their perceived environment. The taxonomy reveals several major branches: one focuses on environmental injection attacks targeting mobile and GUI agents, where adversaries insert misleading cues into web pages or smartphone interfaces (e.g., Environmental Injection Robustness[1], AEIA-MN[2]); another addresses false data injection in multi-agent systems, particularly in distributed estimation and consensus protocols (Resilient Consensus Tracking[4], Distributed Adversarial Detection[13]); a third explores robustness of reinforcement learning agents to adversarial perturbations in state or action spaces (Adversarial State Perturbations[3], Action Space Adversarial[39]); and additional branches cover multi-robot coordination under adversarial conditions (Multi-Agent Adversarial Control[5], Adversarial Multi-Robot Coordination[27]), robotic navigation resilience (Sim2Real Navigation Robustness[17], Deviation-Robust Navigation[37]), and specialized control or cyber-physical system defenses (EV Charging Resilience[10], Multi-Microgrid Reinforcement Defense[47]). Together, these branches illustrate a spectrum from high-level cognitive agents vulnerable to prompt or content injection to low-level control systems facing sensor spoofing or communication attacks. A particularly active line of work centers on benchmark and evaluation frameworks for environmental injection, where researchers develop systematic testbeds to measure agent susceptibility to manipulated observations. GhostEI-Bench[0] exemplifies this direction by providing a structured evaluation suite for mobile agents confronting injected environmental cues, closely aligned with Environmental Injection Robustness[1] and AEIA-MN[2], which similarly probe how agents parse and trust external information. In contrast, works like Hijacking JARVIS[6] and Protocol Exploits Agents[26] emphasize attack construction and exploit discovery in agent protocols, while MobileSafetyBench[16] and Mobile LLM Security[20] broaden the scope to general safety and security concerns in mobile LLM-based agents. The main trade-off across these branches involves balancing detection granularity—whether to focus on fine-grained prompt injections, coarse sensor spoofing, or systemic communication disruptions—against the computational overhead of defense mechanisms. GhostEI-Bench[0] sits squarely within the environmental injection evaluation cluster, offering a controlled setting to assess agent robustness without prescribing specific defenses, thereby complementing attack-focused studies and providing a foundation for comparing mitigation strategies across diverse agent architectures.

Claimed Contributions

GhostEI-Bench benchmark for environmental injection attacks

The authors present GhostEI-Bench, a comprehensive benchmark that systematically evaluates mobile agent robustness against environmental injection attacks in fully operational Android emulators. The benchmark includes 110 test cases spanning seven critical risk fields and three attack vectors, moving beyond static image-based assessments to inject adversarial events into realistic application workflows.

9 retrieved papers
Can Refute
Novel LLM-based evaluation protocol with fine-grained failure analysis

The authors propose an evaluation protocol that uses a judge LLM to analyze agent action trajectories and screenshots, identifying precise failure points in perception, recognition, or reasoning. This protocol enables systematic assessment of both capability and robustness through metrics including Task Completion, Full/Partial Attack Success, and Vulnerability Rate.

10 retrieved papers
Can Refute
Formalization of environmental injection as a distinct threat model

The authors establish environmental injection as a unique threat vector that contaminates agent visual perception through adversarial UI elements like deceptive overlays or spoofed notifications. This formalization defines a unified threat model encompassing three attack vectors: Deceptive Instruction, Static Environmental Injection, and Dynamic Environmental Injection across seven critical risk fields.

8 retrieved papers
Can Refute

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

GhostEI-Bench benchmark for environmental injection attacks

The authors present GhostEI-Bench, a comprehensive benchmark that systematically evaluates mobile agent robustness against environmental injection attacks in fully operational Android emulators. The benchmark includes 110 test cases spanning seven critical risk fields and three attack vectors, moving beyond static image-based assessments to inject adversarial events into realistic application workflows.

Contribution

Novel LLM-based evaluation protocol with fine-grained failure analysis

The authors propose an evaluation protocol that uses a judge LLM to analyze agent action trajectories and screenshots, identifying precise failure points in perception, recognition, or reasoning. This protocol enables systematic assessment of both capability and robustness through metrics including Task Completion, Full/Partial Attack Success, and Vulnerability Rate.

Contribution

Formalization of environmental injection as a distinct threat model

The authors establish environmental injection as a unique threat vector that contaminates agent visual perception through adversarial UI elements like deceptive overlays or spoofed notifications. This formalization defines a unified threat model encompassing three attack vectors: Deceptive Instruction, Static Environmental Injection, and Dynamic Environmental Injection across seven critical risk fields.

Table of Contents