CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs

The paper proposes CompoDistill, a knowledge distillation framework that transfers visual perception abilities from large teacher MLLMs to smaller student models by explicitly aligning visual attention patterns. It resides in the Attention-Based Distillation leaf, which contains only two papers including this work. This leaf sits within the broader Knowledge Distillation Frameworks for MLLMs branch, which encompasses four distinct distillation strategies (attention-based, feature-level, competitive, and domain-specific). The sparse population of the attention-based leaf suggests this specific angle—using attention alignment to preserve compositional visual reasoning—is relatively underexplored compared to feature-level distillation approaches.

The taxonomy reveals that neighboring research directions include Feature-Level Distillation (three papers focusing on intermediate representations) and Competitive Distillation (two papers on bidirectional knowledge transfer). The Visual Encoding and Perception Enhancement branch offers alternative pathways to improve visual understanding through multi-encoder integration or perception tokens, rather than distillation. The Reasoning and Compositional Understanding branch addresses similar compositional challenges but through chain-of-thought methods or reward signals instead of teacher-student knowledge transfer. CompoDistill bridges these areas by framing compositional reasoning as a distillation problem requiring attention-level alignment, distinct from both pure architectural enhancements and reasoning-focused interventions.

Among thirty candidates examined, the contribution identifying visual attention misalignment as a barrier shows one refutable candidate from ten examined, suggesting some prior recognition of attention's role in distillation quality. The CompoDistill framework itself (VAT and TAF modules) encountered no refutations across ten candidates, indicating the specific architectural combination may be novel within this limited search scope. The three-stage training strategy also shows one refutable candidate from ten examined, implying staged distillation approaches exist but may differ in implementation details. The modest refutation counts reflect the constrained search scale rather than exhaustive field coverage.

Based on the top-thirty semantic matches and citation expansion, the work appears to occupy a sparsely populated intersection of attention-based distillation and compositional visual reasoning. The taxonomy structure confirms that while distillation for MLLMs is an active area, attention-specific approaches remain less common than feature-level methods. The analysis cannot rule out relevant work outside the examined candidates, particularly in adjacent communities focused on vision transformers or general knowledge distillation beyond the MLLM context.