OpenNovelty.org

Decentralized Attention Fails Centralized Signals: Rethinking Transformers for Medical Time Series

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.0Download Report PDF
EEGECGDeep learningTransformer
Abstract:

Accurate analysis of Medical time series (MedTS) data, such as Electroencephalography (EEG) and Electrocardiography (ECG), plays a pivotal role in healthcare applications, including the diagnosis of brain and heart diseases. MedTS data typically exhibits two critical patterns: temporal dependencies within individual channels and channel dependencies across multiple channels. While recent advances in deep learning have leveraged Transformer-based models to effectively capture temporal dependencies, they often struggle to model channel dependencies. This limitation stems from a structural mismatch: MedTS signals are inherently centralized, whereas the Transformer's attention is decentralized, making it less effective at capturing global synchronization and unified waveform patterns. To bridge this gap, we propose CoTAR (Core Token Aggregation-Redistribution), a centralized MLP-based module tailored to replace the decentralized attention. Instead of allowing all tokens to interact directly, as in attention, CoTAR introduces a global core token that acts as a proxy to facilitate the inter-token interaction, thereby enforcing a centralized aggregation and redistribution strategy. This design not only better aligns with the centralized nature of MedTS signals but also reduces computational complexity from quadratic to linear. Experiments on five benchmarks validate the superiority of our method in both effectiveness and efficiency, achieving up to a 12.13% improvement on the APAVA dataset, with merely 33% memory usage and 20% inference time compared to the previous state-of-the-art. Code and all training scripts are available in this Link.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper proposes CoTAR, a centralized MLP-based module replacing decentralized attention in transformers for medical time series analysis. It sits within the Transformer-Based Joint Modeling leaf, which contains only three papers including this work. This is a relatively sparse research direction within the broader taxonomy of 50 papers across 36 topics, suggesting the specific approach of replacing attention with centralized token aggregation for joint temporal-channel modeling is not yet heavily explored in the medical time series literature.

The taxonomy reveals that joint spatiotemporal modeling represents one of several major branches, alongside temporal-only architectures (RNNs, attention-based temporal models) and channel-only methods (graph-based, MLP-based channel mixing). The paper's sibling works—Medformer and Dispformer—employ standard transformer blocks for joint modeling, while neighboring leaves include hybrid convolutional-recurrent architectures and multi-scale approaches. The scope notes indicate that transformers focusing solely on temporal attention belong elsewhere, clarifying that this leaf specifically addresses integrated temporal-channel mechanisms. The paper diverges from graph-based channel modeling and attention-based channel mixing by introducing a centralized proxy token rather than direct pairwise interactions.

Among 30 candidates examined, the CoTAR module shows one refutable candidate from 10 examined, while the TeCh framework with Adaptive Dual Tokenization also has one refutable candidate from 10 examined. The identification of structural mismatch between attention and medical time series appears more novel, with zero refutable candidates among 10 examined. This suggests that while the specific architectural components may have some precedent in the limited search scope, the conceptual framing of centralized versus decentralized modeling for medical signals represents a less-explored perspective. The analysis indicates moderate prior work overlap for the core technical contributions but stronger novelty in the problem formulation.

Based on the top-30 semantic matches examined, the work appears to occupy a relatively underexplored niche within transformer-based joint modeling, though the limited search scope means potentially relevant work outside this candidate set remains unexamined. The sparse population of the taxonomy leaf and the conceptual novelty of the centralization argument suggest meaningful differentiation from existing approaches, while the refutable candidates for specific modules indicate some architectural overlap within the examined literature.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
30
Contribution Candidate Papers Compared
2
Refutable Paper

Research Landscape Overview

Core task: Modeling temporal and channel dependencies in medical time series. The field addresses the dual challenge of capturing how clinical variables evolve over time and how they interact with one another across channels. The taxonomy reveals several major branches: Temporal Dependency Modeling Architectures focus on sequential patterns using RNNs, LSTMs, and attention mechanisms; Channel Dependency and Inter-Variable Modeling emphasizes cross-variable relationships through graph-based or correlation-driven methods; Joint Spatiotemporal Modeling integrates both dimensions simultaneously, often via transformers or hybrid architectures; Data Imputation and Reconstruction tackles missing data; Clinical Prediction and Classification applies these models to diagnostic tasks; Representation Learning and Self-Supervision explores unsupervised pretraining; and Specialized Modeling Techniques covers domain-specific innovations. Representative works such as Medformer[1] and Dispformer[39] illustrate transformer-based joint modeling, while approaches like Spatiotemporal Graph Medical[2] and Channel Independence Mamba[7] highlight contrasting strategies for handling variable interactions. A central tension in the field lies between methods that explicitly model channel dependencies versus those that treat channels independently to reduce complexity. Transformer-based joint modeling has emerged as a particularly active direction, balancing expressiveness with computational feasibility. Decentralized Attention Medical[0] sits within this branch alongside Medformer[1] and Dispformer[39], emphasizing efficient attention mechanisms that capture both temporal evolution and inter-channel relationships without prohibitive computational costs. Compared to Medformer[1], which typically employs standard transformer blocks, Decentralized Attention Medical[0] appears to explore alternative attention designs that may distribute computation or focus on localized dependencies. Meanwhile, works like SimTA[5] and Clinical ICD Coding[3] demonstrate how joint modeling supports diverse downstream tasks, from representation learning to multi-label classification, underscoring ongoing questions about how best to balance model capacity, interpretability, and clinical utility in irregular, high-dimensional medical time series.

Claimed Contributions

Core Token Aggregation-Redistribution (CoTAR) module

CoTAR is a centralized MLP-based module that replaces the standard attention mechanism in Transformers. Instead of direct pairwise token interactions, it introduces a global core token that aggregates information from all tokens and redistributes it back, reducing computational complexity from quadratic to linear while better aligning with the centralized nature of medical time series signals.

10 retrieved papers
Can Refute
TeCh framework with Adaptive Dual Tokenization

TeCh is a unified framework built on CoTAR that can adaptively model temporal dependencies, channel dependencies, or both by adjusting the tokenization strategy (Temporal, Channel, or Dual). This flexibility allows the framework to better match the unique characteristics of different medical time series datasets.

10 retrieved papers
Can Refute
Identification of structural mismatch between attention and medical time series

The authors identify and formalize a fundamental mismatch: medical time series signals like EEG and ECG originate from centralized biological sources (brain, heart), while Transformer attention operates as a decentralized graph where all tokens interact equally. This mismatch causes attention to fail at capturing the global synchronization and unified patterns essential for modeling channel dependencies in medical signals.

10 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

Core Token Aggregation-Redistribution (CoTAR) module

CoTAR is a centralized MLP-based module that replaces the standard attention mechanism in Transformers. Instead of direct pairwise token interactions, it introduces a global core token that aggregates information from all tokens and redistributes it back, reducing computational complexity from quadratic to linear while better aligning with the centralized nature of medical time series signals.

Contribution

TeCh framework with Adaptive Dual Tokenization

TeCh is a unified framework built on CoTAR that can adaptively model temporal dependencies, channel dependencies, or both by adjusting the tokenization strategy (Temporal, Channel, or Dual). This flexibility allows the framework to better match the unique characteristics of different medical time series datasets.

Contribution

Identification of structural mismatch between attention and medical time series

The authors identify and formalize a fundamental mismatch: medical time series signals like EEG and ECG originate from centralized biological sources (brain, heart), while Transformer attention operates as a decentralized graph where all tokens interact equally. This mismatch causes attention to fail at capturing the global synchronization and unified patterns essential for modeling channel dependencies in medical signals.

Table of Contents