OpenNovelty.org

Neural Collapse in Multi-Task Learning

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.5Download Report PDF
Neural Collapse
Abstract:

Neural collapse (NC) plays a key role in understanding deep neural networks. However, existing empirical and theoretical studies of NC focus on one single task. This paper studies neural collapse in multi-task learning. We consider two standard feature-based multi-task learning scenarios: Single-Source Multi-Task Classification (SSMTC) and Multi-Source Multi-Task Classification (MSMTC). Interestingly, we find that the task-specific linear classifier and features converge to the Simplex Equiangular Tight Frame (ETF) in the setting of MSMTC. In the setting of SSMTC, task-specific linear classifier converges to the task-specific ETF and these task-specific ETFs are mutually orthogonal. Moreover, the shared features across tasks converge to the scaled sum of the weight vectors associated with the task-specific labels in each task's classifier. We also provide the theoretical guarantee for our empirical findings. Through detailed analysis, we uncover the mechanism of MTL where each task learns task-specific latent features that together form the shared features. Moreover, we reveal an inductive bias in MTL that task correlation reconfigures the geometry of task-specific classifiers and promotes alignment among the features learned by each task.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

This paper investigates neural collapse (NC) in multi-task learning, specifically examining how task-specific classifiers and shared features converge to geometric structures like Simplex Equiangular Tight Frames (ETF). The work resides in the 'Feature Sharing and Representation Learning' leaf, which contains only two papers total. This sparse population suggests the intersection of neural collapse theory and multi-task representation learning remains relatively unexplored, positioning the paper in a niche but potentially underserved research direction within the broader multi-task learning landscape.

The taxonomy reveals that neighboring leaves focus on 'Task-Specific Classifier Design' and 'Dynamic Loss Weighting and Optimization', both addressing complementary aspects of multi-task systems. While sibling work examines pose-invariant recognition and shared representations empirically, this paper provides theoretical analysis of geometric convergence properties. The parent branch 'Multi-Task Learning Architectures and Mechanisms' excludes single-task methods and domain applications, clarifying that this work contributes foundational understanding of how multi-task systems organize learned representations rather than proposing new architectures or application-specific solutions.

Among thirty candidates examined, the contribution on 'Multi-Task Neural Collapse Phenomenon' shows no clear refutation across ten papers reviewed, suggesting this specific geometric analysis may be novel within the limited search scope. However, 'Global Optimality of SSMTC-NC and MSMTC-NC' encountered two potentially refutable candidates among ten examined, and 'Insights on Multi-Task Learning Mechanisms' found one among ten. These statistics indicate that while the core NC phenomenon in multi-task settings appears less explored, theoretical optimality claims and mechanistic insights face more substantial prior work overlap within the candidate pool.

Based on the limited thirty-candidate search, the paper appears to occupy a relatively sparse research intersection between neural collapse theory and multi-task learning. The analysis does not cover exhaustive literature review or systematic comparison across all multi-task representation learning work. The sibling paper count and contribution-level statistics suggest moderate novelty for the core phenomenon, with more caution warranted for optimality and mechanistic claims given the identified overlaps.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
30
Contribution Candidate Papers Compared
3
Refutable Paper

Research Landscape Overview

The core task of this survey centers on understanding how systems learn and optimize across multiple objectives or tasks simultaneously. The taxonomy reveals a rich landscape organized into twelve major branches, spanning architectural innovations (Multi-Task Learning Architectures and Mechanisms), practical deployments (Multi-Task Learning Applications and Domain-Specific Systems), optimization challenges (Multi-Objective Optimization and Constrained Problems), and foundational concerns such as security, theoretical complexity, and human interaction. Some branches emphasize technical mechanisms—how neural networks share features or balance competing objectives—while others focus on domain-specific instantiations in healthcare, robotics, or communication systems. Works like Show Primary Objectives[1] and Complement Objective Training[8] illustrate efforts to manage conflicting goals, whereas branches on datasets and benchmarks (e.g., Agriculture Vision Datasets[11]) provide the empirical grounding for evaluating multi-task methods. Within the Feature Sharing and Representation Learning cluster, a particularly active line of inquiry examines how shared representations emerge and stabilize across tasks. Neural Collapse MultiTask[0] investigates geometric properties of learned features in multi-task settings, exploring whether the neural collapse phenomenon—where within-class features converge to simplex structures—extends beyond single-task scenarios. This work sits alongside studies like Pose Invariant Recognition[46], which also grapples with learning invariant or shared representations under diverse conditions. The central tension in this area revolves around balancing task-specific specialization with the efficiency gains of shared feature spaces, a trade-off that appears across many branches. By situating Neural Collapse MultiTask[0] in this context, we see it as contributing to the theoretical understanding of representation geometry, complementing empirical multi-task architectures and offering insights into when and why feature sharing succeeds or fails.

Claimed Contributions

Multi-Task Neural Collapse Phenomenon

The authors identify and characterize neural collapse phenomena in two multi-task learning settings: Single-Source Multi-Task Classification (SSMTC-NC) and Multi-Source Multi-Task Classification (MSMTC-NC). In SSMTC-NC, task-specific classifiers converge to mutually orthogonal Simplex ETFs and shared features converge to scaled sums of task-specific classifier weights. In MSMTC-NC, task-specific features and classifiers both converge to task-specific Simplex ETFs.

10 retrieved papers
Global Optimality of SSMTC-NC and MSMTC-NC

The authors provide theoretical guarantees by proving that any global minimizer of the training objectives under the Unconstrained Feature Model satisfies the geometric properties characterizing SSMTC-NC and MSMTC-NC phenomena.

10 retrieved papers
Can Refute
Insights on Multi-Task Learning Mechanisms

The authors uncover that shared features in multi-task learning are composed of task-specific latent features learned by individual tasks. They also reveal an inductive bias where task correlation reconfigures the geometry of task-specific classifiers and promotes feature alignment across tasks.

10 retrieved papers
Can Refute

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Within the taxonomy built over the current TopK core-task papers, the original paper is assigned to a leaf with no direct siblings and no cousin branches under the same grandparent topic. In this retrieved landscape, it appears structurally isolated, which is one partial signal of novelty, but still constrained by search coverage and taxonomy granularity.

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

Multi-Task Neural Collapse Phenomenon

The authors identify and characterize neural collapse phenomena in two multi-task learning settings: Single-Source Multi-Task Classification (SSMTC-NC) and Multi-Source Multi-Task Classification (MSMTC-NC). In SSMTC-NC, task-specific classifiers converge to mutually orthogonal Simplex ETFs and shared features converge to scaled sums of task-specific classifier weights. In MSMTC-NC, task-specific features and classifiers both converge to task-specific Simplex ETFs.

Contribution

Global Optimality of SSMTC-NC and MSMTC-NC

The authors provide theoretical guarantees by proving that any global minimizer of the training objectives under the Unconstrained Feature Model satisfies the geometric properties characterizing SSMTC-NC and MSMTC-NC phenomena.

Contribution

Insights on Multi-Task Learning Mechanisms

The authors uncover that shared features in multi-task learning are composed of task-specific latent features learned by individual tasks. They also reveal an inductive bias where task correlation reconfigures the geometry of task-specific classifiers and promotes feature alignment across tasks.

Table of Contents