OpenNovelty.org

Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.0Download Report PDF
training dynamicsrepresentation learninglazy/rich regimeneural collapsegrokkingkernel methods
Abstract:

Dynamic feature transformation (the rich regime) does not always align with predictive performance (better representation), yet accuracy is often used as a proxy for richness, limiting analysis of their relationship. We propose a computationally efficient, performance-independent metric of richness grounded in the low-rank bias of rich dynamics, which recovers neural collapse as a special case. The metric is empirically more stable than existing alternatives and captures known lazy-to-rich transitions (e.g., grokking) without relying on accuracy. We further use it to examine how training factors (e.g., learning rate) relate to richness, confirming recognized assumptions and highlighting new observations (e.g., batch normalization promote rich dynamics). An eigendecomposition-based visualization is also introduced to support interpretability, together providing a diagnostic tool for studying the relationship between training factors, dynamics, and representations.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper proposes a computationally efficient metric for measuring dynamical richness in neural networks that does not rely on predictive accuracy. It sits within the Feature Learning Dynamics Metrics leaf of the taxonomy, which contains only two papers total. This is a notably sparse research direction, suggesting the problem of decoupling dynamics from performance remains underexplored. The sibling paper in this leaf also addresses disentangling rich dynamics from task outcomes, indicating a nascent but focused line of inquiry.

The taxonomy reveals that neighboring leaves pursue related but distinct goals. Complexity Quantification Methods (five papers) measure temporal or spatial complexity through entropy and information theory, while Diversity and Quality Trade-offs (two papers) balance exploration with optimization in evolutionary algorithms. The original paper bridges these areas by grounding its richness metric in low-rank bias rather than entropy or diversity maintenance, positioning it at the intersection of feature learning theory and dynamical systems characterization without direct overlap with prediction-oriented branches.

Among thirty candidates examined, none clearly refute the three core contributions. The Dynamical Low-Rank Measure examined ten candidates with zero refutable matches, as did the connection to neural collapse and the eigendecomposition visualization method. This suggests that within the limited search scope, the specific combination of low-rank bias as a richness proxy, its theoretical link to neural collapse, and the proposed visualization approach appear novel. The absence of refutations may reflect both the sparse literature in this exact niche and the limited scale of the search.

Based on the top-thirty semantic matches and taxonomy structure, the work appears to occupy a relatively unexplored corner of the field. The Feature Learning Dynamics Metrics leaf is small, and no examined candidates provide overlapping prior work for any contribution. However, the search scope is inherently limited, and a broader survey might reveal related metrics or visualizations in adjacent communities not captured here.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
30
Contribution Candidate Papers Compared
0
Refutable Paper

Research Landscape Overview

Core task: measuring dynamical richness independent of predictive performance. This field addresses a fundamental tension in machine learning and complex systems: how to quantify the intrinsic complexity or diversity of a system's internal dynamics without conflating it with task-specific accuracy. The taxonomy reflects four main branches. Theoretical Frameworks for Dynamics-Performance Decoupling develop formal metrics and conceptual tools to separate dynamical properties from prediction quality, often drawing on information theory and feature learning perspectives (e.g., Decoupling Dynamical Richness[0], Disentangling Rich Dynamics[43]). Dynamical Systems Characterization focuses on classical and modern techniques for analyzing temporal evolution, attractors, and complexity measures in physical and biological systems (e.g., Quantum Reservoir Computing[10], EEG Energy Landscape[24]). Prediction Applications with Dynamical Complexity examines domains where rich dynamics coexist with forecasting tasks—such as traffic flow (YOLOv8 Traffic Flow[3], Visual Features Traffic[4]), climate (ENSO RNN LSTM[13]), and landslide prediction (CNN LSTM Landslide[9])—highlighting the interplay between model expressiveness and practical performance. Algorithmic and Computational Methods provide diversity-aware optimization and sampling strategies (Quality Diversity Without Maintenance[5], Determinantal Point Processes[21]) that maintain behavioral variety alongside objective optimization. A particularly active line of work explores quality-diversity algorithms and diversity metrics that preserve exploration in evolutionary and reinforcement learning settings, balancing novelty with performance (Dynamics Aware Quality Diversity[12], Likelihood Diverse Sampling[19]). Another contrasting theme emerges in reservoir computing and recurrent architectures, where internal state complexity is leveraged for temporal prediction (Time History Reservoir[17], Quantum Reservoir Computing[10]), yet the relationship between reservoir richness and generalization remains subtle. Decoupling Dynamical Richness[0] sits squarely within the Theoretical Frameworks branch, specifically addressing Feature Learning Dynamics Metrics. Its emphasis on decoupling aligns closely with Disentangling Rich Dynamics[43], which also seeks to isolate dynamical properties from task outcomes. Compared to diversity-focused works like Quality Diversity Without Maintenance[5] or Determinantal Point Processes[21], which operate in optimization contexts, Decoupling Dynamical Richness[0] appears more concerned with intrinsic characterization of learning trajectories, offering a complementary lens on how systems evolve internally regardless of their final predictive success.

Claimed Contributions

Dynamical Low-Rank Measure (DLR)

The authors propose a computationally efficient metric called DLR that quantifies dynamical richness in neural networks by comparing activations before and after the last layer. This metric is grounded in the low-rank bias of rich dynamics and operates independently of predictive performance, enabling direct evaluation of training dynamics without referencing accuracy.

10 retrieved papers
Connection between DLR and neural collapse

The authors establish that their proposed metric recovers neural collapse conditions (NC1 and NC2) as a special case when the feature kernel operator is a minimum projection operator. This theoretical connection extends the applicability of their metric beyond labeled classification tasks to more general settings.

10 retrieved papers
Eigendecomposition-based visualization method

The authors introduce a complementary visualization technique based on eigendecomposition of the feature kernel operator that quantifies cumulative feature quality, utilization, and relative eigenvalues. This visualization method aids in interpreting the richness metric and provides insights into how features align with tasks and are utilized by the final layer.

10 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

Dynamical Low-Rank Measure (DLR)

The authors propose a computationally efficient metric called DLR that quantifies dynamical richness in neural networks by comparing activations before and after the last layer. This metric is grounded in the low-rank bias of rich dynamics and operates independently of predictive performance, enabling direct evaluation of training dynamics without referencing accuracy.

Contribution

Connection between DLR and neural collapse

The authors establish that their proposed metric recovers neural collapse conditions (NC1 and NC2) as a special case when the feature kernel operator is a minimum projection operator. This theoretical connection extends the applicability of their metric beyond labeled classification tasks to more general settings.

Contribution

Eigendecomposition-based visualization method

The authors introduce a complementary visualization technique based on eigendecomposition of the feature kernel operator that quantifies cumulative feature quality, utilization, and relative eigenvalues. This visualization method aids in interpreting the richness metric and provides insights into how features align with tasks and are utilized by the final layer.

Table of Contents