OpenNovelty.org

Discovering alternative solutions beyond the simplicity bias in recurrent neural networks

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 7.3Download Report PDF
recurrent neural networkscomputational neurosciencedynamical systems
Abstract:

Training recurrent neural networks (RNNs) to perform neuroscience-style tasks has become a popular way to generate hypotheses for how neural circuits in the brain might perform computations. Recent work has demonstrated that task-trained RNNs possess a strong simplicity bias. In particular, this inductive bias often causes RNNs trained on the same task to collapse on effectively the same solution, typically comprised of fixed-point attractors or other low-dimensional dynamical motifs. While such solutions are readily interpretable, this collapse proves counterproductive for the sake of generating a set of genuinely unique hypotheses for how neural computations might be performed. Here we propose Iterative Neural Similarity Deflation (INSD), a simple method to break this inductive bias. By penalizing linear predictivity of neural activity produced by standard task-trained RNNs, we find an alternative class of solutions to classic neuroscience-style RNN tasks. These solutions appear distinct across a battery of analysis techniques, including representational similarity metrics, dynamical systems analysis, and the linear decodability of task-relevant variables. Moreover, these alternative solutions can sometimes achieve superior performance in difficult or out-of-distribution task regimes. Our findings underscore the importance of moving beyond the simplicity bias to uncover richer and more varied models of neural computation.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper introduces Iterative Neural Similarity Deflation (INSD), a method to break the simplicity bias in task-trained RNNs by penalizing linear predictivity of neural activity. It sits within the 'Quantifying and Controlling Solution Degeneracy' leaf, which contains only three papers total. This leaf focuses specifically on methods for measuring and manipulating solution degeneracy across behavioral, dynamical, and weight-space levels. The sparse population suggests this is an emerging rather than saturated research direction, with relatively few prior works directly addressing controlled generation of diverse RNN solutions.

The taxonomy reveals that solution diversity research connects to several neighboring areas. The sibling leaf 'Discovery of Multiple Algorithmic Strategies' (three papers) examines qualitatively different algorithms emerging naturally, while 'Universality and Individuality' (one paper) studies shared versus unique representations across RNN populations. The broader 'Mechanistic Interpretation' branch explores dynamical systems analysis and working memory coding strategies—analytical tools that INSD leverages to characterize discovered solutions. The paper bridges degeneracy control methods with mechanistic interpretation techniques, positioning itself at the intersection of generating diversity and analyzing what makes solutions genuinely distinct.

Among thirty candidates examined, the contribution-level analysis shows mixed novelty signals. The INSD method itself (Contribution 1) examined ten candidates with one refutable match, suggesting some methodological overlap exists within the limited search scope. Similarly, discovering alternative solutions beyond simplicity bias (Contribution 2) found one refutable candidate among ten examined. The framework for generating diverse computational hypotheses (Contribution 3) showed no refutable matches across ten candidates, indicating this framing may be more distinctive. The statistics reflect a focused but not exhaustive literature search, leaving open whether additional relevant work exists beyond the top-thirty semantic matches.

Given the sparse taxonomy leaf and limited search scope, the work appears to address a genuine gap in controlled diversity generation for task-trained RNNs. The one-to-two refutable matches per contribution suggest some methodological precedent exists, but the overall scarcity of papers in this specific research direction indicates the problem remains relatively underexplored. The analysis captures top semantic matches but cannot rule out relevant work in adjacent communities or under different terminological framings.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
30
Contribution Candidate Papers Compared
2
Refutable Paper

Research Landscape Overview

Core task: Discovering diverse solutions in task-trained recurrent neural networks. The field explores how RNNs trained on identical tasks can arrive at qualitatively different internal mechanisms—a phenomenon known as solution degeneracy. The taxonomy reflects a broad landscape organized around several major themes. One branch examines solution diversity and degeneracy directly, investigating how multiple distinct circuit-level strategies emerge from the same objective and how researchers can quantify or control this variability (e.g., Alternative Solutions Simplicity[0], Solution Degeneracy Control[25]). A second branch focuses on mechanistic interpretation, dissecting the internal dynamics and representational structure that task-optimized RNNs develop (Cognitive Strategies Discovery[3], Clock Pizza Mechanistic[5]). A third branch draws comparisons between RNN solutions and biological neural systems, asking whether artificial networks recapitulate known neural coding schemes or circuit motifs (Dynamic Coding Memory[6], Nodes to Networks[7]). Additional branches address architecture design, methodological tools for analysis, and a wide range of application domains—from forecasting and classification to computer vision and resource scheduling—demonstrating that the core questions about solution diversity arise across many practical settings. Within the solution diversity and degeneracy branch, a particularly active line of work seeks to chart the space of possible solutions and develop principled ways to sample or steer networks toward simpler or more interpretable configurations. Alternative Solutions Simplicity[0] sits squarely in this cluster, emphasizing methods to discover and compare alternative circuit implementations that solve the same task. Nearby efforts such as Solution Degeneracy Control[25] and Charting Solution Space[29] share a focus on mapping out the landscape of degenerate solutions and understanding the factors—initialization schemes, regularization, or architectural constraints—that bias networks toward one solution over another. A key open question is whether certain solutions generalize better or align more closely with biological plausibility, and how one might systematically favor such solutions during training. By situating itself among these works, Alternative Solutions Simplicity[0] contributes tools for navigating the rich, often redundant space of learned representations, helping researchers move beyond single-solution analyses toward a more complete picture of what task-trained RNNs can learn.

Claimed Contributions

Iterative Neural Similarity Deflation (INSD) method

The authors introduce INSD, a training procedure that penalizes linear predictivity of neural activity from previously trained RNNs in an iterative manner. This method enables discovery of alternative task solutions that diverge from the prototypical solutions typically found due to simplicity bias in RNNs.

10 retrieved papers
Can Refute
Discovery of alternative RNN solutions beyond simplicity bias

The authors demonstrate that their method uncovers a distinct class of solutions to neuroscience tasks that differ from standard solutions in representational geometry, dynamical motifs, and encoding of task variables. These alternative solutions forgo fixed-point attractors and instead maintain information in dynamically evolving subspaces.

10 retrieved papers
Can Refute
Framework for generating diverse computational hypotheses in neuroscience

The authors address the problem of generating multiple competing hypotheses for neural computation by developing a method that overcomes dynamic collapse. Their approach enables production of genuinely unique solutions that can be evaluated against experimental data, moving beyond the limitations of varying only basic hyperparameters.

10 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

Iterative Neural Similarity Deflation (INSD) method

The authors introduce INSD, a training procedure that penalizes linear predictivity of neural activity from previously trained RNNs in an iterative manner. This method enables discovery of alternative task solutions that diverge from the prototypical solutions typically found due to simplicity bias in RNNs.

Contribution

Discovery of alternative RNN solutions beyond simplicity bias

The authors demonstrate that their method uncovers a distinct class of solutions to neuroscience tasks that differ from standard solutions in representational geometry, dynamical motifs, and encoding of task variables. These alternative solutions forgo fixed-point attractors and instead maintain information in dynamically evolving subspaces.

Contribution

Framework for generating diverse computational hypotheses in neuroscience

The authors address the problem of generating multiple competing hypotheses for neural computation by developing a method that overcomes dynamic collapse. Their approach enables production of genuinely unique solutions that can be evaluated against experimental data, moving beyond the limitations of varying only basic hyperparameters.

Table of Contents