FACT: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations

The paper proposes the Features at Convergence Theorem (FACT) as a first-principles alternative to the empirically-driven Neural Feature Ansatz (NFA), deriving feature learning mechanisms from optimization theory and convergence conditions. It resides in the 'First-Principles and Optimization-Based Analysis' leaf, which contains only two papers total within the broader theoretical foundations branch. This represents a relatively sparse research direction within the taxonomy, suggesting that rigorous optimization-theoretic approaches to feature learning remain underexplored compared to empirical or application-driven methods. The sibling paper in this leaf appears to focus on related mechanistic questions, indicating a small but coherent cluster of work examining fundamental learning dynamics.

The taxonomy reveals that theoretical feature learning research is organized into three main directions: first-principles analysis (where this work sits), dynamics and evolution of representations, and transferability studies. Neighboring branches include unsupervised learning methods and interpretability techniques, which analyze learned features post-hoc rather than modeling their formation. The scope note for this leaf explicitly excludes 'empirical observations without theoretical derivation,' positioning FACT as complementary to the larger body of empirical NFA literature. The work bridges optimization theory with the empirically-validated NFA framework, potentially connecting formal convergence analysis to observed learning phenomena like grokking and phase transitions.

Among the twenty-eight candidates examined through semantic search and citation expansion, none were identified as clearly refuting any of the three main contributions. The FACT theorem itself was evaluated against ten candidates with zero refutable matches, as was the FACT-based Recursive Feature Machine algorithm. The theoretical explanation connecting NFA to first-order optimality examined eight candidates, also with no refutations found. This limited search scope suggests that within the top semantic matches, no prior work explicitly derives convergence-based feature learning mechanisms from first-order optimality conditions in the manner proposed. However, the modest search scale means potentially relevant optimization-theoretic analyses outside this candidate set remain unexamined.

Based on the available signals, the work appears to occupy a relatively novel position within the limited scope examined, particularly in formalizing the empirical NFA through optimization theory. The sparse population of the first-principles analysis leaf and absence of refuting candidates among twenty-eight examined papers suggest limited direct prior work on this specific approach. However, the analysis covers only top semantic matches and does not constitute an exhaustive survey of optimization theory applied to neural network feature learning, leaving open the possibility of related theoretical frameworks in adjacent mathematical or machine learning literature.