Transformers Don’t Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and Implications for Mechanistic Interpretability

The paper demonstrates that layer normalization can be removed from pretrained GPT-2 models through fine-tuning, achieving minimal performance degradation. It resides in the 'Post-Training LayerNorm Removal via Fine-Tuning' leaf, which contains only three papers total within a taxonomy of 19 papers across seven leaf nodes. This represents a moderately sparse research direction focused specifically on post-hoc removal strategies, distinguishing it from the broader field of normalization analysis or training-time architectural modifications. The work's positioning suggests it addresses a relatively focused question within the larger normalization debate.

The taxonomy reveals three major branches: removal techniques, analytical studies, and application-specific optimizations. The paper's leaf sits within the removal branch alongside alternative normalization methods like RMSNorm replacements. Neighboring leaves include architectural analysis examining LayerNorm placement and outlier feature interactions, plus domain-specific optimizations for privacy-preserving inference and compression. The scope boundaries indicate this work differs from training-from-scratch approaches and from studies merely analyzing normalization's role without proposing removal. Its sibling papers similarly explore post-training elimination strategies, suggesting a coherent subfield examining whether pretrained models can shed normalization layers.

Among seven candidates examined across three contributions, four refutable pairs were identified. The core removal technique examined three candidates with two appearing to provide overlapping prior work. The open-source model suite examined two candidates with one refutable match, while the interpretability validation similarly found one refutable candidate among two examined. This limited search scope—seven total candidates rather than dozens—means the analysis captures immediate neighbors but cannot claim exhaustive coverage. The interpretability contribution appears relatively less explored in the examined literature, while the removal methodology itself encounters more substantial prior work within this constrained sample.

Based on examination of seven semantically related candidates, the work appears to operate in a moderately explored space with identifiable prior art in post-training removal techniques. The interpretability angle and scaling analysis may offer distinguishing elements, though the limited search scope prevents definitive assessment of their novelty. The taxonomy structure suggests this represents incremental progress within an established research direction rather than opening entirely new territory, though the specific combination of removal, release, and interpretability testing may differentiate it from immediate predecessors.