OpenNovelty.org

Exploring Mode Connectivity in Krylov Subspace for Domain Generalization

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.0Download Report PDF
Loss landscapemode connectivityKrylov spacedomain generalization
Abstract:

This paper explores the geometric characteristics of loss landscapes to enhance domain generalization (DG) in deep neural networks. Existing methods mainly leverage the local flatness around minima for improved generalization. However, recent theoretical studies indicate that flatness does not universally guarantee better generalization. Instead, this paper investigates a global geometrical property for domain generalization, i.e., mode connectivity, the phenomenon where distinct local minima are connected by continuous low-loss pathways. Different from flatness, mode connectivity enables transitions from poor to superior generalization models without leaving low-loss regions. To navigate these connected pathways effectively, this paper proposes a novel Billiard Optimization Algorithm (BOA), which discovers superior models by mimicking billiard dynamics. During this process, BOA operates within a low-dimensional Krylov subspace, aiming to alleviate the curse of dimensionality caused by the high-dimensional parameter space of deep models. Furthermore, this paper reveals that oracle test gradients strongly align with the Krylov subspace constructed from training gradients across diverse datasets and architectures. This alignment offers a powerful tool to bridge training and test domains, enabling the efficient discovery of superior models with limited training domains. Experiments on DomainBed demonstrate that BOA consistently outperforms existing sharpness-aware and DG methods across diverse datasets and architectures. Impressively, BOA even surpasses the sharpness-aware minimization by 3.6% on VLCS when using a ViT-B/16 backbone.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper proposes a Billiard Optimization Algorithm (BOA) that navigates mode connectivity pathways in Krylov subspaces to improve domain generalization. According to the taxonomy tree, this work occupies the 'Mode Connectivity for Domain Generalization' leaf under 'Domain Generalization via Loss Landscape Geometry'. Notably, this leaf contains only the original paper itself—no sibling papers are listed. This suggests the specific combination of mode connectivity exploration and domain generalization represents a relatively sparse research direction within the broader landscape geometry literature, which includes more populated areas like flatness-based optimization and model merging.

The taxonomy reveals neighboring research directions that provide important context. The sibling leaf 'Flatness-Based Optimization for Generalization' contains three papers exploring sharpness-aware minimization and loss landscape refinement for out-of-domain performance. Meanwhile, the 'Mode Connectivity Theory and Characterization' branch investigates fundamental connectivity properties across architectures, including linear versus non-linear paths and architecture-specific patterns. The paper's focus on global geometric properties (mode connectivity) rather than local properties (flatness) positions it at the intersection of theoretical connectivity characterization and practical domain generalization applications, bridging these two major branches.

Among twenty candidates examined through limited semantic search, the contribution-level analysis reveals mixed novelty signals. The core BOA algorithm contribution examined five candidates and found one potentially refutable prior work, suggesting some algorithmic overlap exists within the limited search scope. In contrast, the Krylov subspace alignment contribution examined five candidates with zero refutations, and the mode connectivity for domain generalization contribution examined ten candidates with zero refutations. These statistics indicate that while the algorithmic mechanism may have precedents, the specific application of Krylov-based mode connectivity navigation to domain generalization appears less explored among the examined candidates.

Based on the limited search scope of twenty semantically similar papers, the work appears to occupy a relatively novel position combining mode connectivity theory with domain generalization practice. However, the analysis explicitly covers only top-K semantic matches and does not constitute an exhaustive literature review. The single-paper taxonomy leaf and zero refutations for two of three contributions suggest potential novelty, though the limited search scale and one algorithmic refutation warrant careful interpretation of these signals.

Taxonomy

Core-task Taxonomy Papers
24
3
Claimed Contributions
20
Contribution Candidate Papers Compared
1
Refutable Paper

Research Landscape Overview

Core task: domain generalization through loss landscape mode connectivity. The field explores how neural networks trained on different domains or tasks can be connected through low-loss paths in parameter space, and how this connectivity relates to generalization performance. The taxonomy reveals several major branches: theoretical characterizations of mode connectivity itself, applications to domain generalization via loss landscape geometry, techniques for merging multiple models, methods for sequential and continual learning, analyses of optimization dynamics and spectral properties, comprehensive surveys, and domain-specific applications. Works like Linear Connectivity Generalization[5] and Transformer Mode Connectivity[12] investigate fundamental connectivity properties across architectures, while studies such as Parameter Competition Balancing[3] and Consistent Flat Minima[18] examine how landscape flatness and geometry influence out-of-distribution robustness. Meanwhile, branches on multi-model integration explore practical merging strategies (Training-free Model Merging[4], Model Merging Survey[20]), and continual adaptation methods address sequential learning scenarios. Particularly active themes include understanding the relationship between connectivity structure and generalization, with some works emphasizing geometric properties of connecting paths (Bezier Surface Connectivity[2], Path Flatness Optimization[14]) while others focus on efficient computation or federated settings (FedGuCci[1], Group Connectivity Federated[9]). A central tension involves balancing theoretical insights about loss landscape topology with practical algorithmic design. Krylov Mode Connectivity[0] sits within the domain generalization via loss landscape geometry branch, specifically addressing mode connectivity for domain generalization. Its emphasis on computational efficiency through Krylov subspace methods distinguishes it from nearby works like Linear Connectivity Generalization[5], which focuses on theoretical conditions for linear connectivity, and from Parameter Competition Balancing[3], which tackles multi-domain scenarios through parameter-level competition mechanisms. The work contributes to an emerging understanding of how connectivity paths can be efficiently discovered and exploited for improved cross-domain performance.

Claimed Contributions

Billiard Optimization Algorithm (BOA) for domain generalization

The authors introduce BOA, a novel optimization algorithm that navigates low-loss pathways connecting distinct local minima in the loss landscape. BOA operates through two core operations: line search to locate loss contour boundaries and reflection to redirect optimization trajectories, mimicking billiard ball dynamics to discover models with superior domain generalization capabilities.

5 retrieved papers
Can Refute
Krylov subspace alignment for bridging training and test domains

The authors discover a geometric regularity where test gradients exhibit strong alignment with the Krylov subspace derived from training gradients. This alignment provides a mechanism to bridge training and test domains, enabling efficient discovery of superior models within a low-dimensional subspace (5-20 dimensions) without requiring access to test data.

5 retrieved papers
Mode connectivity for domain generalization

The authors investigate mode connectivity as a global geometric property for domain generalization, demonstrating that models with significantly different out-of-domain performance are connected by continuous low-loss pathways. This property enables transitions from poor to superior generalization models without leaving low-loss regions, offering an alternative to local flatness-based approaches.

10 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Within the taxonomy built over the current TopK core-task papers, the original paper is assigned to a leaf with no direct siblings and no cousin branches under the same grandparent topic. In this retrieved landscape, it appears structurally isolated, which is one partial signal of novelty, but still constrained by search coverage and taxonomy granularity.

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

Billiard Optimization Algorithm (BOA) for domain generalization

The authors introduce BOA, a novel optimization algorithm that navigates low-loss pathways connecting distinct local minima in the loss landscape. BOA operates through two core operations: line search to locate loss contour boundaries and reflection to redirect optimization trajectories, mimicking billiard ball dynamics to discover models with superior domain generalization capabilities.

Contribution

Krylov subspace alignment for bridging training and test domains

The authors discover a geometric regularity where test gradients exhibit strong alignment with the Krylov subspace derived from training gradients. This alignment provides a mechanism to bridge training and test domains, enabling efficient discovery of superior models within a low-dimensional subspace (5-20 dimensions) without requiring access to test data.

Contribution

Mode connectivity for domain generalization

The authors investigate mode connectivity as a global geometric property for domain generalization, demonstrating that models with significantly different out-of-domain performance are connected by continuous low-loss pathways. This property enables transitions from poor to superior generalization models without leaving low-loss regions, offering an alternative to local flatness-based approaches.

Table of Contents