OpenNovelty.org

Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.5Download Report PDF
information theorymutual informationsliced mutual informationcurse of dimensionality
Abstract:

Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence (even under linear transformations designed to enhance the extraction of information), prioritizes redundancy over informative content, and in some cases, performs worse than simpler dependence measures like the correlation coefficient.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper contributes a critical analysis of Sliced Mutual Information (SMI), identifying saturation behavior, sensitivity failures, and redundancy bias as fundamental limitations. Within the taxonomy, it occupies the 'Limitations and Critical Analysis' leaf under 'Theoretical Foundations and Extensions of Sliced Mutual Information'. Notably, this leaf contains only the original paper itself—no sibling papers exist in this category. This isolation suggests that systematic critical examination of SMI's failure modes represents a sparse research direction, contrasting sharply with the more populated leaves addressing SMI variants and applications.

The taxonomy reveals substantial activity in neighboring areas: 'Max-Sliced Mutual Information' contains four papers exploring optimality conditions, while 'k-Sliced and Higher-Dimensional Extensions' includes two papers on dimensional scalability. The 'Core Sliced Mutual Information Theory' leaf holds one foundational paper. The original work diverges from these directions by questioning SMI's reliability rather than extending its capabilities. Its scope explicitly excludes positive theoretical developments (which belong in sibling leaves) and application-specific issues (which belong under 'Applications to Deep Learning Analysis' or 'Domain-Specific Applications'), focusing instead on intrinsic theoretical and practical limitations.

Among twelve candidates examined through limited semantic search, the 'Saturation and Sensitivity Analysis' contribution shows one potentially refutable candidate from six examined, while 'Redundancy Bias' was examined against zero candidates, and 'Curse of Dimensionality' found no refutations among six candidates. The statistics indicate that within this restricted search scope, most contributions lack substantial overlapping prior work. The saturation analysis appears most vulnerable to existing literature, though even here only one candidate among six provides potential overlap. The redundancy bias and dimensionality curse contributions appear more novel within the examined sample.

Based on the top-twelve semantic matches and taxonomy structure, the work addresses an underexplored critical perspective within SMI research. The analysis does not claim exhaustive coverage of all possible prior work, and the limited search scope means additional relevant papers may exist outside the examined candidates. The taxonomy's sparse 'Limitations' leaf and the low refutation rates suggest the critical angle is relatively unexplored, though definitive novelty claims require broader literature review.

Taxonomy

Core-task Taxonomy Papers
22
3
Claimed Contributions
12
Contribution Candidate Papers Compared
1
Refutable Paper

Research Landscape Overview

Core task: measuring statistical dependence with sliced mutual information. The field centers on sliced mutual information (SMI) as a computationally tractable alternative to classical mutual information for quantifying dependence in high-dimensional settings. The taxonomy reveals four main branches: theoretical foundations explore the mathematical properties and extensions of SMI, including optimality conditions and interpretations; estimation and computational methods address scalable algorithms and practical implementations; applications to deep learning analysis leverage SMI to understand neural network behavior, memorization, and generalization; and domain-specific applications extend these tools to specialized problems such as computational imaging. Representative works like Max-Sliced Mutual Information[3] and k-Sliced Mutual Information[19] illustrate how different slicing strategies balance computational efficiency with theoretical guarantees, while Sliced MI Deep Networks[2] and Sliced Information Plane[5] demonstrate the utility of SMI for analyzing learned representations. A particularly active line of work examines the trade-offs between different slicing approaches, with studies like Slicing Optimality[4] and Max-Sliced Interpretations[1] investigating when and why certain projection strategies succeed or fail. Another thread focuses on practical estimation challenges, where methods such as k-Sliced MI Scalability[13] and Scalable Infomin Learning[14] aim to make SMI feasible for large-scale data. The original paper, Curse of Slicing[0], sits within the theoretical foundations branch and specifically addresses limitations and critical analysis of SMI. It contrasts with works like Max-Sliced Mutual Information[3] by examining potential pitfalls or failure modes of slicing-based approaches, rather than proposing new variants. This critical perspective complements the broader landscape by questioning the conditions under which SMI reliably captures dependence, thereby informing both theoretical refinements and practical deployment decisions.

Claimed Contributions

Saturation and Sensitivity Analysis of SMI

The authors demonstrate both theoretically and empirically that sliced mutual information reaches a plateau early and becomes insensitive to further increases in statistical dependence, even in simple low-dimensional settings. This saturation behavior undermines SMI's ability to accurately track changes in dependence structure.

6 retrieved papers
Can Refute
Redundancy Bias of SMI

The authors provide a counterexample showing that SMI does not favor linearly extractable information as previously believed. Instead, they reveal that SMI prioritizes information redundancy over information content, which can lead to catastrophic failures in applications like representation learning.

0 retrieved papers
Curse of Dimensionality for SMI

The authors reinterpret the curse of dimensionality for SMI, showing that while sample complexity may be favorable, SMI uniformly decays to zero as dimensionality increases. This decay occurs due to diminishing redundancy and makes SMI ineffective for statistical analysis in high-dimensional settings.

6 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Within the taxonomy built over the current TopK core-task papers, the original paper is assigned to a leaf with no direct siblings and no cousin branches under the same grandparent topic. In this retrieved landscape, it appears structurally isolated, which is one partial signal of novelty, but still constrained by search coverage and taxonomy granularity.

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

Saturation and Sensitivity Analysis of SMI

The authors demonstrate both theoretically and empirically that sliced mutual information reaches a plateau early and becomes insensitive to further increases in statistical dependence, even in simple low-dimensional settings. This saturation behavior undermines SMI's ability to accurately track changes in dependence structure.

Contribution

Redundancy Bias of SMI

The authors provide a counterexample showing that SMI does not favor linearly extractable information as previously believed. Instead, they reveal that SMI prioritizes information redundancy over information content, which can lead to catastrophic failures in applications like representation learning.

Contribution

Curse of Dimensionality for SMI

The authors reinterpret the curse of dimensionality for SMI, showing that while sample complexity may be favorable, SMI uniformly decays to zero as dimensionality increases. This decay occurs due to diminishing redundancy and makes SMI ineffective for statistical analysis in high-dimensional settings.

Table of Contents