OpenNovelty.org

Hubble: a Model Suite to Advance the Study of LLM Memorization

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 7.5Download Report PDF
memorizationcopyrightprivacytest set contaminationmembership inferenceunlearning
Abstract:

We present Hubble, a suite of open-source large language models (LLMs) for the scientific study of LLM memorization. Hubble models come as minimal pairs: standard models are pretrained on a large English corpus, and perturbed models are trained in the same way but with controlled insertion of text (e.g., book passages, biographies, and test sets) designed to emulate key memorization risks. Our core release includes 8 models---standard and perturbed, with 1B or 8B parameters, trained on 100B or 500B tokens. Hubble's core experiment establishes that memorization risks are determined by the frequency of sensitive data relative to the training corpus size (i.e., a password appearing once in a smaller corpus is memorized better than the same password in a larger corpus). Our release includes 6 more models with perturbations inserted at different pretraining phases; we observe perturbations without continued exposure can be forgotten. These findings suggest two best practices: to dilute sensitive data by increasing the training corpus size, and to order them to appear earlier in training. Beyond these general findings, Hubble enables a broad range of memorization research. We show that the randomized perturbations in Hubble make it an ideal testbed for membership inference and machine unlearning methods. We invite the community to explore, benchmark, and build upon our work.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper introduces Hubble, a suite of open-source LLMs designed to enable controlled study of memorization through minimal pairs—standard models versus perturbed models with inserted sensitive data. It sits within the Multi-Perspective Empirical Analysis leaf, which contains only three papers total. This leaf focuses on examining memorization through multiple analytical lenses such as embeddings, probability distributions, and input perturbations. The sparse population suggests this specific methodological approach—building dedicated model suites for multi-factor memorization experiments—is relatively underexplored compared to broader characterization work.

The taxonomy tree reveals that Hubble's leaf is nested under Characterization and Measurement of Memorization, which also includes Scaling Laws and Training Dynamics (six papers) and Taxonomies and Comprehensive Surveys (three papers). Neighboring branches address Mechanistic Analysis (examining internal circuits), Detection Methods (extraction attacks and membership inference), and Mitigation Strategies (unlearning and fine-tuning approaches). Hubble bridges characterization and mitigation by not only measuring memorization but also deriving best practices for dilution and ordering. Its infrastructure focus also connects to the separate Infrastructure and Datasets branch, though that category emphasizes general-purpose releases rather than memorization-specific experimental suites.

Among 21 candidates examined, the HUBBLE model suite contribution shows one refutable candidate out of ten examined, suggesting some prior infrastructure work exists but is not densely populated. The perturbation design contribution examined eight candidates with zero refutations, indicating this specific cross-domain approach (copyright, privacy, test contamination) may be less directly addressed in prior work. The best practices contribution examined three candidates with one refutation, implying that dilution and ordering strategies have been discussed but perhaps not systematically validated at this scale. The limited search scope means these findings reflect top-K semantic matches rather than exhaustive coverage.

Given the sparse Multi-Perspective Empirical Analysis leaf and the modest candidate pool examined, Hubble appears to occupy a relatively open methodological niche. The infrastructure contribution overlaps with existing model-suite work, but the integrated perturbation design and best-practice derivation seem less directly anticipated. However, the analysis is constrained by examining only 21 candidates; a broader search might reveal additional overlapping efforts in infrastructure or mitigation guidance. The taxonomy structure suggests the field is more crowded in detection and mechanistic analysis than in controlled experimental infrastructure.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
21
Contribution Candidate Papers Compared
2
Refutable Paper

Research Landscape Overview

Core task: scientific study of large language model memorization. The field has organized itself around several complementary perspectives. At the broadest level, researchers pursue characterization and measurement of memorization—establishing metrics and empirical frameworks to quantify when and how models retain training data—alongside mechanistic analysis that probes the internal circuits and representations underlying memorized content. Detection and extraction methods provide practical tools for identifying verbatim or near-verbatim reproductions, while mitigation strategies and privacy protection explore defenses such as differential privacy and data filtering. Domain-specific memorization studies examine how memorization manifests in specialized contexts like code or biomedical text, and in-context learning research investigates the interplay between few-shot prompting and retrieval of stored knowledge. Parallel branches address memory systems and architectures for LLM agents, interpretability and knowledge representation, and the infrastructure and datasets that enable large-scale memorization experiments. Within characterization and measurement, a particularly active line of work focuses on multi-perspective empirical analysis, examining memorization through lenses such as training dynamics, model scale, and data properties. Hubble[0] exemplifies this approach by conducting comprehensive empirical investigations across multiple factors, situating itself alongside Multi-Perspective Memorization[6] and Memorization or Interpolation[45], which similarly dissect how memorization varies with architecture choices and dataset characteristics. These studies reveal persistent tensions: larger models and longer training often increase memorization risk, yet the relationship between memorization and generalization remains nuanced, as highlighted by works like Quantifying Memorization[3] and Scaling Laws Memorization[5]. Hubble[0] contributes to this dense branch by offering a holistic view that integrates findings on frequency effects, model capacity, and emergent memorization patterns, helping to bridge isolated observations into a more unified understanding of when and why models remember.

Claimed Contributions

HUBBLE model suite for studying LLM memorization

The authors introduce HUBBLE, a collection of open-source language models designed specifically for controlled scientific study of memorization. The suite includes minimal pairs of standard and perturbed models with controlled insertion of text to emulate key memorization risks across copyright, privacy, and test set contamination domains.

10 retrieved papers
Can Refute
Perturbation design across copyright, privacy, and test set contamination domains

The authors systematically design and insert controlled perturbations into training data to emulate real-world memorization risks. These perturbations include book passages, biographies, chat logs, and test sets, inserted at varying frequencies to enable causal study of memorization phenomena.

8 retrieved papers
Best practices for mitigating memorization risks through dilution and ordering

Through controlled experiments, the authors establish that memorization risks can be reduced by two strategies: diluting sensitive data by training on larger corpora, and ordering sensitive data to appear early in training so it can be forgotten without continued exposure.

3 retrieved papers
Can Refute

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

HUBBLE model suite for studying LLM memorization

The authors introduce HUBBLE, a collection of open-source language models designed specifically for controlled scientific study of memorization. The suite includes minimal pairs of standard and perturbed models with controlled insertion of text to emulate key memorization risks across copyright, privacy, and test set contamination domains.

Contribution

Perturbation design across copyright, privacy, and test set contamination domains

The authors systematically design and insert controlled perturbations into training data to emulate real-world memorization risks. These perturbations include book passages, biographies, chat logs, and test sets, inserted at varying frequencies to enable causal study of memorization phenomena.

Contribution

Best practices for mitigating memorization risks through dilution and ordering

Through controlled experiments, the authors establish that memorization risks can be reduced by two strategies: diluting sensitive data by training on larger corpora, and ordering sensitive data to appear early in training so it can be forgotten without continued exposure.

Table of Contents

Hubble: a Model Suite to Advance the Study of LLM Memorization | Novelty Validation