OpenNovelty.org

DP-Fusion: Token-Level Differentially Private Inference for Large Language Models

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.0Download Report PDF
PrivacyLarge Language ModelsDocument Privatization
Abstract:

Large language models (LLMs) do not preserve privacy at inference-time. The LLM's outputs can inadvertently reveal information about the model's context, which presents a privacy challenge when the LLM is augmented via tools or databases containing sensitive information. Existing privacy-preserving methods at inference-time have significant limitations since they (i) lack provable guarantees or (ii) have a poor utility/privacy trade-off. We propose DP-Fusion, a Differentially Private Inference (DPI) mechanism for LLMs that provably bounds the influence a set of tokens in the context can have on the LLM's output. DP-Fusion works as follows: (1) label a subset of sensitive tokens, (2) infer the LLM without any sensitive tokens to obtain a baseline, (3) infer the LLM with the sensitive tokens, and (4) blend distributions so that the final output remains within a bounded distance of the baseline distribution. While this per-token influence bound also mitigates jailbreak-style prompt injection, we focus on document privatization, where the goal is to paraphrase a document containing sensitive tokens, e.g., personally identifiable information, so that no attacker can reliably infer them from the paraphrased document while preserving high text quality. The privacy/utility trade-off is controlled by ϵ\epsilon, where ϵ=0\epsilon=0 hides sensitive tokens entirely, while higher values trade off privacy for improved text quality. We show that our method creates token-level provably privatized documents with substantially improved theoretical and empirical privacy, achieving 6×6\times lower perplexity than related DPI methods.

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper introduces DP-Fusion, a mechanism for differentially private LLM inference that bounds the influence of sensitive tokens on generated outputs. It resides in the 'Differential Privacy for Next-Token Prediction' leaf, which contains five papers total, including the original work. This leaf sits within the broader 'Privacy-Preserving Inference Mechanisms' branch, indicating a moderately populated research direction focused on runtime privacy protections. The taxonomy shows this is an active but not overcrowded area, with sibling papers exploring related noise-injection and decoding strategies for autoregressive generation.

The taxonomy reveals that DP-Fusion's leaf is one of four under Privacy-Preserving Inference Mechanisms, alongside 'Privacy-Preserving In-Context Learning and Prompting' (five papers), 'Cryptographic and Secure Computation for Inference' (four papers), and 'Instance Obfuscation and Masking for Inference Privacy' (two papers). These neighboring leaves address complementary challenges: protecting prompts and exemplars, leveraging cryptographic primitives, or perturbing inputs rather than outputs. The scope note for the parent branch explicitly excludes training-time privacy, clarifying that DP-Fusion's focus on inference-time token influence bounds distinguishes it from adaptation or fine-tuning methods found in other taxonomy branches.

Among twenty-three candidates examined across three contributions, no refutable prior work was identified. The core DP-Fusion mechanism examined ten candidates with zero refutations, the document privatization application examined ten candidates with zero refutations, and the per-group privacy budget framework examined three candidates with zero refutations. This limited search scope—top-K semantic matches plus citation expansion—suggests that within the examined set, the fusion-based approach and per-token influence bounding appear distinct from prior noise-injection or decoding strategies. However, the analysis does not claim exhaustive coverage of all differential privacy inference techniques in the broader literature.

Based on the limited search of twenty-three candidates, DP-Fusion appears to occupy a recognizable niche within differential privacy for next-token prediction, with no clear overlap detected in the examined set. The taxonomy context indicates a moderately active research direction with established sibling work on noise calibration and adaptive budgets, suggesting the paper builds on known challenges in balancing privacy and utility during autoregressive generation. The absence of refutations in this scope does not preclude related work outside the top-K matches or in adjacent taxonomy leaves.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
23
Contribution Candidate Papers Compared
0
Refutable Paper

Research Landscape Overview

Core task: differentially private inference for large language models. The field addresses how to deploy LLMs while protecting sensitive information in user queries, model outputs, or training data. The taxonomy reveals a broad landscape organized around several complementary themes. Privacy-Preserving Inference Mechanisms focus on runtime protections during model serving, including techniques that add noise to next-token predictions or embeddings. Privacy-Preserving Model Adaptation and Fine-Tuning explore how to customize models on private data without leaking individual records, often via differentially private training or parameter-efficient methods. Privacy-Preserving Data Generation and Sharing examine synthetic text creation and secure data exchange protocols. Privacy Risk Analysis and Attack Methods investigate vulnerabilities such as membership inference and prompt extraction, while Comprehensive Privacy Frameworks and Multi-Technique Approaches combine cryptographic, differential privacy, and federated learning tools. Surveys and Overviews synthesize these directions, Domain-Specific Privacy Applications target sectors like healthcare, Distributed and Federated Privacy-Preserving LLM Systems address decentralized settings, and Conceptual and Theoretical Privacy Perspectives provide foundational analysis. Within Privacy-Preserving Inference Mechanisms, a particularly active line of work targets differential privacy for next-token prediction, where the challenge is to add calibrated noise to autoregressive generation without destroying output quality. DP-Fusion[0] sits squarely in this cluster, proposing a fusion-based approach to balance privacy and utility during token sampling. It shares thematic ground with Private Decoding[5], which introduced early noise-injection strategies for decoding, and Private Next-Token[7], which refined noise calibration for sequential predictions. Nearby efforts like Submix[6] and Adaptively Private Prediction[13] explore alternative noise mechanisms and adaptive privacy budgets, highlighting ongoing trade-offs between tight privacy guarantees and coherent text generation. These works collectively grapple with the tension between strong formal privacy and the autoregressive nature of LLMs, a challenge that distinguishes inference-time protections from training-time or data-sharing approaches elsewhere in the taxonomy.

Claimed Contributions

DP-FUSION mechanism for token-level differentially private LLM inference

The authors introduce DP-FUSION, a novel differentially private inference mechanism that provides provable token-level privacy guarantees for large language models. The method works by inferring the LLM with and without sensitive tokens, then blending the output distributions to bound the influence of sensitive tokens on generated outputs.

10 retrieved papers
Document privatization application with improved privacy-utility trade-off

The authors apply DP-FUSION to document privatization, demonstrating that their method can paraphrase documents containing personally identifiable information while achieving substantially better privacy-utility trade-offs than existing methods, with 6× lower perplexity than related DPI approaches.

10 retrieved papers
Per-group privacy budget framework with parallelizable inference

The authors develop a framework that allows assigning different privacy budgets to different groups of sensitive tokens and implements a parallelizable inference procedure that computes multiple distributions (one public and multiple private) per generation step, enabling efficient token-level privacy control.

3 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

DP-FUSION mechanism for token-level differentially private LLM inference

The authors introduce DP-FUSION, a novel differentially private inference mechanism that provides provable token-level privacy guarantees for large language models. The method works by inferring the LLM with and without sensitive tokens, then blending the output distributions to bound the influence of sensitive tokens on generated outputs.

Contribution

Document privatization application with improved privacy-utility trade-off

The authors apply DP-FUSION to document privatization, demonstrating that their method can paraphrase documents containing personally identifiable information while achieving substantially better privacy-utility trade-offs than existing methods, with 6× lower perplexity than related DPI approaches.

Contribution

Per-group privacy budget framework with parallelizable inference

The authors develop a framework that allows assigning different privacy budgets to different groups of sensitive tokens and implements a parallelizable inference procedure that computes multiple distributions (one public and multiple private) per generation step, enabling efficient token-level privacy control.

Table of Contents