OpenNovelty.org

Training LLMs with LogicReward for Faithful and Rigorous Reasoning

ICLR 2026 Conference SubmissionAnonymous Authors
OpenReview Score: 6.5Download Report PDF
reasoninglogicsymbolic
Abstract:

Although LLMs exhibit strong reasoning capabilities, existing training methods largely depend on outcome-based feedback, which can produce correct answers with flawed reasoning. Prior work introduces supervision on intermediate steps but still lacks guarantees of logical soundness, which is crucial in high-stakes scenarios where logical consistency is paramount. To address this, we propose LogicReward, a novel reward system that guides model training by enforcing step-level logical correctness with a theorem prover. We further introduce Autoformalization with Soft Unification, which reduces natural language ambiguity and improves formalization quality, enabling more effective use of the theorem prover. An 8B model trained on data constructed with LogicReward surpasses GPT-4o and o4-mini by 11.6% and 2% on natural language inference and logical reasoning tasks with simple training procedures. Further analysis shows that LogicReward enhances reasoning faithfulness, improves generalizability to unseen tasks such as math and commonsense reasoning, and provides a reliable reward signal even without ground-truth labels. We will release all data and code upon acceptance

Disclaimer
This report is AI-GENERATED using Large Language Models and WisPaper (A scholar search engine). It analyzes academic papers' tasks and contributions against retrieved prior work. While this system identifies POTENTIAL overlaps and novel directions, ITS COVERAGE IS NOT EXHAUSTIVE AND JUDGMENTS ARE APPROXIMATE. These results are intended to assist human reviewers and SHOULD NOT be relied upon as a definitive verdict on novelty.
NOTE that some papers exist in multiple, slightly different versions (e.g., with different titles or URLs). The system may retrieve several versions of the same underlying work. The current automated pipeline does not reliably align or distinguish these cases, so human reviewers will need to disambiguate them manually.
If you have any questions, please contact: mingzhang23@m.fudan.edu.cn

Overview

Overall Novelty Assessment

The paper introduces LogicReward, a reward system that enforces step-level logical correctness using a theorem prover, and Autoformalization with Soft Unification to improve natural language formalization. It resides in the 'Reward-Based and Reinforcement Learning Approaches' leaf under 'Training-Based Methods for Reasoning Enhancement,' alongside three sibling papers. This leaf represents a focused but active research direction within a broader taxonomy of 50 papers across 36 topics, indicating moderate concentration in reward-driven training methods for logical reasoning.

The taxonomy reveals that this work sits at the intersection of training-based and neurosymbolic approaches. Neighboring leaves include 'Knowledge Distillation and Model Compression' and 'Instruction Tuning for Logical Reasoning' within the same branch, while the 'Neurosymbolic Reasoning Integration' branch explores symbolic translation and solver integration without parameter updates. The paper's use of a theorem prover for reward signals bridges these areas, combining training-time optimization with formal verification tools. The taxonomy's scope notes clarify that this leaf excludes supervised fine-tuning without reward mechanisms and inference-time-only prompting methods.

Among 29 candidates examined, the analysis identified limited prior work overlap. The core LogicReward contribution examined 9 candidates, with 1 appearing to provide overlapping prior work on step-level logical verification. Autoformalization with Soft Unification examined 10 candidates with no clear refutations, suggesting this formalization technique may be more distinctive. The performance claim examined 10 candidates without refutation, though benchmark results are inherently time-sensitive. The search scope was constrained to top-K semantic matches plus citation expansion, not an exhaustive survey of all reward-based reasoning methods.

Based on this limited search, the work appears to occupy a relatively novel position by combining theorem prover-based rewards with autoformalization techniques. The single refutable candidate for LogicReward suggests some conceptual overlap exists in step-level verification approaches, but the specific integration with soft unification and the reported performance gains may differentiate this work. The analysis does not cover all possible prior work in formal verification for language models or recent concurrent developments in this rapidly evolving area.

Taxonomy

Core-task Taxonomy Papers
50
3
Claimed Contributions
29
Contribution Candidate Papers Compared
1
Refutable Paper

Research Landscape Overview

Core task: training language models for logically valid and faithful reasoning. The field has organized itself into several major branches that reflect different strategies for improving reasoning capabilities. Prompting-Based Reasoning Enhancement explores how carefully designed prompts—such as Chain-of-thought Prompting[27] and Zero-shot Reasoners[1]—can elicit better reasoning without modifying model weights. Neurosymbolic Reasoning Integration combines neural networks with symbolic solvers to enforce logical consistency, while Training-Based Methods for Reasoning Enhancement focuses on fine-tuning and reinforcement learning to directly optimize reasoning behavior. Evaluation and Analysis branches examine how well models actually reason, often revealing gaps between surface accuracy and true logical validity. Adversarial and Deceptive Reasoning Scenarios probe robustness under misleading contexts, and Faithful Reasoning via Modular Architectures investigates decomposing reasoning into verifiable steps. Surveys and Comprehensive Reviews synthesize these diverse threads, as seen in works like Reasoning Survey[7] and Trustworthiness Survey[43]. Within the Training-Based Methods branch, a particularly active line of work uses reward signals and reinforcement learning to guide models toward more reliable reasoning. LogicReward[0] exemplifies this approach by designing reward functions that explicitly encourage logical validity, situating itself among other reward-driven techniques like Step-aware Verifier[2] and Collaborative Verification[41]. Nearby, SuperCorrect[46] and Search-Based Correction[48] explore iterative refinement strategies that combine training with search or self-correction mechanisms. A central tension across these methods is balancing the efficiency of end-to-end learning against the interpretability and guarantees offered by more structured or symbolic approaches. LogicReward[0] addresses this by embedding logical constraints directly into the reward structure, contrasting with works like Self-verification[3] that rely on the model's own outputs to judge correctness. This cluster highlights ongoing questions about how best to align training objectives with the nuanced requirements of faithful, step-by-step reasoning.

Claimed Contributions

LogicReward: A novel reward system enforcing step-level logical correctness

The authors introduce LogicReward, a reward mechanism that evaluates reasoning chains for both premise validity (grounding in given context) and logic validity (formal logical soundness verified by a theorem prover). This provides step-level supervision with formal logical guarantees, unlike outcome-based or probabilistic process rewards.

9 retrieved papers
Can Refute
Autoformalization with Soft Unification

The authors propose a method that prompts LLMs to supplement implicit assumptions and reduce ambiguities in natural language reasoning steps before converting them to symbolic logic. This improves the success rate of theorem-prover verification by making implicit information explicit.

10 retrieved papers
State-of-the-art performance on NLI and logical reasoning benchmarks

By constructing training datasets using LogicReward and applying standard SFT and DPO procedures, the authors achieve new state-of-the-art results on natural language inference and logical reasoning tasks, demonstrating the effectiveness of their approach with relatively simple training methods.

10 retrieved papers

Core Task Comparisons

Comparisons with papers in the same taxonomy category

Contribution Analysis

Detailed comparisons for each claimed contribution

Contribution

LogicReward: A novel reward system enforcing step-level logical correctness

The authors introduce LogicReward, a reward mechanism that evaluates reasoning chains for both premise validity (grounding in given context) and logic validity (formal logical soundness verified by a theorem prover). This provides step-level supervision with formal logical guarantees, unlike outcome-based or probabilistic process rewards.

Contribution

Autoformalization with Soft Unification

The authors propose a method that prompts LLMs to supplement implicit assumptions and reduce ambiguities in natural language reasoning steps before converting them to symbolic logic. This improves the success rate of theorem-prover verification by making implicit information explicit.

Contribution

State-of-the-art performance on NLI and logical reasoning benchmarks

By constructing training datasets using LogicReward and applying standard SFT and DPO procedures, the authors achieve new state-of-the-art results on natural language inference and logical reasoning tasks, demonstrating the effectiveness of their approach with relatively simple training methods.

Table of Contents