Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences

Core task: omni-modal reward modeling with free-form preferences. The field centers on learning reward functions that can evaluate agent outputs across diverse modalities—text, images, video, and structured data—using human or synthetic preference signals that are not constrained to rigid formats. The taxonomy reveals several major branches: Reward Model Architecture and Training Methodology explores how to build and train these models, including generalist frameworks that unify multiple modalities (e.g., Omni-Reward[0], InternLM-XComposer Reward[12], Unified Reward Multimodal[19]); Preference Optimization and Alignment Methods focuses on techniques like direct preference optimization and modality-balancing strategies (Modality-balancing Preference[4], mDPO[10]); Application Domains and Task-Specific Reward Modeling addresses specialized settings such as vision-language tasks, mathematical reasoning, and agentic environments (Visionreward[1], Agent-RewardBench[18]); Evaluation Benchmarks and Datasets provides standardized testbeds (Multimodal Rewardbench[6]); Surveys and Comparative Studies synthesize progress (Aligning Multimodal Survey[16]); and Cognitive and Behavioral Studies examine how humans process multimodal rewards (Multimodal Rewards Rankings[3], Multimodal Reward Cues[40]). A particularly active line of work involves generalist reward models that handle multiple modalities within a single architecture, aiming to capture cross-modal dependencies and avoid unimodal biases (Unimodal Spurious Correlations[22]). These models often leverage large-scale preference data and sophisticated training recipes to balance modality-specific signals. Omni-Reward[0] sits squarely in this generalist branch, emphasizing free-form preference inputs and broad applicability across tasks. It shares conceptual ground with InternLM-XComposer Reward[12] and Unified Reward Multimodal[19], which also pursue unified architectures, but differs in its explicit focus on flexible, unstructured preference formats rather than fixed rubrics (contrast Rubrics as Rewards[5]) or domain-specific tuning (BaseReward[29], Skywork-VL Reward[35]). A key open question across these efforts is how to efficiently scale generalist models while maintaining fine-grained sensitivity to task-specific nuances and avoiding the pitfalls of modality imbalance or spurious correlations.