Scalable Oversight via Partitioned Human Supervision

Core task: evaluating and training AI systems using complementary labels from domain experts. The field organizes around five main branches that reflect different facets of expert-driven supervision. Complementary-Label Learning Frameworks and Theory develops formal methods for learning from partial or indirect annotations, addressing scalability and bias when experts cannot label every instance exhaustively. Medical Imaging and Diagnostic AI Evaluation focuses on radiology, pathology, and clinical imaging tasks where expert radiologists or pathologists provide ground truth or quality assessments, as seen in works like ARTHUR DIANA Ultrasound[4] and Follicular Lymphoma Grading[9]. Clinical Decision Support and Language Model Evaluation examines how large language models can be assessed or guided by clinicians, for instance through LLM Medical Graders[3] or consensus protocols like LLM Clinical Consensus[34]. General AI Evaluation and Annotation Quality tackles cross-domain challenges such as handling inconsistent annotations (Inconsistent Human Annotations[26]), defining what constitutes expertise (Defining Expert[27]), and ensuring robust evaluation pipelines (AI Ground Truth[5]). Domain-Specific Applications and Specialized Tasks spans diverse settings from wildlife monitoring (Wildlife Camera Traps[20]) to surgical skill assessment (Surgical Expertise Assessment[10]), illustrating how expert feedback adapts to varied problem contexts. Several active lines of work explore trade-offs between annotation cost, label quality, and model performance. One recurring theme is how to aggregate or reconcile disagreements among multiple experts (Multiple Expert Annotators[22], Heterogeneous Expert Consistency[7]) while preserving the nuanced information that partial or complementary labels provide. Another thread investigates hybrid supervision strategies that combine weak labels, semi-automated tools, and targeted expert input (Hybrid Supervision Radiographs[40], Semi-Automated Quality Assurance[42]). Partitioned Human Supervision[0] sits within the Scalable Oversight and Unbiased Estimation cluster, emphasizing methods that partition the supervision task to reduce expert burden while maintaining unbiased learning guarantees. This approach contrasts with works like Recycling Weak Labels[39], which reuse noisy annotations more opportunistically, and aligns closely with efforts such as Measurement Error Correction[32] that formally account for imperfect or incomplete expert signals. The central question across these directions is how to design training and evaluation protocols that respect expert time constraints without sacrificing the reliability or interpretability that domain knowledge brings.