Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas

The paper proposes a doubly-robust estimation framework to correct evaluation sampling bias in generative AI systems, combining synthetic persona ratings from LLM-as-a-judge with human ratings obtained under biased conditions. It resides in the 'Statistical Bias Correction in Evaluation' leaf, which contains only two papers total. This represents a sparse research direction within the broader taxonomy of 17 papers across 11 leaf nodes, suggesting the specific intersection of doubly-robust causal inference methods and GenAI evaluation remains relatively unexplored compared to adjacent areas like synthetic data generation or comprehensive quality assessment frameworks.

The taxonomy reveals neighboring work in generalizability assessment across clinical settings and active cost-aware evaluation, but these address different aspects of the external validity problem. Clinical generalizability studies focus on empirical performance measurement across deployment contexts without statistical bias correction machinery, while cost-aware evaluation optimizes resource allocation rather than correcting for sampling discrepancies. The paper's use of persona simulation connects conceptually to synthetic data generation branches, yet differs fundamentally: those methods augment training data or create evaluation samples, whereas this work uses synthetic ratings as auxiliary information within a causal inference framework to debias estimates from real human ratings.

Among 23 candidates examined across three contributions, none were identified as clearly refuting the proposed methods. The core doubly-robust framework examined 10 candidates with zero refutable matches, the M-estimation generalization examined 3 candidates with zero refutations, and the persona simulation framework examined 10 candidates with zero refutations. This suggests that within the limited search scope, the specific combination of doubly-robust estimation theory, LLM-generated persona ratings, and GenAI system quality assessment appears relatively novel. The statistical machinery itself draws from established causal inference literature, but its application to this evaluation context with synthetic persona ratings as surrogates represents a methodological contribution not directly anticipated by the examined prior work.

Based on the top-23 semantic matches and citation expansion, the work appears to occupy a methodologically distinct position at the intersection of causal inference and GenAI evaluation. The limited search scope means potentially relevant work in broader causal inference or survey methodology literatures may not be fully represented. The taxonomy structure indicates this statistical correction approach is less crowded than synthetic data generation or comprehensive quality assessment directions, though the sibling paper on doubly-robust LLM judges suggests emerging interest in applying rigorous statistical frameworks to address evaluation validity challenges in generative AI systems.