Cost-of-Pass: An Economic Framework for Evaluating Language Models

The paper introduces cost-of-pass, a metric combining accuracy and inference cost to evaluate language models economically. It resides in the 'Cost-Accuracy Tradeoff Metrics and Theoretical Frameworks' leaf, which contains only two papers total. This sparse population suggests the research direction—formal economic frameworks for LLM evaluation—remains relatively underdeveloped compared to optimization-heavy branches like Model Optimization and Compression Techniques or Inference Acceleration Methods, which collectively house over twenty papers. The sibling paper examines training-time scaling laws, whereas this work focuses on deployment-stage cost-effectiveness.

The taxonomy reveals neighboring leaves addressing related but distinct concerns. 'Empirical Model Selection and Routing Systems' contains six papers on dynamic query routing, emphasizing operational deployment rather than theoretical metrics. 'Structural Pruning and Parameter Reduction' and 'Quantization and Low-Rank Approximation' focus on model compression without explicit cost-accuracy formalization. The scope note for the parent branch clarifies that technical optimization methods lacking formal tradeoff frameworks belong elsewhere, positioning this work as foundational theory rather than applied technique. The frontier cost-of-pass concept bridges economic theory and practical model comparison.

Among twenty-one candidates examined, none clearly refute the three core contributions. The cost-of-pass metric examined ten candidates with zero refutations; the frontier framework with human-expert baseline similarly found no overlapping prior work across ten candidates; counterfactual frontier analysis examined one candidate without refutation. This limited search scope—top-K semantic matches plus citation expansion—suggests the contributions appear novel within the examined sample. The sparse sibling count and absence of refutations across all contributions indicate the work occupies relatively unexplored conceptual territory, though exhaustive literature review would strengthen this assessment.

Given the constrained search and sparse taxonomy leaf, the work appears to introduce genuinely new evaluation constructs. The economic grounding and frontier-based analysis distinguish it from empirical routing systems or compression benchmarks. However, the analysis covers only twenty-one candidates from semantic search, leaving open whether broader surveys or domain-specific venues contain related frameworks. The taxonomy structure itself—showing minimal prior work in formal cost-accuracy metrics—corroborates the impression of novelty within the examined scope.