AlphaBench: Benchmarking Large Language Models in Formulaic Alpha Factor Mining

The paper introduces AlphaBench, a systematic benchmark for evaluating large language models across three core tasks in formulaic alpha factor mining: factor generation, evaluation, and searching. According to the taxonomy, the work is positioned in the 'Comprehensive LLM Benchmarking and Surveys' leaf under 'Large Language Model-Based Factor Generation', which contains only one other paper. This places the work in a relatively sparse research direction within the broader LLM-based factor generation branch, which itself comprises six distinct leaves covering prompt-based generation, LLM-guided search, multi-agent collaboration, code-based evolution, hybrid integration, and benchmarking.

The taxonomy reveals that LLM-based factor generation sits alongside two major alternative paradigms: reinforcement learning-based mining (three leaves, nine papers) and evolutionary approaches (two leaves, three papers). The LLM branch appears to be the most actively explored methodological stream, with prompt-based generation and LLM-guided search each containing two to three papers. AlphaBench's focus on systematic evaluation across multiple tasks and configurations distinguishes it from method-specific papers in sibling leaves, such as those proposing novel prompting strategies or search algorithms. The taxonomy's scope and exclude notes clarify that benchmarking work explicitly excludes method-specific contributions, positioning AlphaBench as infrastructure rather than a new mining technique.

Among the three contributions analyzed, the first (systematic benchmark) examined nine candidates with zero refutable matches, suggesting novelty within the limited search scope. The second contribution (formal definition and unified perspective) examined ten candidates and found one refutable match, indicating some overlap with prior conceptual frameworks among the twenty-nine total candidates reviewed. The third contribution (multi-task evaluation framework) examined ten candidates with no refutations. These statistics reflect a targeted literature search rather than exhaustive coverage, meaning the analysis captures top semantic matches and immediate citations but may not encompass all relevant prior work in the broader quantitative finance or LLM evaluation literature.

Based on the limited search scope of twenty-nine candidates, the benchmark contribution appears relatively novel within the specific intersection of LLMs and formulaic alpha mining, though the formal framing shows some conceptual overlap with existing work. The taxonomy structure suggests this is an emerging research direction with fewer established benchmarks compared to the more mature RL and evolutionary branches. The analysis does not cover potential overlaps with general LLM benchmarking literature outside the alpha mining domain or with proprietary industry practices that may not appear in academic publications.