Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs

The paper introduces the training re-evaluation curve (TREC) as a diagnostic for retrospectively assessing how well a trained model retains data encountered at different training stages, then demonstrates that TRECs can be predicted from AdamW's implicit EMA coefficients to enable proactive curriculum design. Within the taxonomy, it occupies the 'Predictive and Diagnostic Curriculum Design' leaf under 'Curriculum Construction Methods', where it is currently the sole member. This places it in a sparse, emerging research direction distinct from the more populated difficulty-based and adaptive curriculum branches.

The taxonomy reveals that most curriculum work clusters in neighboring leaves: 'Difficulty-Based Curriculum Construction' contains four papers using static metrics like perplexity or length, while 'Adaptive and Dynamic Curriculum Strategies' houses four papers that adjust curricula during training based on observed performance. The paper's predictive approach contrasts with these reactive or heuristic methods by forecasting retention patterns before training completes. Its connection to optimizer internals (AdamW EMA coefficients) also distinguishes it from data-centric methods in 'Data Efficacy and Organization Paradigms' and 'Data Selection and Filtering', which focus on sample quality or structural arrangement rather than training dynamics.

Among twenty-three candidates examined across three contributions, none were flagged as clearly refuting the work. The TREC diagnostic itself was assessed against ten candidates with zero refutations, as was the predictive model linking TRECs to AdamW coefficients. The large-scale empirical study connecting TRECs to optimizer timescales examined three candidates, also with no refutations. Given the limited search scope—top-K semantic matches plus citation expansion—these statistics suggest the specific combination of retrospective evaluation curves and optimizer-based prediction has not been extensively explored in prior curriculum literature, though the analysis does not claim exhaustive coverage.

Based on the examined candidates and taxonomy structure, the work appears to occupy a relatively novel position by bridging optimizer theory and curriculum design through predictive diagnostics. The absence of sibling papers in its taxonomy leaf and the zero-refutation outcome across all contributions indicate limited direct prior work within the search scope. However, the analysis is constrained by the twenty-three-candidate sample and does not rule out related ideas in broader optimization or training dynamics literature outside the curriculum-focused search.