EXPO: Stable Reinforcement Learning with Expressive Policies

The paper proposes EXPO, an algorithm for training expressive policies (specifically diffusion and flow-matching models) in online reinforcement learning settings. It resides in the 'Online Diffusion Policy Training and Optimization' leaf, which contains five papers total including the original work. This leaf sits within the broader 'Diffusion-Based Policy Learning' branch, which itself is part of 'Expressive Policy Architectures and Representations'. The taxonomy reveals this is a moderately populated research direction: the immediate leaf has four sibling papers, and the parent branch includes an additional three papers on offline and theoretical diffusion policy work, indicating sustained but not overwhelming activity in diffusion-based RL methods.

The taxonomy structure shows that EXPO's leaf is adjacent to 'Offline and Theoretical Diffusion Policy Work' (three papers) and sits alongside other expressive policy branches including 'Flow-Based and Generative Policy Methods' (two papers) and 'Maximum Entropy and Multimodal Policy Learning' (two papers). The scope note for EXPO's leaf emphasizes 'stable online RL training of diffusion policies addressing gradient pathologies and sample efficiency', while explicitly excluding offline-only methods. This boundary clarifies that EXPO targets the specific challenge of interactive learning with diffusion models, distinguishing it from offline behavioral cloning approaches and from flow-matching methods that use different generative architectures.

Among the three contributions analyzed, the literature search examined twenty-eight candidate papers total. For the core EXPO algorithm contribution, nine candidates were examined with zero refutable matches. The on-the-fly policy parameterization examined nine candidates (zero refutable), and the edit policy with distance constraint examined ten candidates (zero refutable). These statistics indicate that within the limited search scope—roughly thirty semantically similar papers—no prior work was identified that clearly overlaps with EXPO's specific combination of techniques. The absence of refutable candidates across all three contributions suggests the approach may occupy a relatively unexplored niche, though the search scale means this assessment is necessarily provisional.

Based on the limited literature search of approximately thirty candidates, EXPO appears to introduce a novel combination of mechanisms for online diffusion policy training. The taxonomy context reveals this work sits in an active but not saturated research area, with roughly a dozen papers across diffusion-based policy learning. The zero-refutation finding across all contributions should be interpreted cautiously given the search scope: it indicates no obvious prior work among top semantic matches, but does not constitute exhaustive verification of novelty across the entire field.