MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs

Core task: Compressing Mixture-of-Experts based large language models. The field has organized itself around several complementary strategies for reducing the computational and memory footprint of MoE architectures. Expert-Level Structural Compression focuses on pruning, merging, or reorganizing experts themselves—ranging from clustering similar experts (Cluster Expert Pruning[17]) to decomposing expert weights via low-rank factorization (FactorLLM[4], Structured MoE SVD[8]). Quantization and Mixed-Precision Techniques apply bit-width reduction and adaptive precision schemes (QMoE[29], MoEQuant[44]) to shrink model size while preserving accuracy. Inference Optimization and Deployment addresses runtime challenges such as expert offloading (Fast MoE Offloading[12], SwapMoE[25]) and efficient scheduling (Faster MoE Inference[15]). Training Efficiency and Scaling investigates how to grow or initialize MoE models cost-effectively (MoE Scaling Laws[5], Orthogonal Growth MoE[46]), while Domain-Specific and Multimodal Extensions adapt MoE compression to vision-language tasks (MoE-LLaVA[33], DeepSeek-VL2[1]). Parameter-Efficient Fine-Tuning with MoE explores lightweight adaptation methods (X-LoRA[23]), and Benchmarking and Analysis provides empirical insights into trade-offs across these approaches. Within Expert-Level Structural Compression, a particularly active line of work centers on weight decomposition and factorization, where researchers seek to represent expert parameters more compactly without discarding entire experts. MoBE[0] exemplifies this direction by applying block-wise low-rank decomposition to expert weights, aiming to balance compression ratio and task performance. Nearby efforts include Structured MoE SVD[8], which enforces structured singular value decomposition for hardware-friendly compression, and ResMoE[24], which introduces residual connections to preserve information during factorization. Delta Decompression[35] and CAMERA[39] further explore how to efficiently reconstruct or adapt compressed representations at inference time. These factorization-based methods contrast with pruning-centric approaches (Efficient Expert Pruning[19], Mosaic Pruning[26]) that remove redundant experts entirely, and with merging strategies (Super Experts[21]) that consolidate multiple experts into fewer units. MoBE[0] sits squarely in the factorization cluster, sharing the goal of retaining all experts in a compressed form rather than eliminating them, and its block-wise strategy offers a middle ground between fine-grained decomposition and coarse expert-level operations.