Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients
Overview
Overall Novelty Assessment
The paper introduces FedMosaic, a method for personalized federated learning that jointly addresses data and model heterogeneity through task-relevance-aware aggregation and a dimension-invariant module. It resides in the Parameter-Level Heterogeneous Aggregation leaf, which contains four papers including this one. This leaf focuses on aggregating heterogeneous models through parameter factorization or modular networks, distinguishing itself from the six-paper Heterogeneous Model Aggregation via Knowledge Distillation leaf. The relatively small cluster suggests this specific approach to parameter-level aggregation represents an emerging but not yet crowded research direction within the broader model heterogeneity landscape.
The taxonomy reveals that FedMosaic's neighboring research directions include knowledge distillation-based aggregation (six papers), low-rank adaptation methods like LoRA (three papers), and mixture-of-experts approaches (two papers). These branches collectively address model heterogeneity but differ in mechanism: knowledge distillation operates at the feature level, LoRA focuses on parameter-efficient tuning, and mixture-of-experts employs sparse activation. FedMosaic's parameter-level approach sits between these strategies, sharing the goal of enabling cross-architecture collaboration while maintaining distinct technical foundations. The taxonomy's scope notes clarify that parameter-level methods explicitly exclude knowledge distillation, positioning FedMosaic in a complementary rather than overlapping space.
Among the twenty-nine candidates examined through semantic search, no papers were identified as clearly refuting any of FedMosaic's three contributions. The FedMosaic method itself was compared against ten candidates with zero refutable matches; the DRAKE benchmark against nine candidates with zero refutations; and the PQ-LoRA module against ten candidates with zero refutations. This limited search scope suggests that within the top-thirty semantically similar papers, the specific combination of task-relevance-aware aggregation, dimension-invariant modules, and multi-modal benchmarking appears relatively unexplored. However, the analysis does not claim exhaustive coverage of the broader federated learning literature beyond these candidates.
Based on the examined literature subset, FedMosaic appears to occupy a distinct position within parameter-level heterogeneous aggregation, particularly in its joint treatment of data and model heterogeneity with multi-modal considerations. The absence of refuting papers among thirty candidates suggests novelty within this search scope, though the analysis acknowledges limitations inherent to top-K semantic matching. The taxonomy context indicates this work contributes to an active but not saturated research direction, with neighboring methods pursuing complementary rather than directly competing strategies.
Taxonomy
Research Landscape Overview
Claimed Contributions
FedMosaic is a personalized federated learning method that simultaneously handles data heterogeneity (clients working on different tasks) and model heterogeneity (clients using different architectures). It comprises two main components: RELA for task-relevance-aware aggregation and PQ-LoRA for dimension-invariant knowledge sharing across heterogeneous models.
DRAKE is a comprehensive benchmark for multi-modal federated learning that assigns each client a distinct multi-modal task (such as visual question answering or visual reasoning) and incorporates temporal distribution shifts to mimic real-world task diversity and evolving data distributions.
PQ-LoRA introduces dimension-invariant modules (matrices P and Q) within the LoRA framework whose dimensions depend only on the low-rank size rather than model-specific hidden dimensions, enabling parameter sharing and knowledge transfer across heterogeneous model architectures with different dimensions and depths.
Core Task Comparisons
Comparisons with papers in the same taxonomy category
[42] Factorized-fl: Personalized federated learning with parameter factorization & similarity matching PDF
[46] pFedClub: Controllable Heterogeneous Model Aggregation for Personalized Federated Learning PDF
[47] Personalized Federated Learning via Heterogeneous Modular Networks PDF
Contribution Analysis
Detailed comparisons for each claimed contribution
FedMosaic method for heterogeneous personalized federated learning
FedMosaic is a personalized federated learning method that simultaneously handles data heterogeneity (clients working on different tasks) and model heterogeneity (clients using different architectures). It comprises two main components: RELA for task-relevance-aware aggregation and PQ-LoRA for dimension-invariant knowledge sharing across heterogeneous models.
[5] Personalized federated learning through local memorization PDF
[6] Personalized federated learning: A meta-learning approach PDF
[8] Towards personalized federated learning PDF
[22] Hierarchical personalized federated learning for user modeling PDF
[27] Multi-center federated learning: clients clustering for better personalization PDF
[30] Towards personalized federated learning via heterogeneous model reassembly PDF
[51] Fedrema: Improving personalized federated learning via leveraging the most relevant clients PDF
[52] Fairness-guided federated training for generalization and personalization in cross-silo federated learning PDF
[53] An efficient personalized federated learning approach in heterogeneous environments: a reinforcement learning perspective PDF
[54] Layer-wised model aggregation for personalized federated learning PDF
DRAKE benchmark for multi-modal personalized federated learning
DRAKE is a comprehensive benchmark for multi-modal federated learning that assigns each client a distinct multi-modal task (such as visual question answering or visual reasoning) and incorporates temporal distribution shifts to mimic real-world task diversity and evolving data distributions.
[55] Fedmedicl: Towards holistic evaluation of distribution shifts in federated medical imaging PDF
[56] A Survey on Vision-Language Models for Multimodal Federated Learning Tasks PDF
[57] Towards Robust Machine Learning under Distribution Shifts: From Causal Guarantees to Robust Federated Learning PDF
[58] COALA: A Practical and Vision-Centric Federated Learning Platform PDF
[59] Lightweight Multimodal Feature Fusion and Spatiotemporal Learning for Human Action Recognition on Edge Devices PDF
[60] Systematic Review of Domain Adaptation and Generalization in Image Classification and Face Recognition PDF
[61] Test-Time Robust Personalization for Federated Learning PDF
[62] Towards Accessible, Equitable, Generalizable and Useful Camera Health Sensing PDF
[63] FedTAP: Federated Multi-Task Continual Learning via Dynamic Task-Aware Prototypes PDF
PQ-LoRA for cross-architecture knowledge sharing
PQ-LoRA introduces dimension-invariant modules (matrices P and Q) within the LoRA framework whose dimensions depend only on the low-rank size rather than model-specific hidden dimensions, enabling parameter sharing and knowledge transfer across heterogeneous model architectures with different dimensions and depths.