FLHetBench: Benchmarking Device and State Heterogeneity in Federated Learning

Junyuan Zhang*2, Shuang Zeng*1, Miao Zhang3, Runxi Wang2,
Feifei Wang1, Yuyin Zhou5, Paul Pu Liang4, Liangqiong Qu1
1 The University of Hong Kong, 2 Beihang University, 3 New York University,
4 Carnegie Mellon University, 5 UC Santa Cruz
CVPR 2024
*Indicates Equal Contribution
Corresponding Author
This work was conducted when J. Zhang and R. Wang were interns at HKU

What is FLHetBench?

FLHetBench is the first FL benchmark targeted toward understanding:What happens to different FL algorithms when they are employed in real-world FL environments with varying degrees of device and state heterogeneity?


Federated learning (FL) is a powerful technology that enables collaborative training of machine learning models without sharing private data among clients. The fundamental challenge in FL lies in learning over extremely heterogeneous data distributions, device capacities, and device state availabilities, all of which adversely impact performance and communication efficiency. While data heterogeneity has been well-studied in the literature, this paper introduces FLHetBench, the first FL benchmark targeted toward understanding device and state heterogeneity. FLHetBench comprises two new sampling methods to generate real-world device and state databases with varying heterogeneity and new metrics for quantifying the success of FL methods under these real-world constraints. Using FLHetBench, we conduct a comprehensive evaluation of existing methods and find that they struggle under these settings, which inspires us to propose BiasPrompt+, a new method employing staleness-aware aggregation and fast weights to tackle these new heterogeneity challenges. Experiments on various FL tasks and datasets validate the effectiveness of our BiasPrompt+ method and highlight the value of FLHetBench in fostering the development of more efficient and robust FL solutions under real-world device and state constraints.


      author    = {Zhang, Junyuan and Zeng, Shuang and Zhang, Miao and Wang, Runxi and Wang, Feifei and Zhou, Yuyin and Liang, Paul Pu and Qu, Liangqiong},
      title     = {FLHetBench: Benchmarking Device and State Heterogeneity in Federated Learning},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2024},
      pages     = {12098-12108}