FLoCoRA: FEDERATED LEARNING COMPRESSION WITH LOW-RANK ADAPTATION

Lucas Grativol Ribeiro (1, 2, 3, 4) , Mathieu Leonardon (1, 2) , Guillaume Muller (5, 6, 7) , Virginie Fresse (4, 8) , Matthieu Arzel (1, 3)

1 IMT Atlantique - MEE - Département Mathematical and Electrical Engineering
2 Lab_STICC_BRAIn - Equipe Better Representations for Artificial Intelligence
3 Lab-STICC_2AI - Equipe Algorithm Architecture Interactions
4 LHC - Laboratoire Hubert Curien
5 Mines Saint-Étienne MSE - École des Mines de Saint-Étienne
6 FAYOL-ENSMSE - Institut Henri Fayol
7 FAYOL-ENSMSE - Département Informatique et systèmes intelligents
8 TSE - Ecole d'ingénieurs Télécom Saint-Etienne

Low-Rank Adaptation (LoRA) methods have gained popularity in efficient parameter fine-tuning of models containing hundreds of billions of parameters. In this work, instead, we demonstrate the application of LoRA methods to train small-vision models in Federated Learning (FL) from scratch. We first propose an aggregation-agnostic method to integrate LoRA within FL, named FLoCoRA, showing that the method is capable of reducing communication costs by 4.8 times, while having less than 1% accuracy degradation, for a CIFAR-10 classification task with a ResNet-8. Next, we show that the same method can be extended with an affine quantization scheme, dividing the communication cost by 18.6 times, while comparing it with the standard method, with still less than 1% of accuracy loss, tested with on a ResNet-18 model. Our formulation represents a strong baseline for message size reduction, even when compared to conventional model compression works, while also reducing the training memory requirements due to the low-rank adaptation.

Keywords: Low-Rank Adaptation; Federated Learning; Compression

Domains: Machine Learning [cs.LG]; Signal and Image processing

Lucas Grativol Ribeiro, Mathieu Leonardon, Guillaume Muller, Virginie Fresse, Matthieu Arzel. FLoCoRA: FEDERATED LEARNING COMPRESSION WITH LOW-RANK ADAPTATION. 32nd European Signal Processing Conference EUSIPCO, Aug 2024, Lyon, France. ⟨hal-04617632⟩