Amortizing Federated Adaptation: Hypernetwork

Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models

new Abstract: Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning. However, existing federated LoRA methods suffer from two fundamental limitations: (1) structural aggregation bias, where independently averaging low rank factors fails to approximate the true combined update, and (2) client side initialization lag, as clients repeatedly reinitialize LoRA parameters across communication rounds, slowing...