FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLMs over Wireless Networks

Summary

FLoKD is an adaptive knowledge-distillation framework for fine-tuning large language models with federated LoRA over bandwidth-constrained wireless networks. Federated learning keeps raw client data local, but transmitting model parameters remains expensive as the model grows. LoRA reduces trainable parameters, while logit-based distillation avoids full parameter sharing but still requires large token-level outputs because of sequence length and vocabulary size. FLoKD instead uses intermediate LoRA activations as the distillation signal. It also scores transformer-block importance and transmits only the most informative blocks, while selecting public samples that better match local data distributions and provide more useful distillation information. Experiments on WikiText-103, PTB, and Dialog report 50% to 65% lower communication overhead and rapid convergence to competitive perplexity against baseline methods.