PCCL

PCCL: Process Group-Aware Scalable and Generic Collective Algorithm Synthesizer

Announce Type: new Abstract: Distributed machine learning has become increasingly important due to the massive scale of large-scale generative models. Both model parameters and data are distributed across many compute devices, which requires frequent collective communications to synchronize activations and parameter updates. Such collective communications have become a major bottleneck.