Sharded Data Parallelism
No mentions found
This entity hasn't been tracked yet, or Iris is still building its knowledge base.
Related Articles from SNS
Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism
arXiv:2606.09377v1 Announce Type: new Abstract: Formal neural network verification -- proving that a network satisfies safety properties for \emph{all} inputs in a specified domain -- is bounded in practice by GPU memory: standard implementations of bound-propagation algorithms (IBP, CROWN, $\alpha$-CROWN) require weight and relaxation-coefficient matrices to reside entirely on one accelerator. We adapt two parallelism techniques originally developed for large-scale model training to the...
Demystifying Pipeline Parallelism: First Theory for PipeDream
arXiv:2606.03498v1 Announce Type: new Abstract: Training modern machine learning models increasingly requires computation to be distributed across many accelerators. Data parallelism remains the default choice and is often paired with tensor-parallel sharding, but model parallelism becomes unavoidable once parameters, activations, or optimizer states no longer fit on a single device. This paper studies pipeline model parallelism through the lens of PipeDream (PD) (Harlap et al., 2018).
Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
arXiv:2511.17826v2 Announce Type: replace Abstract: Deterministic inference is increasingly critical for large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and Reinforcement Learning (RL). However, existing LLM serving frameworks exhibit non-deterministic behavior: identical inputs can yield different outputs when system configurations (e.g., tensor parallel (TP) size, batch size) vary, even under greedy decoding. This arises from the...
Show HN: Keybench – Scriptable, extensible performance tool for key value stores
guycipher/keybench Folders and files | Name | Name | Last commit date | || |---|---|---|---|---| Repository files navigation keybench ======== A scriptable, extensible performance tool for sorted key value stores.