DeepSpeed
Microsoft open source library From Wikipedia, the free encyclopedia
DeepSpeed is an open source deep learning optimization library for PyTorch.[1]
| |
| Original author(s) | Microsoft Research |
|---|---|
| Developer(s) | Microsoft |
| Initial release | May 18, 2020 |
| Stable release | v0.16.5
/ March 27, 2025 |
| Repository | github |
| Written in | Python, CUDA, C++ |
| Type | Software library |
| License | Apache License 2.0 |
| Website | deepspeed |
Library
The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.[2][3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.[5]
The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.[6]
See also
References
Further reading
External links
Wikiwand - on
Seamless Wikipedia browsing. On steroids.