GPU human docs

GPU Human Documentation

This is a collection of human-readable documentation related to GPU (Graphics Processing Unit) technologies, including tutorials, guides, and explanations of various concepts and practices in the field of GPU computing.

Contents

• GPU Glossary
• GPU Mode
• Outperforming cuBLAS on H100: a Worklog
• cuda mmm

training

• data parallelism
• pipeline parallelism
• baidu ring allreduce
• Allreduce - the basis of multi-device communication for neural network training
• pytorch distributed
• mpi docs
• shallow speed
• The Technology Behind BLOOM Training
• How to Train Really Large Models on Many GPUs?