8000 GitHub - Elvin-Ma/distributed_training: Large-scale model distributed training technology
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Elvin-Ma/distributed_training

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

54 Commits
0_large-scale-model-trainning
0_large-scale-model-trainning
 
 
100_How-to-training-LLM-in-GPUs-Clusters
100_How-to-training-LLM-in-GPUs-Clusters
 
 
10_pytorch-FSDP-v2
10_pytorch-FSDP-v2
 
 
11_deepspeed-ZeRO-1-2-IMPL
11_deepspeed-ZeRO-1-2-IMPL
 
 
12_deepspeed-ZeRO-3-IMPL
12_deepspeed-ZeRO-3-IMPL
 
 
13_megatron-ZeRO-1-IMPL
13_megatron-ZeRO-1-IMPL
 
 
14_TP-Theory
14_TP-Theory
 
 
15_megatron-TP-IMPL
15_megatron-TP-IMPL
 
 
16_pytorch-TP-IMPL
16_pytorch-TP-IMPL
 
 
17_PP-Theory
17_PP-Theory
 
 
18_pytorch-PP-IMPL
18_pytorch-PP-IMPL
 
 
19_deepspeed-PP-IMPL
19_deepspeed-PP-IMPL
 
 
1_offload-and-recompute
1_offload-and-recompute
 
 
20_megatron-PP-IMPL
20_megatron-PP-IMPL
 
 
21_SP-Theory
21_SP-Theory
 
 
22_megatron-SP-IMPL
22_megatron-SP-IMPL
 
 
23_3D-Parallel-Theory
23_3D-Parallel-Theory
 
 
24_megatron-3D-Parallel-IMPL
24_megatron-3D-Parallel-IMPL
 
 
25_pytorch-3D-Parallel-IMPL
25_pytorch-3D-Parallel-IMPL
 
 
26_CP-Theory
26_CP-Theory
 
 
27_megatron-CP-IMPL
27_megatron-CP-IMPL
 
 
28_MOE-Theory
28_MOE-Theory
 
 
29_megatron-MOE-IMPL
29_megatron-MOE-IMPL
 
 
2_amp
2_amp
 
 
30_deepspeed-MOE-IMPL
30_deepspeed-MOE-IMPL
 
 
31_deepspeed-code-IMPL
31_deepspeed-code-IMPL
 
 
32_collective-operations
32_collective-operations
 
 
33_pytorch_distribute
33_pytorch_distribute
 
 
3_pytorch-DP
3_pytorch-DP
 
 
4_pytorch-DDP
4_pytorch-DDP
 
 
5_pytorch-DDP-IMPL
5_pytorch-DDP-IMPL
 
 
6_torchrun-process-group
6_torchrun-process-group
 
 
7_ZeRO-Optimizer
7_ZeRO-Optimizer
 
 
8_pytorch-ZeRO-1
8_pytorch-ZeRO-1
 
 
9_pytorch-FSDP-v1
9_pytorch-FSDP-v1
 
 
Megatron-LM-HybridDeviceOptimizer
Megatron-LM-HybridDeviceOptimizer
 
 
NVIDIA-DGX
NVIDIA-DGX
 
 
collective-operators
collective-operators
 
 
communicate-bandwidth
communicate-bandwidth
 
 
deepspeed-guide
deepspeed-guide
 
 
deepspeed-zero-infinity
deepspeed-zero-infinity
 
 
grpc_comm
grpc_comm
 
 
images
images
 
 
rdma_comm
rdma_comm
 
 
torch-cuda-stream-event
torch-cuda-stream-event
 
 
README.md
README.md
 
 

Repository files navigation

distributed_training

Large-scale model distributed training technology

About

Large-scale model distributed training technology

Resources

Readme
Activity

Stars

5 stars

Watchers

1 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages
0