1 INTRODUCTION  
2 BACKGROUND  
3 INFRASTRUCTURE FOR LLM TRAINING  
4 PARALLELISM SCHEMES FOR LLM TRAINING  
5 COMPUTATION OPTIMIZATIONS  
6 MEMORY OPTIMIZATIONS  
7 COMMUNICATION OPTIMIZATIONS  
8 FAULT TOLERANCE  
9 CONCLUSION AND OUTLOOKS