I. INTRODUCTION

II. FUNDAMENTALS OF DL AND DISTRIBUTED DL

III. COMMUNICATION-EFFICIENT MODEL SYNCHRONIZATION

IV. COMMUNICATION-EFFICIENT DATA COMPRESSION

V. LARGE-SCALE RESOURCE ALLOCATION AND TASK SCHEDULING

VI. LARGE-SCALE COMMUNICATION INFRASTRUCTURES

VII. LARGE-SCALE DISTRIBUTED TRAINING OF LARGE MODELS: A CASE STUDY ON LLMS

VIII. CONCLUSION