Prologue: What this survey is and is not about  
1 What is a neural network?  
2 What is the use of a neural network?  
3 Network training is an optimization problem  
4 Solving the optimization problem  
5 Activation functions  
6 Loss functions  
7 Overfitting and regularization  
8 Validation and hyper-parameter tuning  
9 Procedure summary  
10 Introduction  
11 Density in polynomial approximation  
12 Density of two-layer networks  
13 Convergence rate of approximation by two-layer networks  
14 Complexity of deep ReLU networks  
15 Introduction  
16 Standard ReLU networks and free knot linear splines  
17 ReLU networks are at least as expressive as free knot linear splines  
18 The power of depth  