1 Introduction

2 Token Embeddings

3 Pre-training Methods for Contextual Embeddings

4 Cross-lingual Polyglot Pre-training for Contextual Embeddings

5 Downstream Learning

6 Model Compression

7 Analyzing Contextual Embeddings

8 Current Challenges