1 INTRODUCTION

2 BACKGROUND

3 TAXONOMY OF TRANSFORMERS

4 ATTENTION

5 OTHER MODULE-LEVEL MODIFICATIONS

6 ARCHITECTURE-LEVEL VARIANTS

7 PRE-TRAINED TRANSFORMERS

8 APPLICATIONS OF TRANSFORMER

9 CONCLUSION AND FUTURE DIRECTIONS