I. Introduction
II. Background on Transformer Networks
III. Motivation and Overview
IV. Knowledge Distillation
V. Pruning
VI. Quantization
VII. Efficient Transformer Design
VIII. Neural Architecture Search
IX. Hardware Optimization Techniques
X. Conclusion and Future Work