1. Introduction
2. Taxonomy of LLM Architectures & Development Phases
3. Training Data Memorization in LLMs
4. LLM Privacy Attack Threat Models
5. LLM Privacy Attack Systemizations
6. Membership Inference Attacks on LLMs
7. Data Extraction Attacks on LLMs
8. Attribute Inference Attacks on LLMs
9. LLM Defence Systemization
10. Data Pre-processing Approaches
11. Privacy-Preserving Training
12. Post-training Approaches
13. Discussion & Future Opportunities