24S - Efficient ML

EECE695E: Efficient ML Systems (Spring 2024) #

W1. Deep Learning Recap
- (2/19) Introduction to Efficient ML & Logistics
- (2/21) Computations of DL
W2. Sparsity
- (2/26) Algorithmic aspects of pruning
- (2/28) System aspects of pruning
W3. Quantization + Special Lecture
- (3/4) Basic ideas of quantization
- (~~3/6~~ -> 3/8, Fri) Special Lecture by Tae-ho Kim @ Nota
W4. Quantization + Distillation
- (3/11) PTQ and QAT
- (3/13) Distillation
W5. Neural Architecture Search
- (3/18) Search Space, Search Strategy (pt 1)
- (3/20) Search Strategy (pt 2), Performance Evaluation
W6. Adaptation
- (3/25) Continual Learning
- (3/27) Meta-Learning & Test-time Training
W7. Parallelism
- (4/1) Data and Model Parallelism
- (4/3) Advanced Topics
W8. (Mid-term week & Election Day)
- (4/8, 4/10)
W9. Model Merging & Editing
- (4/15) Model Merging
- (4/17) Model Editing
W10. Data Efficiency
- (4/22) Dataset Compression
- (4/24) Data Compression
W11. Topics on LLM - 1
- (4/29) Transformers & LLM Basics
- (5/1) Efficient Attention
W12. Topics on LLM - 2
- (5/6) Parameter-Efficient Fine-Tuning
- (5/8) Quantization & Pruning
W13. Topics on LLM - 3
- (5/13) Decoding & Batching Strategies
- ~~(5/15)~~ Buddha’s Day
W14. Topics on Diffusion Models
- (5/20) Diffusion Model Basics
- (5/22) Acceleration & Tuning
W15. Presentations Week - 1
- (5/27, 5/29)
W16. Presentations Week - 2
- (6/3, 6/5)

Blog Posts
- LLM inference performance engineering by Databricks
Videos
- State-space models tutorial by Sasha Rush
- Trends in deep learning hardware by Bill Dally
Related Courses
- Efficient deep learning systems at HSE university and Yandex
- Machine learning compilation at CMU
- TinyML and efficient deep learning at MIT
- Machine learning hardware and systems at Cornell
- Advances in Foundation Models at Stanford
Books and surveys
- Algorithms for modern hardware by Sergey Slotin
- Efficient deep learning book by Menghani and Singh
- Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities by Bartoldson et al.