EECE695E: Efficient ML Systems (Spring 2024) #
Team #
- Instructor. Jaeho Lee 이재호
firstname.lastname@postech.ac.kr - TA. Hagyeong Lee 이하경
firstnamelastname@postech.ac.kr
Location & Time #
- Class. Mondays & Wednesdays, 11:00AM–12:15PM, PIAI 122.
- Office Hr. Mondays 5:00PM–6:00PM, Eng. Building #407 (+ by appointment).
Schedule (tentative) #
- W1. Deep Learning Recap
- (2/19) Introduction to Efficient ML & Logistics
- (2/21) Computations of DL
- W2. Sparsity
- (2/26) Algorithmic aspects of pruning
- (2/28) System aspects of pruning
- W3. Quantization + Special Lecture
- (3/4) Basic ideas of quantization
- (
3/6-> 3/8, Fri) Special Lecture by Tae-ho Kim @ Nota
- W4. Quantization + Distillation
- (3/11) PTQ and QAT
- (3/13) Distillation
- W5. Neural Architecture Search
- W6. Adaptation
- (3/25) Continual Learning
- (3/27) Meta-Learning & Test-time Training
- W7. Parallelism
- (4/1) Data and Model Parallelism
- (4/3) Advanced Topics
- W8. (Mid-term week & Election Day)
- (4/8, 4/10)
- W9. Model Merging & Editing
- (4/15) Model Merging
- (4/17) Model Editing
- W10. Data Efficiency
- (4/22) Dataset Compression
- (4/24) Data Compression
- W11. Topics on LLM - 1
- (4/29) Transformers & LLM Basics
- (5/1)
Efficient Attention
- W12. Topics on LLM - 2
- (5/6)
Parameter-Efficient Fine-Tuning - (5/8) Parameter-Efficient Fine-Tuning
- (5/6)
- W13. Topics on LLM - 3
- (5/13) Decoding Strategies
(5/15)Buddha’s Day
- W14. Topics on Diffusion Models
- (5/20) LLM Compression
- (5/22) Topics in long sequence handling
- W15. Presentations Week - 1
- (5/27, 5/29)
- W16. Presentations Week - 2
- (6/3, 6/5)
Recommended Materials #
- Blog Posts
- LLM inference performance engineering by Databricks
- Videos
- State-space models tutorial by Sasha Rush
- Trends in deep learning hardware by Bill Dally
- Related Courses
- Efficient deep learning systems at HSE university and Yandex
- Machine learning compilation at CMU
- TinyML and efficient deep learning at MIT
- Machine learning hardware and systems at Cornell
- Advances in Foundation Models at Stanford
- Books and surveys
- Algorithms for modern hardware by Sergey Slotin
- Efficient deep learning book by Menghani and Singh
- Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities by Bartoldson et al.