MSCH: Microbatch-Based Selective Activation Checkpointing With Recomputation Hidden for Efficient Training of LLM Models

Activation checkpointing is a widely-used technique to reduce GPU memory consumption during model training. While it helps to conserve memory, it introduces additional computational load. Existing solutions such as selective activation checkpointing (SAC) and microbatch-based selective recomputation...

Descripció completa

Dades bibliogràfiques

Autors principals:	Cheng Zhang, Minjun Yu, Li Yu, Pengyu Cong, Yuchao Yan, Jie Bao, Jian Jiang, Xiaozheng Wang, Xiaolong Ye, Tao Tang, Liang Xiao
Format:	Artigo
Idioma:	Inglês
Publicat:	IEEE 2024-01-01
Col·lecció:	IEEE Access
Matèries:	Activation checkpointing Deepspeed large language model (LLM) training Megatron selective checkpointing one forward pass followed by one backward pass (1F1B)
Accés en línia:	https://ieeexplore.ieee.org/document/10670415/
Etiquetes:	Afegir etiqueta Sense etiquetes, Sigues el primer a etiquetar aquest registre!