11B model gets OOM after using deepspeed zero 3 setting with 8 32G V100

[John6666]

Recent similar issues also…

OutOfMemoryError: CUDA out of memory(LLM) Tuning

I am currently working on large language model (LLM) and frequently encounter the error: “OutOfMemoryError: CUDA out of memory.” Despite trying various solutions from multiple sources, I have been unable to resolve the issue. Could you please anyone can guide me to resolve this? TIA [lk3]

2B Model Fill Up Memory Usage on 4xA100s

With 4xA100s, I am training a 2B model (Gemma-2-2b-it) to perform SequenceClassification on a custom dataset. My code is quite boilerplate, but what I don’t understand is why would model training fill up the GPU memory? As I see it, With a model like gemma-2b, the total number of parameters is 2B. If we assume full precision training, i.e. float32, then the total GPU memory required to hold the model is 2B*(32/8=4 bytes/parameter) = 8GB of memory. When the model undergoes training, there is also…