Hi, I want to train Trainer scripts on single-node, multi-GPU setting. Do I need to launch HF with a torch launcher (torch.distributed, torchX, torchrun, Ray Train, PTL etc) or can the HF Trainer alone use multiple GPUs without being launched by a third-party distributed launcher?

See the documentation on running scripts . :slight_smile:

My impression with HF Trainer is HF has lots of video tutorials and none talks about multi GPU training using Trainer (assuming it is so simple) but the key element is lost in the docs, which is the command to run the trainer script which is really hard to find. So the easiest API is made hard by m…

I have a script that uses HF Trainer and works fine when I run it. But if I run the command for multi-gpu training torchrun --nproc_per_node 4 my_script.py I get an error: [rank1]: Traceback (most recent call last): [rank1]: File "/home/jpiabrantes/rosetta/fine_tune_coder.py", line 128, in <modu…

hello, I also encounter this problem, when I load the model using AutoModelForCausalLM.from_pretrained("xxx", device_map="auto"). It will shard my model on each device. And I set Trainer as follows: args = TrainingArguments( output_dir="./output", per_device_train_batch_size=2, gradient…

How to run single-node, multi-GPU training with HF Trainer?

🤗Transformers

brando August 17, 2022, 3:38pm 3

I think the docs are insufficient. See my questions here: Using Transformers with DistributedDataParallel — any examples?

Topic		Replies	Views
Single Node Multi GPU FlanT5 fine-tuning using HF Dataset and HF Trainer 🤗Transformers	4	2132	July 5, 2023
Training using multiple GPUs Beginners	20	20420	February 25, 2024
Multi gpu training 🤗Transformers	3	6110	April 24, 2022
How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? Intermediate	17	18538	September 6, 2023
Boilerplate for Trainer using torch.distributed Beginners	4	2125	January 11, 2022

How to run single-node, multi-GPU training with HF Trainer?

Related topics