Accelerate not spreading on multiple CPUs

mvonwyl
Tue, 01 Aug 2023 16:27:05 +0000

Partially answering my own question.

Not using accelerate improved execution speed tremendously.

Instead of using load_checkpoint_and_dispatch as shown here

from accelerate import load_checkpoint_and_dispatch

model = load_checkpoint_and_dispatch(
    model,
    checkpoint=weights_location,
    device_map=device_map,
    no_split_module_classes=["Block"],
)

Using simply

model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    use_auth_token=True,
    device_map=device_map[checkpoint],
)

Makes the model much faster. Not sure why, though.

Read full topic