Finetuning whisper attention mask not set and canot be inferred

huangruizhe
Sat, 20 Jul 2024 21:55:45 +0000

This example for Longform transcription doesn’t work out-of-the-box either:
https://huggingface.co/docs/transformers/v4.42.0/en/model_doc/whisper#transformers.WhisperForConditionalGeneration
It runs into the same error message as above.

I added a line: kwargs["attention_mask"] = attention_mask here:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/generation_whisper.py#L487
And it worked and got reasonable ASR results. I thought the transformers library changed some APIs at some point and Whisper didn’t pass in the attention_mask argument properly.

Not sure what’s the optimal solution.