Finetuning whisper attention mask not set and canot be inferred

duonguyen
Wed, 17 Jul 2024 08:06:48 +0000

In my knowledge, I think the cause of the warning is the eos_token_id=pad_token_id=50257 in the tokenizer, but it’s fine because we replaced the pad_token_ids with -100 at the line labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100) in the class DataCollatorSpeechSeq2SeqWithPadding.