Where in the code does masking of tokens happen when pretraining BERT

sgugger
Tue, 11 Aug 2020 02:53:47 +0000

There is no script to pretrain BERT in the examples, transformers is primarily there to help you finetune a model like BERT on a downstream task.

That being said, the DataCollatorForLanguageModeling masks random tokens when creating batches, if you need it.