Where in the code does masking of tokens happen when pretraining BERT

rsvarma
Tue, 11 Aug 2020 02:19:56 +0000

Hi all,

I was making myself familiar with the BertForPreTraining and BertTokenizer classes, and I am unsure where in the code the masking of tokens actually happens. I have tried tracing through but am getting lost in the weeds of various tokenizer subclasses. Any help with this would be much appreciated.