Where in the code does masking of tokens happen when pretraining BERT

rsvarma
Tue, 11 Aug 2020 04:43:54 +0000

@sgugger Thanks for your prompt responses and bearing with me

I understand that the tokens with indices -100 are used to determine which tokens are used in the computation of loss, as well as that they are used to prevent you from learning on padded tokens. However, my understanding is that in the masked language modelling task BERT only trains on masked tokens, e.g. tokens replaced with [MASK], in which case I would also want the model to ignore results from any non-masked tokens, which is why I would need to know which tokens are masked.

This leads back to my main point of confusion now: Is BertTokenizer replacing tokens with [MASK] as described in the BERT paper? I see that it has a mask_token parameter, what does it use this parameter for?