Where in the code does masking of tokens happen when pretraining BERT

sgugger
Tue, 11 Aug 2020 03:21:58 +0000

You’re mixing masking tokens with indices -100 for loss computation and the masking of tokens. The loss ignores tokens with indices -100 because that’s how PyTorch has its default losses. You can use it to ignore the results of padded tokens.

The tokens masked are considered in the loss computation (otherwise your model isn’t learning to predict them). The masked token is [MASK] for BERT, you can replace any token you like by it.