ELECTRA training reimplementation and discussion

RichardWang
Sun, 17 Sep 2023 01:08:13 +0000

1. Minibatch

2. Discriminator of Electra small is as the same size as BERT, generator of electra small is smaller than regular BERT. Note that we only use discriminator in finetuning.

3. Both bert
4.No backprop in sampling step, they train generator and discriminator under a multi task setting.

I suggest you can read the paper thoroughly, as it have already reveal the information of your questions.