ELECTRA training reimplementation and discussion

Srinjoy
Fri, 08 Sep 2023 04:35:16 +0000

Hey, I have just started studying the ELECTRA paper. And had a few doubts. I was wondering if you could help me with those?

1. What exactly does the “Step” mean in step count? Does it mean 1 epoch or 1 minibatch?

2. Also, in paper I saw (specifically in Table 1) ELECTRA-SMALL and BERT-SMALL borh have 14M parameters, how is that possible as ELECTRA should have more parameters because its generator and discriminator module are both BERT based?

3. ALso, what is the architecture of both generator and discriminator?Are they both BERT to something else?

4. Also, we have a sampling step between generator and discriminator . How are you back-propogating the gradients through this?

Thanks in advance