ELECTRA training reimplementation and discussion

Venk1997
Fri, 15 Oct 2021 15:07:26 +0000

This is awesome. Thanks for sharing. I plan to warm start with google’s pre-trained models and continue pre-training on my domain-specific corpus. Can I use the same script for continual pre-training. The only changes would be to load generator and discriminator weights using ElectraForPreTraining.from_pretrained(“google/disc”) right? Thanks in advance.