ELECTRA training reimplementation and discussion

RichardWang
Tue, 06 Oct 2020 08:51:37 +0000

I have fixed several bugs to get closer to the official ELECTRA. And I found the content of BookCorpus hubbed on HuggingFace now is scattered, so I choose to switch to OpenWebText corpus, which the authors also train small model on.

If you are using the old version of this implementation, be sure to git pull and pip install -r requirements.txt