See our collection for all versions of ELECTRA.
Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow
zeromodels/electra_base_discriminator
Paper: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555) · HF Papers
ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token discriminator (with a smaller generator producing the corrupted tokens). This repo is the encoder / downstream checkpoint. WordPiece tokenizer; mask token [MASK].
For more details on the model, please go to the upstream model card.
Pure-Keras 3 conversion of google/electra-base-discriminator for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.
✨ Quick start (encoder / downstream)
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.electra import ElectraModel, ElectraTokenizer
model = ElectraModel.from_weights("zeromodels/electra_base_discriminator")
tokenizer = ElectraTokenizer.from_weights("zeromodels/electra_base_discriminator")
out = model(tokenizer("The quick brown fox."))["last_hidden_state"] # (1, L, H)
The same repo also serves the task heads, loaded the same way: ElectraSequenceClassify, ElectraTokenClassify, ElectraQnA, ElectraMultipleChoice (each takes the pretrained encoder and a randomly-initialized head, ready for fine-tuning).
Load any ELECTRA variant the same way with from_weights("zeromodels/<variant>"):
| Size | Discriminator (encoder / downstream) | Generator (masked-LM) |
|---|---|---|
| small | zeromodels/electra_small_discriminator | zeromodels/electra_small_generator |
| base | zeromodels/electra_base_discriminator | zeromodels/electra_base_generator |
| large | zeromodels/electra_large_discriminator | zeromodels/electra_large_generator |
Available classes
Load any of these from this repo with from_weights("zeromodels/electra_base_discriminator") (or on the fly via the hf: prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a hf: fine-tune).
| Class | Task |
|---|---|
ElectraModel | Encoder backbone |
ElectraSequenceClassify | Sequence classification |
ElectraTokenClassify | Token classification (NER / POS) |
ElectraQnA | Extractive question answering |
ElectraMultipleChoice | Multiple choice |
from zeromodels.models.electra import ElectraSequenceClassify
model = ElectraSequenceClassify.from_weights("zeromodels/electra_base_discriminator")
Tips
- Set
KERAS_BACKENDbefore importing Keras / zeromodels. - Prefer
ElectraTokenizer.from_weights(...)so WordPiece tokenization matches. - Downstream tasks (classification / QA / NER) use the discriminator repos; the generator repos are the masked-LM.
- See ELECTRA docs and Loading Weights.
- Community / upstream safetensors still work via the
hf:prefix, e.g.ElectraModel.from_weights("hf:google/electra-base-discriminator").
Special Thanks
A huge thank you to the Google ELECTRA authors for creating and releasing these models.
License: Apache 2.0.