t5-small-nl-to-regex
google-t5/t5-small fully fine-tuned to translate an English description of a text pattern into a regular expression, on the three DeepRegex corpora (KB13, NL-RX-Synth, NL-RX-Turk).
Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("thealper2/t5-small-nl-to-regex")
model = AutoModelForSeq2SeqLM.from_pretrained("thealper2/t5-small-nl-to-regex")
PREFIX = "translate natural language to regex: "
# T5's SentencePiece vocabulary cannot represent \\ ^ ` { } ~, so the
# model emits them as '#b #c #g #l #r #n'. Decode them back:
CODEC = {"#b": "\\", "#c": "^", "#g": "`", "#l": "{", "#r": "}", "#n": "~"}
def decode_regex(text):
out, i = [], 0
while i < len(text):
pair = text[i:i + 2]
if pair in CODEC:
out.append(CODEC[pair]); i += 2
elif pair == "#h":
out.append("#"); i += 2
else:
out.append(text[i]); i += 1
return "".join(out)
def generate_regex(text):
ids = tokenizer(PREFIX + text, return_tensors="pt")
out = model.generate(**ids, max_length=128, num_beams=4, do_sample=False)
return decode_regex(tokenizer.decode(out[0], skip_special_tokens=True))
generate_regex("lines that contain the word 'dog'")
Output dialect
Outputs follow the DeepRegex regex representation, which is not Python re syntax. It adds two operators:
| Operator | Meaning | Example |
|---|---|---|
~X | complement — matches every string X does not | ~(.*e.*) |
X&Y | intersection — matches strings matched by both | ([A-Z])&([AEIOUaeiou]) |
Matching is whole-line (full-match) semantics. re.compile accepts these strings but reads ~ and & as literal characters, so outputs must be translated before use with Python re.
String literals follow the datasets' own preprocessing: words in quotes are normalised to the placeholders dog, truck, ring, lake in order of appearance. A description mentioning two quoted words yields a regex over dog and truck, which the caller substitutes back.
Training data
20824 examples from the three DeepRegex corpora:
| Corpus | Examples | Unique regexes | Description style |
|---|---|---|---|
| KB13 | 824 | 732 | Kushman & Barzilay (2013), human-written |
| NL-RX-Synth | 10000 | 9648 | synthetic descriptions generated from the regex |
| NL-RX-Turk | 10000 | 9648 | Mechanical Turk paraphrases of the NL-RX-Synth regexes |
Split 80/10/10 with seed 42 (train 16659 / validation 2082 / test 2083), stratified by corpus. NL-RX-Synth and NL-RX-Turk have byte-identical targ.txt files, so they are split on a shared permutation of the line index; otherwise every NL-RX-Turk test regex would also be an NL-RX-Synth training target.
127 of 2083 test regexes (6.1%) also occur as a training target; these are genuine repeats in the corpora and were not removed.
Training procedure
Full fine-tuning of all parameters — no LoRA, adapters or quantisation.
| Setting | Value |
|---|---|
| Base model | google-t5/t5-small |
| Parameters | 60,506,624 |
| Task prefix | translate natural language to regex: |
| Learning rate | 0.0003 |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| LR schedule | linear |
| Epochs (max) | 15.0 |
| Effective batch size | 64 |
| Max source / target length | 128 / 128 |
| Precision | bf16 |
| Early stopping patience | 3 epochs on validation exact match |
| Best checkpoint | checkpoint-3915 |
Evaluation
Held-out test split (2083 examples), beam search with num_beams=4, do_sample=False.
Metrics
- Exact match — string equality after stripping surrounding whitespace only.
- Regex validity — the output parses as a well-formed expression of the dialect.
- DFA equivalence — exact language equivalence decided with automata. Not decidable when either side contains a
\bassertion, so it is reported with its coverage. - Probe agreement — approximate: the prediction and the reference agree on every generated probe string. Finite-sample agreement, not a proof of equivalence.
- Functional accuracy — exact matches plus predictions judged equivalent by DFA equivalence, falling back to probe agreement where DFA equivalence is undecidable. The headline number; it mixes an exact and an approximate criterion and is reported as such.
Overall
| Metric | Value |
|---|---|
| Exact match | 66.78% |
| Regex validity | 99.76% |
| Functional accuracy | 78.54% |
| DFA equivalence (exact, 77.39% of examples decidable) | 80.40% |
| Probe agreement (approximate) | 79.98% |
| Character similarity | 0.9016 |
| Baseline: most frequent training regex | 0.00% exact match |
Per corpus
| Corpus | n | Exact match | Functional accuracy | Validity |
|---|---|---|---|---|
| KB13 | 83 | 42.17% | 68.67% | 100.00% |
| NL-RX-Synth | 1000 | 85.90% | 89.70% | 99.90% |
| NL-RX-Turk | 1000 | 49.70% | 68.20% | 99.60% |
Limitations
- Output is in the DeepRegex dialect, not Python
re;~and&need translation before use. - Quoted strings are placeholders (
dog,truck,ring,lake), inherited from the datasets' preprocessing. The model does not copy real literals. - The descriptions are short, single-sentence and template-like; behaviour on free-form or multi-sentence requests is untested.
- Probe agreement is finite-sample and can call two regexes equivalent when a longer counter-example exists.
- DFA equivalence cannot decide expressions containing
\b, roughly 20% of the corpus. - 60M parameters and a domain-specific vocabulary — not a general-purpose regex assistant.
- Generated regexes are not validated for catastrophic backtracking; do not run them on untrusted input without review.
Citation
Datasets from Locascio et al., Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge (EMNLP 2016), and Kushman & Barzilay, Using Semantic Unification to Generate Regular Expressions from Natural Language (NAACL 2013).