DistilWord2-23k
DistilWord2-23k is the second-generation ultra-compact language model (~23k parameters) in the DistilWord series.
Trained on 12,000 clean, low-entropy synthetic words generated by Harley-ml/LargeWord-1.5M, it achieves state-of-the-art sample efficiency and morphology for its size class.
Benchmark Progression
| Model | Parameters | Training Distribution | Val Loss | Val PPL | Signal PSNR | Behavior |
|---|---|---|---|---|---|---|
| MicroWord-23k (Original) | ~23k | 750k Natural Words | 6.4105 | 608.18 | 2.57 dB | Attention collapsed (wwww..., zzzz...) |
| DistilWord-23k (v1) | ~23k | 3.8k Synthetic (TinyWord2) | 3.0633 | 21.40 | 4.37 dB | Stutter eliminated; basic morphology |
| DistilWord2-23k (Ours) | ~23k | 12k Synthetic (LargeWord-1.5M) | 2.5415 | 12.70 | 4.50 dB | Real morphemes (afields, appers, zrings) |
Qualitative Generations
| Prompt | Original MicroWord-23k | DistilWord2-23k (Ours) | Learned Morphology |
|---|---|---|---|
a | a (Stalled) | afields | Complete dictionary compound word (a + field + -s) |
app | appco | appers | Agent noun pluralization (app + -er + -s) |
z | zzzzx's (Stutter loop) | zrings | Complex inflectional cluster (-ing + -s) |
dis | dis | disers | Morpheme suffix attachment |
el | elel (Repeat loop) | elys | Adverbial suffix (-ly + -s) |
sub | sub | subs | Standard plural inflection |
Quick Inference
import torch
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
repo_id = "Useruser2statsaltalt/DistilWord2-23k"
tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
model.eval()
prompt = "app"
bos = tokenizer.bos_token or ""
inputs = tokenizer(bos + prompt, return_tensors="pt", add_special_tokens=False)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=12,
do_sample=True,
temperature=0.80,
top_p=0.90,
top_k=35,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
prompt_len = inputs["input_ids"].shape[-1]
completion = tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True)
print(f"Generated: {prompt + completion}")
## Architecture Specifications
Architecture: Qwen Causal LM (Micro-scale)
Hidden Layers: 1
Hidden Size: 16
Attention Heads: 1
Intermediate (SwiGLU): 56
Tied Embeddings: True
Unique Parameters: ~23,024