Hush-Nano
Hush-Nano is a 22M decoder-only small language model trained from scratch on an 8.5B-token corpus.
This is a base model, not an instruction-tuned chat model.
Model Details
Hush-Nano has the following features:
- Type: Causal Language Models
- Training Stage: Pretraining
- Architecture: transformers with RMSNorm, RoPE, SwiGLU, QK-Norm and tied word embeddings
- Number of Parameters: 22M (22,621,056)
- Number of Layers: 12
- Number of Attention Heads (GQA): 6 for Q and 3 for KV
- Context Length: 1,024
It is not recommended to use this model for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.
Training Data
The model was trained on 8.5B tokens (8,554,042,292) drawn from the following subsets:
| Source | Training tokens | Share |
|---|---|---|
| FineWeb-Edu | 4,539,286,619 | 53.07% |
| DCLM | 2,890,209,171 | 33.79% |
| FineMath4plus | 1,124,546,502 | 13.15% |
Evaluation
Zero-shot normalized accuracy, evaluated in fp32 using EleutherAI/lm-evaluation-harness. Scores may vary slightly with the evaluation setup and environment.
| PIQA | ARC-Easy | ARC-Challenge | HellaSwag | |
|---|---|---|---|---|
| Hush-Nano | 58.27% | 38.93% | 21.84% | 28.89% |
Usage
This model includes custom Transformers code and so requires trust_remote_code=True.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Soulitude/Hush-Nano"
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
).to(device).eval()
prompt = "Every effort makes you"
inputs = tokenizer(
prompt,
return_tensors="pt",
add_special_tokens=False,
).to(device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=256,
do_sample=True,
temperature=0.7,
top_p=0.95,
repetition_penalty=1.1,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Intended Use and Limitations
Hush-Nano is intended for small language model research, text continuation, and evaluation of pretrained model behavior. It has not been instruction-tuned and may not follow conversational instructions reliably. Its configured maximum context length is 1,024 tokens.
License
Apache 2.0