Nova v3
Nova v3 is a decoder-only causal language model with 148.8 million parameters. This repository provides the Transformers checkpoint, tokenizer, and custom model code required to load the model.
Model specifications
| Specification | Value |
|---|---|
| Parameters | 148.8M |
| Vocabulary | 32,768 tokens |
| Hidden size | 640 |
| Transformer layers | 26 |
| Attention | 10 query heads, 5 key/value heads |
| Maximum context | 2,048 tokens |
| Checkpoint step | 3,500 |
The model supports key/value caching during generation. Its custom implementation supports Transformers 4.57 and 5.x. Review the custom modeling files before loading with trust_remote_code=True.
Load with Transformers
Install PyTorch and Transformers:
pip install torch transformers
Load the tokenizer and model:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "plasmova/nova-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
system_prompt = (
"You are Nova, a helpful and accurate assistant. Answer the user's request directly and concisely. "
"For simple arithmetic, calculate the result and state it plainly. Do not invent names, scenarios, or equations."
)
prompt = (
f"<|system|>{system_prompt}<|/system|>"
"<|user|>Explain why the sky appears blue in one sentence.<|/user|>"
"<|assistant|>"
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
output = model.generate(
**inputs,
max_new_tokens=160,
do_sample=True,
temperature=0.2,
top_p=0.85,
top_k=12,
repetition_penalty=1.12,
no_repeat_ngram_size=3,
eos_token_id=[
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|/assistant|>"),
],
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Recommended starting settings are temperature 0.2, top-p 0.9, top-k 40, and repetition penalty 1.08. These focused settings are a starting point; adjust them for your task. The local inference script applies this concise system instruction by default. Recommended generation defaults are temperature 0.2, top-p 0.85, top-k 12, repetition penalty 1.12, no-repeat n-gram size 3, and up to 160 new tokens. Keep the total prompt and generated text within the 2,048-token context. The example runs on CPU; for GPU inference, move the model and inputs to a supported device and dtype.
Evaluation
The following zero-shot results use the first 500 examples from each task with lm-evaluation-harness 0.4.13 on CPU. These are limited samples, not full validation-set results.
| Benchmark | Samples | Accuracy | Normalized accuracy |
|---|---|---|---|
| HellaSwag | 500 | 37.4% | 43.6% |
| ARC Easy | 500 | 51.0% | 48.2% |
| ARC Challenge | 500 | 21.8% | 25.0% |
| WinoGrande | 500 | 52.6% | 52.2% |
| OpenBookQA | 500 | 18.8% | 30.0% |
Comparison with SmolLM2-135M
In a matched zero-shot run using 500 examples per task, Nova v3 scored 43.6% normalized accuracy on HellaSwag and 52.2% on WinoGrande, compared with 44.2% and 50.6% for SmolLM2-135M. The WinoGrande difference is small relative to sampling uncertainty; treat both results as early directional measurements, not established performance claims. The comparison details and per-run metrics are in benchmark_comparison.json.
On a separate 100-question GSM8K sample with five examples in context, Nova v3 and SmolLM2-135M-Instruct each answered 4 questions correctly (4.0% exact match). This small sample does not establish equivalence. Details are in gsm8k_comparison.json.
Training and license
The v3 training code specifies pretraining on a mixture of FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath, Wikipedia, and CodeParrot, followed by SmolTalk fine-tuning. Dataset contents remain subject to their respective upstream terms.
The model weights and accompanying custom model code are released under the Apache License 2.0. See LICENSE.