aviai-e4b
aviai-e4b is a full fine-tune of Gemma 4 E4B (instruction-tuned) for structured decoding of aviation text:
METAR and TAF reports into canonical JSON, and NOTAMs into category-specific extraction
rows or one of 13 operational classes. One model covers all four tasks. The fine-tune was trained as a
rank-16 LoRA and merged into the base weights, so this repo is a plain Transformers checkpoint (plus
GGUF conversions for llama.cpp); the original adapter is included under adapter/ for anyone who
prefers to apply it to the base themselves.
It is a research artifact from the avtext study (dataset engineering + evaluation harness + fine-tuning on a single AMD Strix Halo box). It is not a certified aeronautical product: do not use its output for operational or flight-safety decisions without independent verification.
What's in the repo
| file(s) | format | size | use |
|---|---|---|---|
model-*.safetensors + config.json, tokenizer and processor files | Transformers checkpoint, bf16, 4 shards | 15.9 GB | AutoModelForCausalLM.from_pretrained(<repo>) |
aviai-e4b-Q8_0.gguf | llama.cpp, 8-bit | 7.9 GB | llama-server -m … (the quantization the study's numbers were measured with) |
aviai-e4b-f16.gguf | llama.cpp, 16-bit | 14.9 GB | for re-quantizing to other formats |
adapter/ | PEFT LoRA (rank 16) + GGUF LoRA | 140 MB + 70 MB | apply to unsloth/gemma-4-E4B-it instead of downloading merged weights |
prompts/ | text | — | the exact prompt templates the model was trained on (required, see How to use) |
The merged GGUF reproduces the adapter-on-base serving path record for record (300-record METAR check: identical exact-match outcomes), and the merged Transformers checkpoint decodes identically to the PEFT path. The vision and audio towers of Gemma 4 E4B are carried over unchanged (the fine-tune touched only the text tower); the model still loads with the multimodal classes but was trained and evaluated as a text model.
Results: before and after fine-tuning
Same frozen evals, same prompts, greedy decoding, same Q8_0 quantization served by llama.cpp; "base" is Gemma 4 E4B alone, "aviai-e4b" is this fine-tune. One row per task, full sets.
| task | exact match · base | exact match · aviai-e4b | Δ (pts) | value recall · base | value recall · aviai-e4b | hallucination · base | hallucination · aviai-e4b |
|---|---|---|---|---|---|---|---|
| METAR → JSON | 24.8 % | 94.4 % | +69.6 | 88.8 % | 99.9 % | 21.4 % | 7.0 % |
| TAF → JSON | 7.2 % | 93.3 % | +86.1 | 89.4 % | 99.6 % | 11.8 % | 1.7 % |
| NOTAM → extraction rows | 0.8 % | 82.3 % | +81.5 | 11.0 % | 64.7 % | 14.9 % | 4.4 % |
| NOTAM → class (13) | 78.2 % ¹ | 95.3 % ¹ | +17.1 | — | — | — | — |
¹ classification is scored as accuracy (macro-F1: 74.2 % → 94.3 %).
The base model knows the vocabulary but cannot hold a whole structured schema; the fine-tune teaches the schema and the unit conventions, not new meteorology.
Metric notes. Eval sets: METAR 6,200 records (v2), TAF 5,294 (taf-v1, 2048-token output cap),
NOTAM extraction 2,257 (notam-v1), NOTAM classification 4,047 (notam-cls-v1). Exact match is per whole record. Value recall is the share of reference fields the
model reproduced with the correct value. Hallucination is the share of asserted values the reference
does not support. Outputs with no parseable JSON are scored as abstaining on every field. The
eval sets hold out unseen stations and unseen time windows. Exact definitions live in the avtext
harness (score.py, score_taf.py, score_notam.py).
Protocol notes. METAR, TAF and classification rows were measured with identical serving on both sides. The base NOTAM-extraction run used 100-record llama.cpp sessions and the base METAR run the chat endpoint; the fine-tune's runs used 20-record sessions and the raw completion endpoint (the fine-tune is sensitive to template drift, the base is not).
How to use
The fine-tune is hard-tuned on exact prompt templates. Send the templates in prompts/ verbatim
(the {raw} placeholder takes the report text); outputs drift off-distribution otherwise.
Transformers
import json, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "<this repo id>"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
template = open("prompts/metar.txt", encoding="utf-8").read()
raw = "METAR EPGD 111200Z 27012KT 9999 FEW030 SCT045 18/09 Q1015 NOSIG"
msgs = [{"role": "user", "content": template.format(raw=raw)}]
enc = tok.apply_chat_template(
msgs, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)
out = model.generate(**enc, max_new_tokens=400, do_sample=False)
text = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
print(json.loads(text[text.index("{"): text.rindex("}") + 1]))
Use max_new_tokens ≥ 2048 for TAFs (long multi-period forecasts) and ≥ 512 for NOTAM extraction.
llama.cpp
llama-server -m aviai-e4b-Q8_0.gguf --flash-attn on --reasoning-budget 0 -c 8192 --port 8080
Send the raw /completion endpoint the turn wrapper in prompts/turn_wrapper.txt around the
filled template (<|turn>user\n{prompt}<turn|>\n<|turn>model\n; the server prepends BOS) with
temperature 0. The chat endpoint's template engine renders Gemma 4's chat template slightly
differently from HF Transformers, and this fine-tune is sensitive to that drift.
For NOTAM extraction the study's harness additionally constrains decoding with a JSON grammar derived
from prompts/notam_fields.json (the row schema per category); without a grammar expect a few more
invalid outputs on long NOTAMs.
Adapter instead of merged weights
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("unsloth/gemma-4-E4B-it", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, repo, subfolder="adapter")
Training
| base | unsloth/gemma-4-E4B-it (weights mirror of google/gemma-4-E4B-it), bf16 |
| method | LoRA rank 16, alpha 32, dropout 0, on q/k/v/o/gate/up/down_proj of the text tower only (vision/audio towers untouched); 34.9 M trainable parameters, merged into the base weights after training (merge_and_unload) |
| data | 49,214 chat examples: 20,000 METAR, 16,000 TAF, 9,049 NOTAM extraction, 4,165 NOTAM classification |
| schedule | 1 epoch, AdamW (lr 2e-4, weight decay 0.01, linear decay, 10 warm-up steps), batch 6 × grad-accum 2, max sequence 2,048 tokens |
| tricks | length-grouped batching, torch.compile, length-adaptive gradient checkpointing (recompute only above 1,800 tokens) |
| hardware | one AMD Ryzen AI Max+ 395 (Radeon 8060S, gfx1151, 123 GiB unified memory), ROCm/TheRock nightly PyTorch; 16 h 58 min |
| final train loss | ≈ 0.13 (mean of the last 100 steps; 0.62 over the first 100) |
Limitations
- Research decode aid, not a certified aeronautical tool. Verify independently before any operational use.
- METAR shows a ~7 % hallucination floor shared by every fine-tune in the study (mostly optional fields the reference leaves empty).
- TAF: long multi-period forecasts need a ≥ 2,048-token output budget; below that ~3 % of outputs are truncated.
- NOTAM extraction: ~7 % of very long "area" NOTAMs (airspace restrictions with coordinate lists) yield no valid output; the model is also sensitive to prompt drift and server session length (restart llama.cpp sessions periodically on long batches).
- English / ICAO-format inputs only; trained on the four prompt templates shipped here.
License and notices
These weights are released under the Apache License 2.0 (see LICENSE). Gemma 4 E4B is
released by Google DeepMind under the Apache License 2.0 and subject to the
Gemma Prohibited Use Policy, which also applies to
this derivative. See NOTICE for attributions. Gemma is a trademark of Google LLC; this project is not
affiliated with or endorsed by Google.