Qwen3.6-35B-A3B — Slovenian (merged full model)
Trained by MediaAtlas — LLM fine-tuning on your own data, trained in the EU, weights delivered. Pricing · All our models
Na kratko: Slovenski jezikovni model (35B MoE, 3B aktivnih) za pogovor, znanje in prevajanje EN ⇄ SL, z razmišljanjem. Glede na neuglašeno osnovo: slovenski nabor acc_norm 0,623 → 0,654, prevajanje BLEU EN→SL 23,8 → 26,3 in SL→EN 30,9 → 35,0. Sprejme tudi slike.
Vision / slike: the vision encoder of the base Qwen3.6 model is kept in this checkpoint, so it also takes images (load it with the processor, as the base model). Our training was text-only: image understanding is the base model's and was not evaluated in Slovenian.
Merged full model (bf16) of Qwen/Qwen3.6-35B-A3B with Slovenian fluency, knowledge, and
en↔sl translation baked in (continued-pretraining + SFT). This is the LoRA adapter
texdata/Qwen3.6-35B-A3B-Slovenian-LoRA
already merged into the base, so you can load it directly. MTP (multi-token-prediction)
tensors are preserved (model-mtp.safetensors) for speculative decoding.
On held-out evals vs the untuned base: Slovenian-LLM-Eval acc_norm 0.623 → 0.654; translation BLEU en→sl 23.8 → 26.3, sl→en 30.9 → 35.0. Reasoning model.
💡 For plug-and-play chat, a GGUF build (LM Studio / llama.cpp) is at
texdata/Qwen3.6-35B-A3B-Slovenian-GGUF; the LoRA adapter alone is attexdata/Qwen3.6-35B-A3B-Slovenian-LoRA.
⚠️ How to run — load in 4-bit (not bf16)
This qwen3_5_moe arch has a broken fp16/bf16 forward in current transformers (zero hidden
state → constant logits → garbage). Even though the weights here are bf16, load in 4-bit
(bitsandbytes nf4) — the quantization it was trained/served under — or use the GGUF build.
(vLLM does not run this arch on Blackwell sm_120 GPUs.)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
repo = "texdata/Qwen3.6-35B-A3B-Slovenian"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, quantization_config=bnb,
trust_remote_code=True, device_map="auto")
# reasoning model: enable_thinking=False for direct answers/translation (else it spends tokens on <think>)
msg = [{"role": "user", "content": "Prevedi v slovenščino: Good morning!"}]
text = tok.apply_chat_template(msg, add_generation_prompt=True, tokenize=False, enable_thinking=False)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=128)
print(tok.decode(out[0], skip_special_tokens=True))
Benchmarks
All numbers below use the 4-bit bnb (nf4) eval path — the quantization this model was
trained/served under (see How to run; the native bf16 forward for this arch is broken).
n=100/task, held-out. Random baseline ≈ 0.25–0.50.
Slovenian-LLM-Eval — acc_norm
| Task | base Qwen3.6-35B-A3B | this model (CPT+SFT sl) | Δ |
|---|---|---|---|
| arc_easy | 0.71 | 0.78 | +0.07 |
| arc_challenge | 0.48 | 0.51 | +0.03 |
| hellaswag | 0.59 | 0.62 | +0.03 |
| piqa | 0.65 | 0.64 | −0.01 |
| openbookqa | 0.48 | 0.51 | +0.03 |
| winogrande | 0.62 | 0.63 | +0.01 |
| boolq | 0.83 | 0.89 | +0.06 |
| AVERAGE | 0.623 | 0.654 | +0.031 |
Wins 6/7 tasks (+3.1 pts avg). Base at n=200 is consistent (0.614), so the gain is not a small-sample artifact; the piqa −1 is within n=100 noise (±~5 pts).
Translation — en↔sl BLEU / chrF (n=150 held-out OPUS, greedy, thinking off)
| Direction | base Qwen3.6-35B-A3B | this model | Δ BLEU |
|---|---|---|---|
| en→sl | BLEU 23.79 / chrF 51.82 | BLEU 26.26 / chrF 50.53 | +2.47 |
| sl→en | BLEU 30.85 / chrF 55.76 | BLEU 34.95 / chrF 58.30 | +4.10 |
Model details
- Type: merged full model (bf16), MoE
qwen3_5_moe(16 shards +model-mtp.safetensors). - Base:
Qwen/Qwen3.6-35B-A3B(Apache-2.0, official — not an uncensored variant). - Adaptation: CPT + SFT LoRA (
sft_qwen35a3b_sl) merged in;template: qwen. MTP preserved.
License & data provenance
Base Qwen/Qwen3.6-35B-A3B is Apache-2.0. CPT data is open-licensed (attribution applies) →
this repo is license: other:
| Data | License |
|---|---|
| Slovenian Wikipedia (CPT) | CC BY-SA 4.0 (attribution + ShareAlike) |
| FineWeb2 sl (CPT) | ODC-BY 1.0 |
Attribute the above. Comply with each source's terms before commercial use.
Citation
Cite this work (Tadej Fius, MediaAtlas Ltd):
@misc{fius2026qwen35bsl,
title = {Qwen3.6-35B-A3B Slovenian (merged)},
author = {Fius, Tadej},
year = {2026},
publisher = {MediaAtlas Ltd},
howpublished = {Hugging Face},
url = {https://huggingface.co/texdata/Qwen3.6-35B-A3B-Slovenian}
}
Upstream / source citations:
@misc{qwen3, title={Qwen3 Technical Report}, author={{Qwen Team}}, year={2025}, url={https://huggingface.co/Qwen}}
@misc{wikipedia_sl, title={Slovenian Wikipedia}, author={{Wikimedia Foundation}}, note={CC BY-SA 4.0}, url={https://sl.wikipedia.org}}
@misc{penedo2024fineweb2, title={FineWeb2: A sparkling update with 1000s of languages}, author={Penedo, G. and others}, year={2024}, url={https://huggingface.co/datasets/HuggingFaceFW/fineweb-2}}
CPT data: Slovenian Wikipedia (CC BY-SA 4.0) and FineWeb2-sl (ODC-BY 1.0).