JyroAI-0.86B-PADMOE-UQ

Unquantized source weights (bf16 safetensors). The quantised releases are in JyroAI-0.86B-PADMOE-GGUF.

Read this before you use it

This model computes exactly the same function as Qwen/Qwen3-1.7B. Not approximately. Bit-for-bit identical output. Please verify that claim yourself before trusting it — the evidence is below.

It is a real Mixture-of-Experts checkpoint and a real inference target, but it is not a better model than Qwen3-1.7B. It is a scaffold for a training experiment that has not been trained yet.

What is actually in here

The "0.86B" in the name is the active parameter count, which is what actually costs compute per token. The total is 2.78B. Both numbers are correct and they measure different things.

The honest part

Expert 0 is a byte-for-byte copy of the Qwen3-1.7B FFN:

gate  max|diff| = 0.000e+00
up    max|diff| = 0.000e+00
down  max|diff| = 0.000e+00

Expert 1 is the Qwen3-0.6B FFN, zero-padded from (3072, 1024) up to (6144, 2048). It occupies 25% of its tensor; the rest is structural zeros. Measured output magnitude on a random hidden state is 0.72 against expert 0's 10.54.

The router is initialised to all zeros, so topk breaks the tie toward index 0, and norm_topk_prob rescales that single weight to exactly 1.0. Therefore every token, at every layer, computes precisely the Qwen3-1.7B FFN and nothing else.

Attention, embeddings and norms are Qwen3-1.7B. The 0.6B contributes only its FFN weights, in a slot that does not currently fire.

What this means for you

• Do not expect better quality than Qwen3-1.7B. You will get exactly the same quality, and the same answers.
• Do not expect speed either. 2.78B of bf16 weights is more memory traffic per token than 1.7B, for identical results. This model is strictly worse than just using Qwen3-1.7B.
• The value here is structural: it is a correctly-wired two-expert MoE with a real router and a genuine empty slot, which is a starting point for training. Expert 1 needs training to become meaningful. The 0.6B weights were learned in a 1024-dim space and are not a valid FFN in a 2048×6144 tensor until something trains them into that space.

Provenance and licences

Built from:

• Qwen/Qwen3-1.7B — expert 0, attention, embeddings, norms (Apache-2.0)
• Qwen/Qwen3-0.6B — expert 1, zero-padded (Apache-2.0)

No weights were trained. This is a structural recombination of two Apache-2.0 models, so the combined work is Apache-2.0. The Qwen3 models are governed by their own licence terms; review those before commercial use.

Training this model

Anyone can train or fine-tune this — the experts individually, or the whole model end to end. Nothing here is locked, obfuscated, or restricted. It is a standard Qwen3MoeForCausalLM checkpoint in plain safetensors, and every weight is an ordinary nn.Parameter that a normal PyTorch training loop can reach.

Three things you might want to do, in increasing order of ambition:

Train expert 1 only. Expert 1 is the interesting one — it starts as a zero-padded slot, so training it is how the model acquires anything the parent lacks. Freeze everything else and put your data through just model.layers.*.mlp.experts.down_proj[1] and its matching gate_up_proj[1]. Expert 0 stays bit-identical to Qwen3-1.7B, so any change in output is attributable to expert 1 alone. That makes it very easy to measure.

Train the routers. All 28 model.layers.*.mlp.gate.weight tensors are zero-initialised, which pins every token to expert 0. Moving them off zero is what makes the model conditional rather than dense-in-disguise — and it is the single highest-leverage change you can make. A router collapse (one expert taking most tokens) is the failure mode to watch for.

Fine-tune everything. Standard full fine-tuning, or QLoRA/LoRA for a lighter touch. On a 2.78B parameter model this fits comfortably in 8 GB of RAM with 4-bit quantisation and adapters; full-precision training of all 2.78B parameters wants considerably more.

import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "goldodemon/JyroAI-0.86B-PADMOE-UQ", dtype=torch.bfloat16
)

# Example: adapt only the expert FFN projections, and let the routers move.
config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    target_modules=["experts.gate_up_proj", "experts.down_proj", "mlp.gate"],
)
model = get_peft_model(model, config)

Two honest warnings before you start. First, measure against a baseline — perplexity on a held-out set for unmodified Qwen3-1.7B — or you will not be able to tell whether training helped or just changed things. Second, expert 1's weights came from a 1024-dim model and are not a valid FFN in this 2048×6144 tensor, so it needs enough training to become meaningful rather than a light touch.

Using it

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("goldodemon/JyroAI-0.86B-PADMOE-UQ")
model = AutoModelForCausalLM.from_pretrained(
    "goldodemon/JyroAI-0.86B-PADMOE-UQ", dtype=torch.bfloat16
)
ids = tok("The capital of France is", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=16)[0]))

Reproducing the check

Load the same prompt through this model and through Qwen/Qwen3-1.7B in transformers, and compare logits. They should agree to within bf16 rounding.