DynamicMind-MoE

DynamicMind-MoE is a sparse mixture-of-experts decoder-only model trained on FineWeb-Edu, SmolLM-Corpus and FineMath

The model has 30.2M total parameters but activates only 8.9M per token, and was initialised by sparse-upcycling DynamicMind-Mini. It keeps the same custom 8k-token byte-level BPE tokenizer with digit-aware tokenization, so inference cost per token is unchanged from the dense model while total capacity is 3.4x larger.

Model Details

Mixture of Experts

Each block's dense MLP is replaced by one always-on shared expert plus 14 fine-grained routed experts, of which the router selects the top 2 per token. Shared + top-2 at intermediate size 256 reproduces the dense model's exact active parameter count.

Load balancing uses an auxiliary-loss-free bias (DeepSeek-V3 style): a per-expert bias steers selection toward idle experts while the combining weights come from the unbiased softmax, so balancing costs no gradient interference.

Routing is per token, per layer — a single sequence touches many different experts, and the 9 routers are independent of each other.

Tokenizer

DynamicMind-MoE uses the same digit-aware 8k tokenizer as DynamicMind-Mini.

Digits are kept as separate tokens so numbers do not collapse into large number tokens during tokenization.

Digit IDs:

Training

Data mixture: FineWeb-Edu sample-10BT 50.0%, Cosmopedia-v2 27.8%, FineMath-4plus 22.2%.

Embeddings, attention and norms transferred 1:1 from the dense model. The dense MLP (intermediate size 768) splits exactly into three 256-wide slices — because down_proj sums over the intermediate axis — and each expert was seeded from one slice plus small noise to break router symmetry.

Benchmarks

Self-reported results from the official BananaMind Base Bench 1.1 script, all measured with the same runner, dtype (bfloat16) and GPU.

Detailed DynamicMind-MoE result

Against the dense DynamicMind-Mini the MoE gains +44 Elo overall at identical inference cost, improving in six of seven categories. Quantitative is the exception (822 vs 837), despite FineMath being 22% of the corpus.

Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings. Cross-tokenizer comparisons (e.g. against GPT-2's 50k vocabulary) carry a residual tokenizer effect that mean-per-token log-probability does not fully remove.

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeProGames/DynamicMind-MoE"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()

prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is a base model, not instruction-tuned — it continues text rather than following instructions. At 8.9M active parameters it reproduces register and structure well (encyclopedic text reads encyclopedic, code keeps valid indentation) but is frequently wrong on facts and arithmetic. Keep a finite generation limit and do not use it for high-stakes decisions.

License

Apache 2.0