Qwen3-4B-Diffusion-Base

Qwen3-4B-Diffusion-Base is a 4.41B-parameter masked-diffusion language-model backbone initialized from Qwen/Qwen3-4B-Base. It is the Stage 0 diffusion base used for the 4B-scale Continuous Interaction Diffusion (CID) training path.

This repository contains the diffusion-converted Qwen3 backbone at CID Stage 0, before Stage A / Stage B training. It serves as the 4B CID initialization checkpoint and as a standalone artifact for diffusion-LM research, reproduction, and ablation studies.

Summary

The model adds a dedicated <|cid_mask|> token and uses a diffusion-aware remote-code loader. Generation follows masked-diffusion semantics with bidirectional attention.

Files

• model-00001-of-00003.safetensors … model-00003-of-00003.safetensors — model weights.
• model.safetensors.index.json — shard index and parameter metadata.
• config.json — Qwen3 configuration adapted for bidirectional masked-diffusion use.
• diffusion_config.json — diffusion objective, mask token, training corpus, and checkpoint metadata.
• modeling_cid_diffusion.py — Hugging Face remote-code implementation with bidirectional forward, denoising, and diffusion-aware generate().
• tokenizer files — tokenizer inherited from the Qwen3 base with the CID mask token.
• logs/ — sanitized training and evaluation metrics plus public run metadata.
• SHA256SUMS — checksums for release files.

Optimizer state, FSDP rank-local checkpoint state, raw stdout/stderr, hostnames, local absolute paths, process IDs, and private job metadata are not part of this release.

Usage

The repository can be loaded through Transformers with trust_remote_code=True:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "fwerkor/Qwen3-4B-Diffusion-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)

inputs = tokenizer("Diffusion language models can", return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    diffusion_steps=64,
    block_length=16,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The loader also exposes denoise() for directly resolving existing mask tokens. Direct generations reflect base-model behavior; instruction following and full CID behavior are introduced in later training stages.

For CID Stage A / Stage B training and runtime integration, use the main CID repository:

https://github.com/fwerkor/continuous-interaction-diffusion

Stage 0 training

Stage 0 converts the autoregressive Qwen3 base into a native masked-diffusion backbone by continued pretraining with bidirectional attention and a LLaDA-style masked-diffusion objective.

The published checkpoint uses:

• 2 × NVIDIA GeForce RTX 3090 (24 GiB)
• 2-way FSDP with CPU offload
• micro-batch size 4 per rank
• global batch size 96
• gradient accumulation: 12
• sequence length: 2,048
• AdamW, learning rate 2e-5
• weight decay 0.1
• constant learning-rate schedule, no warmup
• gradient checkpointing
• mask-ratio sampling over [0.001, 1.0]
• checkpoint interval: 100 optimizer steps
• evaluation interval: 250 steps

The public checkpoint is step 3,300, corresponding to 648,806,400 training tokens. Public training logs are truncated at this checkpoint so that the released metrics correspond exactly to the published weights.

Training data

The Stage 0 stream mixes four public corpora:

Exact source metadata is also preserved in diffusion_config.json.

Training logs

Sanitized release logs are under logs/:

• logs/train_metrics.jsonl — rank-0 training metrics through the published step-3300 checkpoint.
• logs/eval_metrics.jsonl — periodic Stage 0 evaluation loss.
• logs/run_config.json — public training configuration and provenance.
• logs/manifest.json — checksums and privacy metadata for the released logs.
• logs/README.md — field and sanitization notes.

The public logs contain numerical training statistics only. Raw process logs are intentionally excluded because they may contain local paths, host information, process metadata, and transient infrastructure details.

At the published checkpoint, the rank-0 logged training loss is 3.0079. The latest evaluation before the checkpoint is step 3,250 with loss 2.8881. These values are raw run metrics and should be interpreted within this training run.

Checkpoint provenance

• Published source checkpoint: step 3,300
• Published tokens seen: 648,806,400
• Release loader source commit: 3be85aa0ab31b7465a7fa4a28ae33af9988d6b0c
• Exact training script SHA-256: 60808056f8fa01fb7a88c8ee66bee6a34fdc711bdde7f819c73d159dd1dd57f0
• Exported model implementation SHA-256: 9658daf8fd5b0b6db3c8fbb546425334ca6fe83110f8a2e8d87c39c685298230

The training source tree had local changes relative to the recorded Git commit, so the exact training-script hash above is the authoritative code provenance for this run.

Intended use and limitations

This checkpoint is intended for:

• diffusion-language-model research;
• CID Stage A initialization;
• training/reproduction studies;
• diffusion decoding and runtime experiments;
• ablations on AR-to-diffusion conversion.

This release corresponds specifically to the Stage 0 diffusion base that initializes the later 4B CID training path. CID Stage A/B checkpoints add the task-specific training and runtime behavior used by the full system.

Stage 0 loss alone is insufficient for downstream benchmark claims.

Paper and citation

This checkpoint accompanies the CID paper: Continuous Interaction Diffusion: A Diffusion-Native Architecture for Asynchronous Tool-Augmented Reasoning (arXiv:2608.10438, DOI).

If you use this model or the CID runtime, please cite:

@article{cao2026continuous,
  title   = {Continuous Interaction Diffusion: A Diffusion-Native Architecture for Asynchronous Tool-Augmented Reasoning},
  author  = {Cao, Yuhang and Mu, Yanzhou and Fang, Chunrong and Chen, Zhenyu},
  journal = {arXiv preprint arXiv:2608.10438},
  year    = {2026},
  doi     = {10.48550/arXiv.2608.10438},
  url     = {https://arxiv.org/abs/2608.10438}
}

License

This checkpoint is derived from Qwen/Qwen3-4B-Base, which is released under the Apache License 2.0. This release uses the same Apache-2.0 license designation.