JOSIE-Atom-Qwen3.5-Random
Randomly initialized Qwen3.5-style causal language model with a ByteLevel BPE tokenizer trained completely from scratch. The decoder uses a 3:1 hybrid stack of Gated DeltaNet linear-attention and full-attention layers.
Model details
- Architecture:
Qwen3_5ForCausalLM - Attention layout: 3:1 Gated DeltaNet linear attention / full attention
- Transformers version:
5.17.0 - Parameters:
90,584,480 - Vocabulary size:
32,768 - Context length:
8,192 - Hidden size:
1024 - Attention head dimension:
64 - Layers:
4 - Attention heads:
16 - Key/value heads:
2 - Intermediate size:
3072 - RoPE theta:
5000000.0 - Tied embeddings:
True - Tokenizer algorithm: ByteLevel BPE
- Tokenizer backend:
TokenizersBackend - Tokenizer training data:
openbmb/Ultra-FineWeb(default) - Tokenizer vocabulary size:
32,768 - EOS token:
<|im_end|> - PAD token:
<|im_end|> - UNK token:
<|unknown|>
The tokenizer vocabulary and BPE merge rules were trained entirely from scratch. No vocabulary, merge rules, normalizer, pre-tokenizer or decoder was inherited from another model.
This checkpoint contains random model weights and must be pretrained before use.