ATTENTION
This model is considered broken and should not be used.
It's repaired version Conversate-XL-v1.1 is also broken. Please stop using Conversate-XL-v1 and Conversate-XL-v1.1
The whitespace here was put to make the alert more visible.
Ivme-Conversate-XL-v1-Base
Dense decoder-only transformer, 125.6M parameters, trained from scratch by IvmeLabs. Part of the Conversate family — see the IvmeLabs organization page for related models (Conversate-S, mainline Conversate, and this XL tier).
Architecture
- 12 layers, hidden size 768, 12 attention heads (head_dim 64)
- SwiGLU feed-forward, ffn_dim 3072
- RoPE positional encoding (theta=10000.0)
- RMSNorm (pre-norm), tied input/output embeddings, no bias terms
- Vocabulary: 16000 tokens (BPE)
- Max sequence length: 1024
Training
Trained on a 5.0B-token mix (backbone: DCLM-baseline, FineWeb-Edu, FineMath; supplement: Wikipedia-en, Project Gutenberg-en) using Muon (body weights) + AdamW (embeddings/norms), on a single AMD Instinct MI300X (ROCm 7.14.0, PyTorch 2.12.0).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"IvmeLabs/Ivme-Conversate-XL-v1-Base", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-XL-v1-Base")
inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=50)
print(tokenizer.decode(outputs[0]))
Note: requires trust_remote_code=True since this uses a custom
architecture (modeling_ivme.py in this repo), not a built-in
transformers model class. Review that file before trusting it, as with
any trust_remote_code=True model.
Checkpoint
This repo contains checkpoint(s) from step(s): 160, 320, 480, 640, 800, 960, 1120, 1280, 1440, 1600, 1760, 1920, 2080, 2240, 2400, 2560, 2720, 2880, 3040, 3200, 3318