Vega-1-65m-exp-base

I am introducing the experimental release of Qyrou-1 — a family of Small Language Models (SLMs) designed for practical, efficient parameter sizes. The series includes Vega (65M) and explores Sao (35M) and Aeon (125M) as potential variants. This model represents an experimental checkpoint from the initial pre-training phase. It is provided as-is for custom fine-tuning and further research experiments.

This experimental checkpoint has been trained on a diverse corpus containing mostly English, as well as math, grade-school science, basic coding, and step-by-step reasoning samples. It has not been trained on the full curated dataset. I am working on a newer model as part of my research and plan to use a standard architecture.

The Qyrou-1 series is built for researchers and developers exploring Small Language Models (SLMs), novel architecture designs, and practical model integration.

Qyrou-Vega models are the mid-sized variants within the Qyrou model family. They are designed to handle a broad range of tasks typically supported by small language models, including summarization, fill-in-the-middle (FIM), basic code completion, text generation, mathematical reasoning, and more.

Quickstart

This model is compatible with standard Hugging Face Transformers workflows and may be used as a remote-code model. Before proceeding, install the required Python dependencies. An NVIDIA GPU with a recent driver is strongly recommended for optimal performance; additional guidance is provided in the GPU acceleration section below.

pip install --upgrade transformers torch tokenizers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qyrou/Qyrou-1-EXP-Base", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Qyrou/Qyrou-1-EXP-Base")
model = model.to("cuda").to(torch.bfloat16) if torch.cuda.is_available() else model.eval()

inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Generation is supported through the standard GenerationMixin interface, including greedy decoding, sampling, beam search, and batch decoding with padding. The hybrid incremental cache, which combines attention key/value states with short-convolution state, is created and updated automatically; as a result, no explicit cache management is required on the caller side. Generation with use_cache=True is functionally equivalent to a full forward pass, including for right-padded and unequal-length batches.

The serving contract follows standard Transformers conventions. The model accepts output_hidden_states, output_attentions, attention_mask, position_ids, past_key_values, cache_position, use_cache, and the standard framework arguments. Attention weights are returned for every GQA layer when output_attentions=True, and hidden states are returned after the embedding layer and after each block. Unsupported arguments are rejected with clear errors rather than being silently ignored, which helps surface integration issues early.

Model Architecture

Qyrou-Vega is a 65.3M-parameter language model built around the qyrou-arch architecture. It was trained with a 2,048-token context window, although it could be extended beyond that with updated RoPE settings and cache support; there is no confirmed maximum yet (Although training beyond a sequence length of 2k could degrade a 65M SLM). The model has 21 layers: 17 use grouped-query attention, while 4 use causal convolutions in layers 4, 9, 14, and 19. Each layer uses RMSNorm and SwiGLU, with a hidden size of 512, a 20,000-token vocabulary, eight attention heads, two key/value heads, and tied input and output embeddings. Query and key states are normalized with per-head RMSNorm (QK-norm) before the rotary embedding is applied.

Benchmarks

Tokenizer

Qyrou-1-EXP uses a flexible, extensible byte-level BPE tokenizer designed to support continued experimentation across a broad range of workloads. The tokenizer is suitable for:

• General natural-language processing
• Language understanding and reasoning
• Conversational and instruction-following tasks
• ChatML-formatted interactions
• Code completion and Fill-in-the-Middle (FIM)
• Mathematical and symbolic reasoning
• Vision-, audio-, and multimodal token integration
• Tool-use and structured interaction formats

The included tokenizer is ready for immediate use with the model and provides a stable foundation for fine-tuning, evaluation, and experimentation across these domains. It uses a 20,000-token vocabulary and preserves byte-level coverage, enabling robust handling of Unicode text, source code, mathematical notation, formatting, and previously unseen text without relying on an unknown-token fallback in normal usage.

The tokenizer also includes a broad set of reserved and atomic control markers. These cover common conventions used by contemporary language models, including ChatML, reasoning boundaries, tool calls, FIM workflows, multimodal placeholders, vision and audio boundaries, and reserved extension tokens. Markers are encoded atomically to ensure that control structures remain stable and unambiguous during training and inference.

Core token IDs are fixed for compatibility:

The tokenizer uses a byte-level pre-tokenizer and decoder, allowing lossless round-trip encoding for supported text. It is configured with a 2,048-token model context length and includes a ChatML-compatible chat template using <|im_start|>, role markers such as <|user|> and <|assistant|>, and <|im_end|> delimiters.

Some control markers are intentionally represented as atomic regular-text tokens rather than removable special tokens. This allows fine-tuning workflows to preserve reasoning, tool-use, FIM, and multimodal markup when required. Applications should therefore select skip_special_tokens deliberately when decoding model outputs.

Users are free to fine-tune Qyrou-1-EXP for their own tasks, including domains that require additional control symbols or structured token sequences. If a required token or marker is not currently available, please open a discussion in this model repository describing the intended use case and the proposed token. Feedback from these experiments will be considered for future tokenizer revisions and subsequent model versions.

Data Used For Training

Note: The benchmark datasets listed here are intended for evaluation. For any datasets used during training, only the official training split was utilized.

Nvidia GPU Acceleration

I built this experimental model to run best on NVIDIA hardware, so use an NVIDIA GPU whenever possible. For maximum performance, enable BF16, cuDNN or Flash Attention, torch.compile, fused QKV and SwiGLU projections, and the included custom Triton kernels. Install Liger Kernel and Cut Cross-Entropy to enable the additional optimized paths. Together, these features reduce memory usage, kernel-launch overhead, and unnecessary computation while taking advantage of NVIDIA Tensor Cores.

Development

The repository is organized in a relatively straightforward manner. The implementation is contained in the qyrou_arch/ directory, the test suite is located in the tests/ directory, and the standalone utilities are stored in the scripts/ directory. The test suite may be executed with pytest from the source/ directory. It evaluates the core behavior of the model, including numerical parity for each layer against manual references, cached and uncached forward passes (including left-padded, right-padded, and uneven batch sizes), the serving interface, and the CUDA-gated Triton kernel paths. The CPU-based tests are expected to pass directly, while the Triton-specific tests assess FP32 and BF16 behavior, edge cases such as tail widths and non-contiguous inputs, and larger GPU batch sizes.

The scripts/benchmark_micro.py script functions as a lightweight benchmarking utility for the primary kernels, including packed QKV and SwiGLU GEMMs, RMSNorm, SDPA backends, and the cross-entropy path. It produces a JSON report containing median timings, which is useful for identifying regressions across different hardware configurations. CUDA is required to run this script.

The Git history may contain files such as attributions.md. Please disregard this file, as it contains a contaminated list of datasets from a separate project and is not associated with Qyrou-1-EXP. The datasets relevant to this project are documented in this README. Any other files identified as contaminated have been removed. I appreciate your understanding, as maintaining an open-source project can involve significant challenges. If you encounter any bugs or other issues with the repository, please open a discussion in the repository to report them!

Limitations

Qyrou-Vega was pretrained on a diverse dataset; however, as a small language model (SLM), it does not match the capabilities or performance of larger models trained on substantially larger and more diverse corpora. This release represents the first-stage pretraining (PT) run and is provided strictly for experimental and research purposes. As the first experimental base model in the Qyrou-1 family, it is intended primarily for experimentation and fine-tuning and is not recommended for production use until I release an official stable version. Furthermore, Qyrou-Vega is a base model and has not been instruction-tuned, meaning it is not optimized to follow user instructions or structured response formats and may produce inconsistent or unstable outputs until properly fine-tuned. By using this model, you acknowledge its experimental nature and assume full responsibility for its use. I am not liable for any damages, losses, or consequences arising from the use or misuse of this model.