SRIQ-Spark-V1.4
SRIQ-Spark-V1.4 is a language model by SRIQ, supervised fine-tuned from XHToken/Spark-X2.5-4B. Our focus is reasoning traces that verify their own answer before committing to it.
The name combines SRIQ and Spark.
| Base model | XHToken/Spark-X2.5-4B |
| Method | LoRA supervised fine-tuning, merged into the released weights |
| Dataset | sriq-ai/sriq-sft-v1.4 |
| Training context | 262,144 tokens (configured ceiling) |
| Precision | bfloat16 |
Training
Supervised fine-tuning on sriq-ai/sriq-sft-v1.4, a ShareGPT-format set whose assistant targets pair compressed Simplified Chinese reasoning with the original final answer. The compression step never rewrites the final answer.
v1.4's defining change over v1.3 is a final verification step: traces close by checking the
answer against the original constraints — substituting the result back, re-deriving the invariant,
walking the stated edge cases — introduced by 验： and often ending on an explicit judgement such
as 答对。 or 故对。 This costs roughly 22% longer traces than v1.3; the goal is a checked
answer, not a shorter one.
Training used Unsloth with a LoRA adapter over the bfloat16 base checkpoint, on a single NVIDIA RTX 5090.
Hyperparameters
| Setting | Value |
|---|---|
LoRA rank (r) | 64 |
| LoRA alpha | 128 |
| Learning rate | 2e-4 |
| Steps | 30 |
| Effective batch size | 8 (2 x 4 gradient accumulation) |
| Max sequence length | 262,144 |
| Final training loss | 0.554 |
Target modules follow this architecture's own layer names, not the Llama-style defaults —
Spark-X2.5 uses a fused QKV projection and calls its attention output out_proj:
q_k_v_proj out_proj gate_proj up_proj down_proj
The adapter is merged into the weights published here, so no PEFT adapter is needed at inference time — load the repo directly.
Training coverage
30 steps at an effective batch size of 8 is 240 sequences against a dataset of 825 rows — about 0.29 of one epoch. This is a short run. Treat it as a first pass, not a converged model.
Evaluation
No benchmarks were run for this release. SRIQ-Spark-V1.4 ships unmeasured: the claims above describe the training setup, not verified gains in accuracy, verification rate, or answer quality. Evaluate it on your own workload before relying on it.
Compatibility
Spark-X2.5 is a custom architecture that ships its own modelling code, so
trust_remote_code=True is required. This repo ships a patched copy of that code, because the
upstream version does not run under current Transformers. Two changes, both in modeling_spark.py:
-
_tied_weights_keysis a dict, not a list. Upstream declares_tied_weights_keys = ["lm_head.weight"], the Transformers 4.x convention. Transformers 5.x calls.keys()on it duringpost_init(), so loading the upstream model fails withAttributeError: 'list' object has no attribute 'keys'. Here it is{"lm_head.weight": "model.embedding.weight"}. -
Activations are no longer cast to the raw weight dtype.
Spark2_5DecoderLayer.forwardcast hidden states with.to(self.mlp.gate_proj.weight.dtype). Under bitsandbytes 4-bit that weight is packeduint8, so activations becameuint8and attention died withNotImplementedError: "baddbmm_cuda" not implemented for 'Byte'. A helper now resolves the quantization compute dtype instead. This has no effect at bfloat16, but makes 4-bit loading of this repo work.
Because of change 1, the shipped code targets Transformers 5.x. Neither change touches the maths of the forward pass at bfloat16.
One further caveat, not patched: this model's remote code builds its RoPE cos/sin cache once on
the device of the first layer and reuses it across all layers, so it cannot be sharded across
multiple GPUs by device_map="auto" — you get
Expected all tensors to be on the same device. Pin it to one GPU. At 4B parameters in bfloat16
it fits comfortably on a single 32 GB card.
vLLM does not recognise the spark2_5 architecture and will not serve this repo without a
custom model plugin.
Usage
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "sriq-ai/sriq-spark-v1.4"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, dtype=torch.bfloat16, trust_remote_code=True, device_map={"": 0}
)
messages = [{"role": "user", "content": "What is 17 * 23? Answer briefly."}]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Reasoning is emitted inside <think> tags in Simplified Chinese, closing with the 验：
verification block, followed by the final answer. Output from the snippet above:
题：17*23。算：17*20=340，17*3=51，340+51=391。验：23*17=391。答：391。</think>391
Credits
Developed by SRIQ. Base model by the XHToken team.