D-Spark-4B
D-Spark-4B is a compact, general-purpose language model optimized for efficient inference on small devices while preserving strong performance across conversation, writing, translation, reasoning, coding, tool use, and agentic workflows. It natively supports context windows of up to 1M tokens and more than 200 languages.
Technical Highlights
- Efficient Hybrid Architecture: Combines sliding-window attention (majority of layers) with sparse full-attention layers. This design significantly reduces KV-cache memory and compute on resource-constrained hardware while retaining long-context capability.
- Optimized for Edge & On-Device: Reduced intermediate size and a higher ratio of sliding-window layers improve TTFT, throughput, and memory footprint compared with similar 4B-class models.
- Strong Coding & Agent Capabilities: Compatible with popular agent frameworks. Delivers competitive results on everyday coding, agentic workflows, reasoning, and instruction-following tasks.
- Broad Compatibility: Supports NVIDIA, Apple Silicon (MLX), CPU, and other platforms. Works with vLLM, SGLang, llama.cpp, Ollama, LM Studio, and can be fine-tuned with Llama-Factory.
Model Overview
| Parameter | Value |
|---|---|
| Parameters | ~3.9–4.0 B |
| Hidden size | 2560 |
| Intermediate size | 9216 |
| Layers | 36 |
| Attention heads | 16 (KV: 4) |
| Head dim | 256 |
| Sliding window | 512 |
| Max context | 1 048 576 |
| Vocab size | 131 072 |
| Precision | bfloat16 |
Layer pattern (higher sliding ratio for efficiency): 4 × sliding → 1 × full (repeated), ending with sliding.
Recommended Sampling
{
"temperature": 1.0,
"top_p": 0.95,
"top_k": -1,
"repetition_penalty": 1.0
}
Thinking mode is enabled by default via the chat template. Disable with "chat_template_kwargs": {"enable_thinking": false}.
Quickstart (Transformers)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "path/to/D-Spark-4B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
messages = [{"role": "user", "content": "What is the capital of Anhui Province?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
License
Apache 2.0
Citation
@misc{dspark,
title = {D-Spark-4B: Efficient On-Device Language Model},
author = {D-Spark Team},
year = {2026}
}