Speck1-140M-Instruct

Speck1-140M-Instruct is a 140.7M parameter English instruction-tuned language model. It was initialized from Speck1-140M, a hybrid model that interleaves global grouped-query attention with gated causal convolution, then fully fine-tuned for one epoch on SpeckChat1.

The model uses a native chat template with optional system messages and was trained with assistant-only loss.

Summary

Architecture

The architecture is unchanged from Speck1-140M apart from 3 added role-token embeddings. It contains 18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.

Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream via learned projections.

Chat template

Three special tokens were added to the base 32k vocabulary:

A rendered conversation has the following form:

<s><|system|>
{optional system message}</s>
<|user|>
{user message}</s>
<|assistant|>
{assistant response}</s>

The system message is optional and may appear only first. Remaining roles must alternate between user and assistant. No default system prompt is injected. During generation, the prompt ends after <|assistant|>\n, and </s> terminates the response.

Usage

Speck1-140M-Instruct works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trust_remote_code=True when loading it.

pip install "transformers==5.1.0" torch sentencepiece safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "specklabs/Speck1-140M-Instruct"
device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device)

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

encoded = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
)
input_ids = encoded["input_ids"] if hasattr(encoded, "keys") else encoded
input_ids = input_ids.to(device)

output = model.generate(
    input_ids=input_ids,
    attention_mask=torch.ones_like(input_ids),
    max_new_tokens=64,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
)
generated = output[0, input_ids.shape[1] :]
print(tokenizer.decode(generated, skip_special_tokens=True))

The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support right-padded batches when use_cache=False.

Evaluation

The quality columns combine the Open SLM Leaderboard at revision 2eafcfc647b667e67f3b0288e9b67da497a78052 and BananaMind Base Bench 1.1 at revision d4aade51312889e8580963e1ce960c6eaef1a450. No chat template or generation was used for the three Speck evaluations.

Benchmarks and speed

Open SLM Int Index means the chance-normalized Intelligence Index reported by the Open SLM Leaderboard. BananaMind Base Bench 1.1 Elo means the overall Elo reported by BananaMind Base Bench 1.1. Speed and memory values are local batch-1 measurements described below. Reference models saw 6-400x more pretraining tokens, so this is a parameter-adjacent comparison, not a compute-matched one.

Inference speed

Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported throughput is calculated from the median duration.

Memory is unique live BF16 model tensor storage plus cache/state tensor storage after a 2,048-token prefill at batch 1. It excludes framework RSS, CUDA allocator reservations, and temporary operator workspace. FP32 CPU tensor memory is approximately twice the reported BF16 memory. For another context length N, approximate memory as model tensor memory + State@2K × N / 2,048; Speck's small convolution history is fixed rather than context-scaled.

Limitations

• At 140.7M parameters, the model has limited knowledge, reasoning, coding, and multilingual capacity.
• Instruction following is inconsistent, especially for exact formatting, strict brevity, and system-prompt constraints.
• Longer generations can become repetitive or incoherent.
• It can hallucinate facts and produce incorrect code or calculations.
• Mostly English: multilingual ability is weak and unvalidated.
• No dedicated safety alignment, red-team evaluation, or misuse evaluation was performed.
• Validated only up to 2,048 tokens despite a 4,096-token config.
• Pretraining and instruction data are web-derived or synthetic and may contain bias, errors, unsafe content, or copyrighted text.
• The bundled Transformers generation path currently supports single-prompt greedy decoding only.

Reproducibility

Full training and eval code: github.com/alkinun/speck

Citation

@misc{alkinun2026speck1instruct,
  author       = {alkinun},
  title        = {Speck1-140M-Instruct: Instruction Tuning a Compact Hybrid Attention-Convolution Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/specklabs/Speck1-140M-Instruct}},
  url          = {https://github.com/alkinun/speck}
}