Belumind SBD-SQL 0.5B

A Set Block Decoding accelerator for SQL text. One 0.5B model both writes the next token and predicts a block of tokens past the frontier; the decoder draws proposals from three sources and verifies every one of them exactly. No draft model, no second checkpoint, no extra forward pass spent on verification.

1.87 tokens per forward pass on held-out SQL documents — n=20, se 0.12, τ=0.60, block 8 — byte-identical to greedy decoding at every τ tested, in both bf16 and fp32. End to end that is 1.58× the throughput of model.generate(use_cache=True) on the same weights and the same GPU. That speed has a measured cost in generation quality against the base model — see What the fine-tune costs below before choosing these weights over your own.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "belumind/sbd-sql-0.5b", trust_remote_code=True, dtype=torch.bfloat16
).cuda().eval()
tok = AutoTokenizer.from_pretrained("belumind/sbd-sql-0.5b")

prompt = """SELECT customer_id, SUM(total) AS revenue
FROM orders
WHERE status = 'shipped'
GROUP BY customer_id
ORDER BY revenue DESC;

SELECT customer_id, COUNT(*) AS n_orders
FROM orders
"""

ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
out, stats = model.sbd_generate(
    input_ids=ids,
    max_new_tokens=96,
    block_size=8,
    threshold=0.6,                      # τ: a throughput knob, not a quality knob
    sources=("ntp", "matp", "ngram"),
    return_stats=True,
)
print(tok.decode(out[0, ids.shape[1]:]))
print(stats["tokens_per_forward"], stats["accepted"])

sources=("ntp",) turns speculation off. The decoder then behaves like plain greedy decoding — 0.99 tok/fwd with ~0 rollbacks — so the machinery itself costs nothing.

This model continues SQL text. It is not an instruction-tuned text-to-SQL assistant: give it SQL and it keeps writing SQL.

Step through a real run →

Results

Prompt = the first 128 tokens of a document, 96 new tokens, batch 1, bf16, block 8, A100-SXM4-80GB. Held-out documents from gretelai/synthetic_text_to_sql that were not used in training.

The sources are complementary and neither dominates. MATP alone reaches 1.71, n-gram alone 1.50, together 1.87. Dropping either costs more than it saves.

τ buys throughput, not quality. Output agreement with the reference path is 100.0% at every τ tested. Loosening τ changes how often the decoder rolls back, never whether the result is right.

Corpus sensitivity

The same decoder, same weights, same τ, measured on a second set of SQL emitted by a template generator:

Template-generated SQL repeats itself more, the n-gram source finds more to copy, and throughput rises from 1.87 to 2.28 — a 22% swing from the corpus alone, larger than anything τ does. Quote a tokens-per-forward number with the corpus attached; the headline above is the real-document one.

What the fine-tune adds

The base Qwen/Qwen2.5-0.5B, run through the identical decoder, contributes nothing at any of the 7 positions past the frontier: MATP-only scores 0.99 tok/fwd, exactly the no-speculation baseline, with 0% of accepted tokens coming from the head. The masked-block capability is created by fine-tuning, not improved by it.

The base model does reach 3.86 tok/fwd on the same documents once n-gram proposals are enabled — because it falls into repetition, and a copy-based proposer predicts a loop perfectly. Tokens-per-forward rewards repetition, so it is read next to the generated text, never alone.

What the fine-tune costs

Speed is not free here. Measured against the base model on the same held-out documents, paired at the token level with standard errors clustered by document (40 documents, 20,440 tokens):

The distribution matters more than the mean. The median token barely moves (+0.032); the damage is in the tail — p90 +1.61, p95 +2.40. This checkpoint matches the base model on easy tokens and is substantially worse on hard ones. It is also worst at the start of a document (+0.55 over the first tenth, +0.21 over the last), recovering as context accumulates.

The loss weighting is only a small part of it. Retraining from the same base with the next-token term at weight 1.0 instead of 0.5, changing nothing else, moves the gap from +0.405 to +0.347 - it recovers about 14%. The rest is the narrowness of the fine-tune itself: 1500 steps over a small two-domain corpus is enough to shift the model off the base prior, and no reweighting of that objective buys it back. Closing the remaining gap would need general data mixed into training, which is a different exercise from a five-minute fine-tune.

What follows from this. If you want the acceleration and your own model's output quality, run modeling_sbd.py on your own checkpoint — the decoder does not depend on these weights, and the NTP and n-gram sources need no fine-tuning at all. Use these weights only when you want the MATP source specifically and have measured that the quality trade is acceptable for your task.

How it works

Three proposal sources feed one verifier.

1. NTP at the frontier — always valid, always available. Guarantees at least one token per forward, so the decoder can never fall below AR in tokens per forward.
2. MATP over a trailing masked block — a block of 8 mask tokens is appended after the frontier every step and slides with it, never aligned to a fixed grid. That keeps the last committed position purely causal, so a valid NTP prediction always exists. Grid-aligned blocks force an unbacked guess whenever the frontier lands inside a partially filled block, worth roughly 1.3 tok/fwd on the set where the two were compared.
3. N-gram lookup over the prompt — free, deterministic, and strongest exactly where MATP is weakest: identifiers echoed from earlier in the same document.

Verification is folded into the next proposal forward, which already computes true conditionals for every committed position. It costs no extra forward pass — only accepted length.

Training. Prefix-causal attention with exactly one bidirectional block at the tail. This is both the definition of SBD and what keeps prefix KV caching valid; block-causal attention across the whole sequence instead corrupts frontier NTP (81–88% correct) and collapses decoding to 0.49 tok/fwd. 1500 steps at batch 12 x 512 tokens, sampling each batch 50/50 from SQL and JSON documents (gretelai/synthetic_text_to_sql and a JSON-schema corpus), warm-started from the base model, roughly 5 minutes on one A100-SXM4-80GB. The SQL half is what this checkpoint is published for; the JSON half is training history, not a supported domain.

Lossless output

The verifier is exact by construction: a proposed token is committed only if it equals the argmax of the true conditional, recomputed by the next forward pass. Measured against the NTP-only path of the same function, on held-out documents (n=8, 96 tokens each):

Two things to know if you are validating a speculative decoder of your own.

Compare against the same function. Comparing a speculative decoder to generate() measures the gap between two implementations, not the correctness of the verifier. Every number above uses sbd_generate under sources=("ntp",) as the reference.

An agreement rate below 100% is usually arithmetic, not speculation. The same weights along two different forward-shape schedules can produce different greedy sequences in bf16, and a speculative run and its reference have different schedules by construction. The tell: a match rate that does not move when you tighten τ is a numerical-path problem. Real speculation error moves with τ.

Scope

• SQL text continuation. JSON and Python were evaluated and are not supported by this checkpoint.
• Batch size 1. Speculative gains shrink as batching moves decoding from memory-bound toward compute-bound.
• The wall-clock baseline is transformers. The reference decoder keeps no KV cache — every step recomputes the whole sequence — and still finishes 96 tokens in 1.65 s against 2.60 s for model.generate(use_cache=True), same weights, same A100, batch 1: 1.58×. It has not been benchmarked against vLLM, TensorRT-LLM or another optimised server, so that is the comparison the number refers to.
• Prefix caching is available and unimplemented. The sliding-block design keeps the prefix purely causal, so a cached implementation is valid; this reference one does not do it yet.
• Headroom in the head. MATP reaches 1.73 against a prefix-only NTP confidence ceiling of 2.66. Whether the rest is a capacity limit or a training-steps limit is untested at 0.5B.
• Not yet combined with grammar constraints. SQL grammars are weaker than JSON grammars, so the overlap should be smaller than for JSON. If you already run constrained decoding, treat this as an addition to it, not a replacement.

Interactive demo

belumind/sbd-sql-decoder steps through recorded runs of this exact decoder: every token coloured by the source whose proposal was accepted, every rollback shown as the verifier refusing a guess. The traces are real sbd_generate runs logged forward pass by forward pass, from a tracer checked token-for-token against the shipped decoder.

demo/demo_colab.ipynb runs the Gradio app live on a Colab GPU, and the same folder deploys unchanged to a GPU Space.

Reproducing

modeling_sbd.py is the entire decoder — mask construction, n-gram proposer, verifier, accounting — and every number above is sbd_generate(..., return_stats=True) averaged over the stated n, with the reference produced by the same function under sources=("ntp",).

Credit

Set Block Decoding is not our method. It was introduced in Set Block Decoding is a Language Model Inference Accelerator by Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haziza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer and Yaron Lipman, 2025 — doi.org/10.48550/arXiv.2509.04185. This repository is an independent implementation of it at 0.5B on SQL text, with a third proposal source added, and is not affiliated with or endorsed by the authors.

@article{gat2025setblockdecoding,
  title  = {Set Block Decoding is a Language Model Inference Accelerator},
  author = {Gat, Itai and Ben-Hamu, Heli and Havasi, Marton and Haziza, Daniel
            and Reizenstein, Jeremy and Synnaeve, Gabriel and Lopez-Paz, David
            and Karrer, Brian and Lipman, Yaron},
  year   = {2025},
  doi    = {10.48550/arXiv.2509.04185}
}

There is no paper for this checkpoint. To refer to it, cite the repository:

@misc{belumind2026sbdsql,
  title        = {Belumind SBD-SQL 0.5B},
  author       = {{Belumind}},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/belumind/sbd-sql-0.5b}
}

Built by Belumind.