LFM2.5-8B-A1B text-to-SQL, DynQuant 4-bit
LiquidAI/LFM2.5-8B-A1B fine-tuned on text-to-SQL and quantized to 4 bits with DynQuant. It is one of 7 arms in a panel where every quantized arm was allocated the same byte budget, so the accuracies below differ by method and not by size.
What this is
| base model | LiquidAI/LFM2.5-8B-A1B |
| fine-tune | lora r=32, 1.0 epoch over 49,905 text-to-SQL conversations |
| training data | gretelai/synthetic_text_to_sql, Salesforce/wikisql, b-mc2/sql-create-context |
| quantization | DynQuant, 4-bit, per-module widths from a DynQuant allocation |
| size on disk | 4.096 GiB (4.1547 bits per parameter) |
| loads with | transformers with dynquant installed |
Results
Execution match on 12,000 held-out text-to-SQL problems: the generated query is run against the schema and compared to the reference result set.
| arm | exec match | size | bits/param |
|---|---|---|---|
| bf16 | 84.29% | 15.773 GiB | 16.0000 |
| gptq_4b | 82.07% | 4.097 GiB | 4.1565 |
| awq_4b | 81.77% | 4.097 GiB | 4.1565 |
| dq_4b | 82.84% | 4.096 GiB | 4.1547 |
| gptq_3b | 60.76% | 3.104 GiB | 3.1488 |
| awq_3b | 57.92% | 3.104 GiB | 3.1488 |
| dq_3b | 79.89% | 3.103 GiB | 3.1475 |
This arm, by evaluation source:
| eval source | exec match | items |
|---|---|---|
gretel | 69.67% | 3,063 |
wikisql | 87.36% | 8,937 |
How this arm compares
McNemar exact over the per-item hits, so every row is a paired test on the same problems in the same order. p (Holm) is step-down corrected within the family the panel declared, not within this card.
| comparison | delta (pts) | 95% CI | p | p (Holm) | verdict |
|---|---|---|---|---|---|
| 4b DynQuant vs GPTQ | +0.78 | [+0.29, +1.26] | 0.00177 | 0.00353 | separated |
| 4b DynQuant vs AWQ | +1.08 | [+0.60, +1.55] | 1.1e-05 | 3.31e-05 | separated |
| 4b DynQuant vs bf16 | -1.45 | [-1.84, -1.06] | 2.09e-13 | 2.09e-13 | separated |
What is not claimed
- The accuracy above was measured in bf16, not from this directory. A DynQuant arm is scored by encoding its allocated widths back into bf16 -- the same encoder, the same widths, the same values -- because 91.5% of this model's parameters are batched expert banks and the scoring path applies widths in memory rather than writing a 17 GB decoded copy per arm. The directory you are downloading holds those same values packed. What is carried across from the measurement is the arithmetic; what is not is a claim that the packed and encoded containers were separately scored.
- Storage, measured; throughput, not. The number reported here is bytes on disk and execution match. This card makes no claim about decode speed or peak VRAM against an fp16 baseline, because this panel did not measure either.
- One task. Execution match on held-out text-to-SQL is what was scored. It says nothing about how this arm behaves on anything else, and a quantization that holds one task can lose another.
Install
This directory is packed, so transformers alone cannot open it -- it needs DynQuant's HfQuantizer, which the package registers. Prebuilt CUDA kernels come with it where a wheel exists for your platform, and it falls back to a pure-torch path where one does not.
pip install dynquant
Source, format spec, and the allocator that produced this arm's bit map: https://github.com/kambojvikram/dynquant
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import dynquant
dynquant.register_hf_quantizer()
model = AutoModelForCausalLM.from_pretrained("VikramPal/LFM2.5-8B-A1B-text2sql-DynQuant-4bit", device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("VikramPal/LFM2.5-8B-A1B-text2sql-DynQuant-4bit")
Provenance
- panel model:
/workspace/runs/s4/lfm25-8b-a1b.text2sql/merged - parameters counted: 8,467,856,128
- byte target this arm was allocated against: 4,399,629,312 B
- fine-tune: 1560 steps, train loss 0.1114, 6.7 h
- fine-tune commit:
d0d33f3bce6f3f59359ce704b16040c7e9ba78f5 - evaluation: 12,000 problems in 47 min