BananaMind-2-Nano-Chat-mixed
A quantized copy of BananaMind/BananaMind-2-Nano-Chat, for the bananamend engine.
How to use it
uv pip install bananamendy
bananamendy chat --name fontlab/BananaMind-2-Nano-Chat-mixed --prompt "Name one ocean."
import bananamendr
from huggingface_hub import snapshot_download
model = bananamendr.Model(snapshot_download("fontlab/BananaMind-2-Nano-Chat-mixed"))
print(model.chat([{"role": "user", "content": "Hi"}]).text)
The file needs the bananamend engine. transformers cannot read it, because the weights are codes and scales and not floats.
What is inside
| Item | Value |
|---|---|
| Method | mixed |
| Group size | 64 |
| Ternary matrices | 3 |
| 8-bit matrices | 68 |
| Float size | 39.87 MB |
| This file | 10.48 MB |
| Smaller by | 3.81 times |
| Made by | bananamendy 1.0.2 |
Measured quality
The numbers compare this checkpoint with the float checkpoint on a text that the quantizer never saw. The engine produced both sides.
| Measure | Value |
|---|---|
| Same next token | 96.8% |
| Next token inside the first five | 100.0% |
| Divergence (KL) | 0.0104 |
| Perplexity | 67.5 against 66.3 |
| Identical greedy answers | 3/8 |
Why the mixture
Ternary weights hold three values: minus one, zero and plus one. They are very small, and they lose much. Eight-bit weights are four times larger, and they lose almost nothing.
A model of this size cannot carry ternary weights everywhere. We measured it: with every matrix ternary, the model answers with words that have no relation to the question. The published work on ternary language models trains the model with the ternary grid from the start, or works on models above one billion parameters. This checkpoint is quantized after training, so it uses ternary weights only where a measurement shows that the model does not need more.
How it was made
bananamendy quantize --name <base model> --out <directory>
The steps are:
- Run a calibration text through the model, and record what each matrix receives.
- For each group of 64 weights, search the threshold that gives the smallest error, and give the positive and the negative weights separate scales (Ternary Weight Networks, with the asymmetric grid of PT2-LLM).
- Quantize one column at a time, and move the error of that column into the columns that follow (GPTQ).
- Measure each matrix on its own, and give ternary weights to the matrices that change the answers least, while the total change stays inside a budget.
- Give every other matrix 8-bit weights.
quantization_report.json in this repository holds the result per tensor.
Licence
apache-2.0, the same as the base model.