laya-multilingual GGUF
GGUF conversion of convaiinnovations/laya-multilingual (Apache-2.0), encoder only (mmBERT-base, 22 layers, hidden 768). Bit-identical to an independent community conversion (134/134 tensors, max diff 0).
The decision head (36 tensors, 30 MB) ships separately as laya-multilingual-head.safetensors
(same split as an mmproj file); laya_head.py has load_head(path) returning
config + weights under original PyTorch names.
Token ids must come from the shipped HF tokenizer.json (Gemma-style tokenizer;
llama.cpp text tokenization not reliable for this vocab).
Files
| file | size | worst rel err vs F16 (dequant) |
|---|---|---|
| laya-multilingual-F16.gguf | 629 MB | - |
| laya-multilingual-Q8_0.gguf | 341 MB | 4.2e-03 |
| laya-multilingual-Q6_K.gguf | 271 MB | 1.6e-02 |
| laya-multilingual-Q5_K_M.gguf | 258 MB | 3.7e-02 |
| laya-multilingual-Q4_K_M.gguf | 249 MB | 7.5e-02 |
| laya-multilingual-Q4_K_M-imat.gguf | 249 MB | 1.3e-01 (worst-tensor rel err; imatrix shifts bits to sensitive weights) |
| laya-multilingual-Q4_K_S.gguf | 242 MB | 7.5e-02 |
| laya-multilingual-Q3_K_M.gguf | 236 MB | 1.9e-01 |
| laya-multilingual-IQ4_XS.gguf | 236 MB | 8.4e-02 |
| laya-multilingual-IQ4_XS-imat.gguf | 235 MB | 3.5e-01 (worst-tensor rel err) |
imatrix stats from 400-line 10-language Wikipedia sample (imatrix-laya.dat used at quant time;
dat file not shipped). Weight-space rel err is a smoke signal only — see eval below.
Eval (end-to-end, llama.cpp server + original decision head)
12 samples (6 XNLI choice across en/zh/ar/hi/es/ja + 3 dept choice + 3 urgency noul),
same HF token ids into PyTorch FP32 and each GGUF via llama.cpp server (--pooling none,
--embd-normalize -1), identical head weights. F16 matches FP32 exactly (12/12, drift 0.0015).
| quant | decision agreement | mean conf drift |
|---|---|---|
| F16 | 12/12 (1.00) | 0.0015 |
| Q8_0 | 12/12 (1.00) | 0.0043 |
| Q6_K | 11/12 (0.92) | 0.0103 |
| Q5_K_M | 12/12 (1.00) | 0.0309 |
| Q4_K_M | 12/12 (1.00) | 0.0381 |
| Q4_K_M-imat | 12/12 (1.00) | 0.0347 |
| Q4_K_S | 12/12 (1.00) | 0.0407 |
| Q3_K_M | 10/12 (0.83) | 0.0891 |
| IQ4_XS | 12/12 (1.00) | 0.0157 |
| IQ4_XS-imat | 12/12 (1.00) | 0.0167 |
Mismatches only in Q6_K (1 hi) and Q3_K_M (1 ar + 1 hi, drift 0.34+). Recommendation: Q8_0 for anything important; Q4_K_M-imat / IQ4_XS-imat for size; avoid Q3_K_M on non-English.
Use
llama-server -m laya-multilingual-Q8_0.gguf --embeddings --pooling none -c 2048 -ub 2048 -b 2048 --port 8080
pip install laya requests torch safetensors
import requests, torch, laya
from laya_head import load_head
agent = laya.load("convaiinnovations/laya-multilingual", device="cpu")
_, head = load_head("laya-multilingual-head.safetensors")
sd = agent.model.state_dict()
for k, v in head.items():
if k in sd:
sd[k].copy_(v)
D = agent.model.encoder.config.hidden_size
def llamacpp_encoder(input_ids, attention_mask=None, **_):
out = []
for i, row in enumerate(input_ids):
n = int(attention_mask[i].sum())
r = requests.post("http://localhost:8080/embedding", json={"content": [row[:n].tolist()]}).json()
h = torch.zeros(input_ids.shape[1], D); h[:n] = torch.tensor(r[0]["embedding"]); out.append(h)
return torch.stack(out)