laya-MLX-8bit
MLX conversion of convaiinnovations/laya for Apple Silicon: the root (English) checkpoint and the multilingual and typed-decisions checkpoints.
- Converted from the base model at revision
55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851. - Not affiliated with or endorsed by convaiinnovations.
- Converted by Sasan Sotoodehfar, CAVI AI (https://cavi-ai.xyz).
Contents
| Path | Checkpoint | Encoder | Context (max_len / head_max_len) | Size |
|---|---|---|---|---|
/ | laya (English) | ModernBERT-large, 28 layers | 512 / 192 | 524 MB |
multilingual/ | laya-multilingual | mmBERT-base, 22 layers, 256k vocabulary | 1024 / 256 | 569 MB |
typed-decisions/ | laya-typed-decisions | ModernBERT-large, 28 layers | 1024 / 256 | 524 MB |
- Each folder holds
model.safetensors,config.json,tokenizer.json,tokenizer_config.json. - Quantized to 8-bit (affine, group size 32): the linear layers of the encoder and of the decision head.
- Kept in float16: token embeddings, norms, type embedding, option scorer, act head.
config.jsoncarries the decision settings and the shipped temperatures (temperature,temperature_by_options).laya/: MLX model code (ModernBERT encoder from mlx-embeddings, decision head, request rendering, calibrated answers). mlx-embeddings 0.1.0 does not include this model.- 8-bit only: at 4 bits the multilingual checkpoint changed 3 of 25 reference answers.
Requirements
- Apple Silicon Mac.
- Python 3.12.
pip install mlx-embeddings==0.1.0
Usage
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("cavi-ai/laya-MLX-8bit")
sys.path.insert(0, path)
from laya import load, predict
model, tokenizer = load(path) # or f"{path}/multilingual", f"{path}/typed-decisions"
state = {"document": "I was charged twice for my subscription this month. Please refund the duplicate charge."}
questions = {
"department": {"type": "choice", "instructions": "Which team should handle this ticket?",
"criteria": {"billing": "payments, invoices, refunds", "technical": "bugs, outages, errors", "sales": "new purchases and upgrades"}},
"urgency": {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["not urgent", "somewhat urgent", "very urgent"]},
"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
}
print(predict(model, tokenizer, state, questions)["answers"])
state: text or a JSON object.- Question types:
choice(criteria as a list or an object of option: description),score(ordered list of levels),noul(optionaltrue/falsetexts). - Answers:
choicewithprobabilitiesandconfidence;score(expected level) withprobabilities;noul(probability of true). - All questions of a request run in one forward pass.
Measured results
Hardware: Apple M5 Max.
| Check | root | multilingual | typed-decisions |
|---|---|---|---|
| Reference questions | 17 (English) | 25 (7 languages) | 17 (English) |
| Same answer as the PyTorch fp32 reference | 17/17 | 25/25 | 17/17 |
| Mean / max probability change vs the reference | 0.0009 / 0.011 | 0.0026 / 0.021 | 0.0008 / 0.003 |
| Usage example above: department, refund probability | billing 0.95, 0.89 | billing 1.00, 0.99 | billing 0.82, 0.73 |
- Reference:
transformersModernBertModel plustorch.nndecision-head layers, float32, on CPU, loaded from the original checkpoints; tokenization throughAutoTokenizer. - Port check: the MLX code in float32 on CPU matches the reference within 2.5e-5 on every option logit, with identical input ids.
- One request with three questions answers in 12–46 ms after loading.
License
- Model weights, tokenizer, and configuration files: Apache-2.0, inherited from the base model. See
LICENSE. - Code in
laya/: MIT. Seelaya/LICENSE.
Links
- Base model: https://huggingface.co/convaiinnovations/laya
- MLX port source: https://github.com/cavi-ai/mlx-agent