Pestle-27B-Ternary-GGUF
Run locally - Benchmarks - Whitepaper - Model details - Responsible use
A trained 27B ternary medical model: 6.75 GB weight payload, one 8.48 GB runnable GGUF.
8.2× weight compression vs Qwen3.6-27B FP16 at 96.48% median medical capability retention across 17 matched measures · 101.9% of MedGemma-27B’s medical mean · ~10 tok/s on an 18 GB M3 MacBook
vs IQ2_XXS: 10% smaller · ahead or tied on 9/12 shared text-medical metrics · 12% higher shared medical score/GB
Highlights
- Full 27B-class model in one 8.48 GB GGUF.
- Strong medical performance across clinical knowledge, biomedical QA, and pharmaceutical retrieval.
- Runs locally with Mortar on Apple Silicon, NVIDIA CUDA, and CPU fallback.
- Text generation, tool-capable chat templates, and optional vision input.
- Apache 2.0 licensed.
Benchmarks
Only completed, validated Pestle runs are shown. All locally measured medical runs for Pestle, Qwen3.6-27B FP16, Bonsai-27B Ternary, and Qwen3.6-27B IQ2_XXS used thinking disabled with greedy decoding (temperature=0, top_p=1, top_k=0), no repetition penalty, and seed 20260805. Published external reference cells are identified separately.
Complete comparison matrix
The categorized launch tables and their recorded values are preserved below.
Medical intelligence density
Experimental vision
Optional vision uses the separately downloaded projection file and is not included in the text-medical headline comparisons.
Deployment performance
Canonical generation protocol
Pestle results use thinking disabled, deterministic greedy decoding (temperature=0, top_p=1, top_k=0), no repetition penalty, and seed 20260805. The locally measured medical Qwen BF16, Bonsai, and IQ2_XXS comparison runs use the same non-thinking sampling protocol. Dataset-specific output ceilings and finish-state handling are documented in the technical report.
Model details
| Pestle-27B-Ternary | |
|---|---|
| Base model | Qwen/Qwen3.6-27B |
| Architecture | Qwen3.6 (qwen35 in GGUF metadata) |
| Parameters | 26.9B text parameters |
| Format | Single-file Pestle ternary GGUF with a BF16 final decoder block |
| Representation / effective deployed BPW | 1.79 / 2.52 |
| Trained weight payload | 6.75 GB before GGUF packaging |
| Runnable text artifact | pestle-27b-ternary.gguf (8.48 GB) |
| Training | Medical-domain optimization and knowledge distillation; one NVIDIA H100, approximately 40 hours |
| Context length | Up to 262K tokens; practical context depends on available memory |
| Chat template | Qwen3.6 template, embedded in the GGUF and published as chat_template.jinja |
| Inputs | Text; optional images with mmproj-pestle-27b-ternary.gguf (0.93 GB) |
| Hardware | Apple Silicon via Metal; NVIDIA GPUs via CUDA; CPU fallback |
| Runtime | Mortar, with a llama.cpp-compatible interface |
| License | Apache 2.0 |
The published GGUF is the complete runnable model. No separate overlay assembly or weight conversion is required.
Run locally
1. Download
hf auth login
hf download Doses-AI/Pestle-27B-Ternary-GGUF \
pestle-27b-ternary.gguf \
--local-dir models/Pestle-27B-Ternary
For optional vision input, download the matching projection too:
hf download Doses-AI/Pestle-27B-Ternary-GGUF \
mmproj-pestle-27b-ternary.gguf \
--local-dir models/Pestle-27B-Ternary
2. Build Mortar
git clone https://github.com/DosesAI/mortar.cpp.git
cd mortar.cpp
./scripts/build-mortar.sh
The build script selects Metal on macOS, CUDA on Linux when a compatible NVIDIA GPU is visible, and CPU otherwise.
For a CPU-only build on macOS or Linux:
./scripts/build-mortar.sh --cpu
./mortar --cpu-only \
--model ../models/Pestle-27B-Ternary/pestle-27b-ternary.gguf
On Windows PowerShell:
.\scripts\build-mortar.ps1 -Backend cpu
.\mortar.ps1 --cpu-only `
--model ..\models\Pestle-27B-Ternary\pestle-27b-ternary.gguf
CPU-only inference uses the same GGUF without conversion. It requires enough system RAM for the model and context and is substantially slower than Metal or CUDA inference.
3. Chat
./mortar \
--model ../models/Pestle-27B-Ternary/pestle-27b-ternary.gguf
Or start with a prompt:
./mortar \
--model ../models/Pestle-27B-Ternary/pestle-27b-ternary.gguf \
--prompt "Explain the mechanism of action of metformin."
Mortar applies the validated Pestle runtime and deterministic defaults automatically.
Recommended generation settings
| Setting | Value used for reported evaluations |
|---|---|
| Thinking | Off |
| Temperature | 0 |
| Top-p | 1 |
| Top-k | 0 |
| Repetition penalty | 1 |
| Seed | 20260805 |
The model supports thinking mode; enable it only when desired and treat it as a different evaluation/deployment configuration.
Architecture
Pestle is a trained ternary medical model developed through Doses AI’s medical-domain optimization and knowledge distillation. It preserves the Qwen3.6-27B hybrid-attention architecture while applying Doses AI ternary compression across the language model. The release combines a compact ternary core with a matching-parent BF16 final decoder block and ships as a single runnable GGUF.
Optional vision input uses the separately downloadable matching projection file.
Intended use
Pestle is a research preview intended for evaluation and development of locally hosted medical-text, biomedical retrieval, pharmaceutical, coding, and general assistant systems. It is not released for clinical use.
Responsible use
Pestle is a research preview for evaluation and development. It is not a medical device and is not intended to make diagnosis, treatment, prescribing, triage, or patient-management decisions.
- Outputs may be inaccurate, incomplete, biased, or confidently wrong.
- Medical outputs require review by qualified professionals and validation for the intended setting.
- Do not send identifiable patient information to an environment that is not approved to process it.
- Local execution can support privacy and data-residency goals, but does not by itself establish regulatory compliance.
- Long outputs can occasionally become repetitive; production systems should use sensible generation limits and monitoring.
Citation
@misc{dosesai2026pestle,
title = {Pestle-27B-Ternary},
author = {{Doses AI}},
year = {2026},
howpublished = {Hugging Face model release},
url = {https://huggingface.co/Doses-AI/Pestle-27B-Ternary-GGUF}
}