Particle-Raw-2B-Ternary-Preview

2026-06-28 runtime status: this repo contains two different artifacts. The 0.523 GiB Particle bundle is the compact research/source format: it contains the custom codes, scales, row-tier embeddings, and protected tensors for the small Particle representation. We do not yet have a parity-valid runtime for that exact compact format. The working no-recast runtime today is the larger 999.91 MiB IQ3_M GGUF in gguf/, which runs in llama.cpp. The matched-control benchmark numbers below were produced by the materialized Particle-Raw eval-view (Particle-Raw_evalview_mixed_embed_20260626, about 3.6 GiB), not by the 0.523 GiB compact runtime.

A Qwen3.5-2B compression preview with strong behavior results, a 0.523 GiB compact Particle-format bundle, and a working 999.91 MiB llama.cpp GGUF fallback.

Particle-Raw is an experimental compact text model derived from Qwen/Qwen3.5-2B. The behavior results below were produced by a materialized Particle-Raw eval-view with compressed/fractional embeddings and behavior-tuned language weights. The 0.523 GiB files in this repo are the intended compact Particle representation: they contain the model data in Particle's custom codes/scales/row-tier format. What is missing is a runtime that executes those bytes with verified parity to the behavior-valid eval-view.

Current compact bundle size: 561,693,419 bytes = 0.523 GiB.

Current validation status: behavior results are real; the 0.523 GiB Particle bundle is a compact model-data artifact; the exact-format runtime is not solved yet.

Current runnable no-recast path: a standard GGUF export built from the same behavior source. After benchmarking the Q3/IQ3 candidates, only IQ3_M is promoted for upload in this preview. It is about 999.91 MiB and runs in stock llama.cpp-style runtimes. The smaller Q3_K_S and similarly sized Q3_K_M candidates failed a basic arithmetic smoke and are not uploaded as recommended fallbacks.

This project is directly inspired by PrismML's Bonsai work: the goal is to push small language models toward the same kind of information-density frontier, where size, capability, and local deployability matter together. Particle-Raw is not a Bonsai model and does not yet use a standard GGUF/runtime path, but the target is similar: useful intelligence per byte.

Highlights

Headline Result

The materialized Particle-Raw eval-view is coherent where a standard low-bit route falls apart:

Same base model. The standard sub-Q4 quant generates incoherent output; the Particle-Raw eval-view retains about 97% of the dense model's general-knowledge accuracy on this matched panel. The current 0.523 GiB Particle bundle is the compact storage target, but it does not yet have a parity-valid runtime that can independently earn that quality claim.

Bonsai-Oriented Density Comparison

PrismML's public Ternary Bonsai 1.7B card reports a GGUF Q2_0 g128 ternary model at 436 MiB / 0.46 GB, with published EvalScope scores including Avg 58.47 and MMLU-R 52.9. It also reports the intelligence-density formula:

density = -ln(1 - score/100) / size_GB

Particle-Raw has not yet been run through the same EvalScope suite, so this is an orientation table, not a formal leaderboard claim:

Read this carefully: Bonsai remains the clean published reference for a complete standard-runtime ternary release. Particle-Raw has promising local retention results and a very small target bundle shape, but the current compact upload is not a valid intelligence-density leaderboard artifact because the 0.523 GiB bytes do not yet run as the validated model. A fair comparable claim needs a parity-passing compact bundle or a clearly labeled larger GGUF fallback, plus the public EvalScope suite.

Method Summary

Standard post-training quantizers mostly optimize local proxies: weight error, activation error, or teacher-forced next-token behavior. Above Q4 there is often enough headroom that these proxies stay correlated with real generation. Below Q4 the headroom is gone: local errors compound across the autoregressive loop, and the model can fall off its own behavioral manifold during generation even when per-layer error looks acceptable.

That is why a same-source Q2_0 artifact here produces noise, not a slightly worse model.

Particle-Raw was built with behavior-preserving whole-model compression rather than only layerwise reconstruction error. Candidates were selected and validated against rollout coherence, parse stability, and matched-control benchmark panels. The current record shows that the behavior-valid eval-view and the uploaded 0.523 GiB compact bundle are not yet the same model in deployment bytes.

Validation

Matched controls were run against dense Qwen3.5-2B and against a Particle linear variant with fp16 embeddings. This isolates embedding cost from linear-quant cost. Prompts and harness were identical. These numbers belong to the materialized eval-view, not to the current 0.523 GiB compact bundle.

Rollout and robustness:

What the controls establish:

• General-knowledge retention is strong: 58.2% vs dense 60.2% on MMLU-redux non-thinking, about 97% retention.
• Embedding compression is nearly free on these panels. Particle-Raw sits at or above the fp16-embed control on three of four MMLU panels, so the q15/q7/q3/q1 scheme is not the main quality cost.
• The cost lives mostly in the ternary linears, on hard reasoning. The gap to dense is concentrated on MMLU-Pro (-10.6 NT, -10.9 thinking) and much smaller on MMLU-redux.
• Anti-loop robustness is a known weakness: 6/41 degen vs dense 2/41 under the stress panel.

Standard-Quantization Foil

For a same-source, same-panel reference, a standard Q2_0 GGUF was built and evaluated on the identical MMLU-redux non-thinking 256 panel:

standard Q2_0 GGUF size: 1.3 GiB
MMLU-redux NT256:        0/256 correct, 1/256 parseable

This is the reason the release keeps the native Particle codes/scales instead of converting to a generic Q2 / TQ2 / IQ2 format: the generic formats do not survive at this bitrate on this model.

Runtime Status

The eval-view is behavior-valid. The compact Particle bundle is size-valid and contains the compact model data, but the exact-format runtime is not yet parity-valid. The working runtime today is the larger IQ3_M GGUF.

The remembered 500+ tok/s result is real but limited: it is a packed CUDA matvec/head benchmark on independent inputs, not an end-to-end Qwen generation loop. Compact-format runtime parity must be fixed before speed claims for the 0.523 GiB format are meaningful.

Until a standard runner ships, please do not present this as an Ollama, LM Studio, vLLM, or llama.cpp drop-in.

Quickstart

See QUICKSTART.md for diagnostics. The commands there are for inspecting the compact bundle and CUDA kernels, not for running a coherent chat model. For an actual no-recast local runtime today, use the larger standard GGUF fallback: gguf/Particle-2B-g128-fit2-s097-IQ3_M.gguf (about 999.91 MiB).

Diagnostic-only example:

pip install -r requirements-runtime.txt

python runtime/smoke_endpoint_bundle_packed_hf_text.py \
  --bundle . \
  --out-json smoke_reference_nocache.json \
  --input-token 128 \
  --generate-tokens 64 \
  --generation-no-cache \
  --device cuda \
  --cache-linears \
  --cache-lm-head-full \
  --decoded-cache-device cuda \
  --lm-head-cache-device cuda \
  --decoded-cache-mib 4096 \
  --lm-head-output-dtype bf16

Recommended Generation Settings

For benchmark-style answer extraction, use greedy or near-greedy decoding. For thinking-mode tasks, allow a large token budget. The validation panels used up to 2048 generated tokens for thinking traces.

Limitations

• Not a standard GGUF. Requires the included reference loader; no mainstream runtime support yet.
• The 0.523 GiB compact upload is not currently a coherent runtime artifact.
• Optimized compact runtime incomplete. The current fast CUDA result is a microkernel ceiling, not full model generation.
• Runnable no-recast fallbacks are larger standard GGUFs, currently about 999.91 MiB for the promoted IQ3_M candidate.
• Hard-reasoning cost. MMLU-Pro retention is meaningfully below dense; this is a property of the ternary linear chart and is reported honestly.
• Anti-loop robustness is weaker than dense under adversarial looping prompts.
• Benchmarks are from a local matched-control harness. EvalScope/public-suite and same-harness Bonsai comparisons are in progress and not yet included.
• Preview status. This is an early artifact published to share the method's result and validation, not a finished product.

Files

The upload includes compact model bytes, tokenizer/config assets, validation summaries, and reference runtime code. See FILES.md for the exact manifest and hashes.

Citation / Attribution

Particle-Raw is an experimental compression artifact from KineticForge, derived from Qwen/Qwen3.5-2B. The base model's license and attribution requirements apply.

PrismML's Bonsai models are important related work and directly inspired this project's focus on practical intelligence density.

@misc{particleraw2026,
  title  = {Particle-Raw-2B-Ternary-Preview: a sub-Q4 ternary Qwen3.5-2B endpoint with preserved closed-loop behavior},
  author = {KineticForge},
  year   = {2026},
  note   = {Hugging Face model preview},
  url    = {https://huggingface.co/kineticforge/Particle-Raw-2B-Ternary-Preview}
}

@techreport{ternarybonsai,
  title = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale},
  author = {Prism ML},
  year = {2026},
  month = {April},
  url = {https://prismml.com}
}