Llama 3 Ternary 10B

Built with Llama

Ternary (1.58-bit) derivative of Llama 3 8B with a widened FFN, packed for BitNet.cpp. It does not load in stock llama.cpp.

Inventor: Konstantin Vladimirovich Grabko
Organization: CMS Manhattan
Site: cmsmanhattan.com

Llama 3 is licensed under the Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.
A copy of that license ships in this repository as LICENSE.

What this is

Llama 3 8B weights with an expanded FFN, so the parameter count is about 10B, then quantized to ternary {-1, 0, 1}. Inference is meant for Microsoft BitNet.cpp kernels (AVX2 / AVX-512), not for stock llama.cpp.

Use this checkpoint as a small, low-RAM chat model. Domain adaptation (your data, your facts) is a separate paid job — write to grabko@cmsmanhattan.com.

Fine-tune with LoRa

• Cheap fine-tune with LoRa for home or small companies
• Easy LoRa is best way to adapt for you domain

Service option

• Welcome to retrain on your domain with QAT/SFT and pack to Microsoft Bitnet kernels on llama fork on bitnet.ccp
• So BitNet.cpp much faster than llama.cpp
• More options with pre-train to raise more quality and more level on quantization for advances bitnet kernels

What runs where

FP16 source weights are not published. They are the seed for later quants.

Quick start

Docker (linux/amd64):

docker run -d \
  --name llama3-ternary-10b \
  -p 7869:7869 \
  --restart unless-stopped \
  cmsmanhattan/llama3-ternary-10b-tq2:latest

JiRack UI

• Once the container is running, open your browser and navigate to: http://localhost:7869 This opens the JiRack UI — a clean web interface.

• Replace the image name with the tag actually published next to this card.

• BitNet.cpp from source (Microsoft repo, not a private fork):

git clone https://github.com/microsoft/BitNet.git
cd BitNet && mkdir build && cd build && cmake .. && cmake --build . -j
./bitnet --model /path/to/Llama3Ternary10b-TQ2_0.gguf -p "Capital of France?"

./llama-completion  -m /mnt/nfs_share/JiRackTernary10_TQ_2/Llama3Ternary10b-TQ2_0.gguf  -c 8192 -t 24 -n 256  -no-cnv -e --no-display-prompt  --temp 0 --repeat-penalty 1.0   -p "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"  2>/dev/null

# Llama3 Ternary 10b (TQ2_0) — llama-server + web UI / OpenAI API proxy
#
# Put this file in /mnt/nfs_share/Llama3Ternary10b/Docker next to the Dockerfile,
# proxy.py, start.sh, web/ and Llama3Ternary10b-TQ2_0.gguf, then:
#   docker rm -f jirack10b          # once: remove the container started by run.sh
#   docker compose up -d --build    # build + start
#   docker compose logs -f          # wait for "server adds 1 BOS -> ... total 2"
#   docker compose down             # stop and remove
# Settings can be overridden in a .env file next to this one (see .env.example).

services:
  Llama3Ternary10b:
    image: cmsmanhattan/llama3-ternary-10b-tq2:latest
    container_name: Llama3Ternary10b
    restart: unless-stopped
    ports:
      - "${HOST_PORT:-7869}:7869"               # web UI + /v1/chat/completions
    environment:
      MODEL_NAME: ${MODEL_NAME:-jirack-10b-ternary}
      THREADS: ${THREADS:-24}
      THREADS_BATCH: ${THREADS_BATCH:-24}
      CTX_SIZE: ${CTX_SIZE:-4096}
      PARALLEL: ${PARALLEL:-1}
      # generation: greedy + no repetition penalty = verified best for this model
      TEMP: ${TEMP:-0}
      TOP_P: ${TOP_P:-0.9}
      REPEAT_PENALTY: ${REPEAT_PENALTY:-1.0}
      REPEAT_LAST_N: ${REPEAT_LAST_N:-64}
      MAX_TOKENS: ${MAX_TOKENS:-512}
      FORCE_SAMPLING: ${FORCE_SAMPLING:-1}     # 1 = ignore temperature etc. sent by clients
      HISTORY_TURNS: ${HISTORY_TURNS:-0}       # 0 = single-turn, like training
      SYSTEM_PROMPT_FILE: ${SYSTEM_PROMPT_FILE:-}   # e.g. /app/system_prompt.txt
    # Optional: edit the system prompt without rebuilding the image
    #volumes:
    #  - ./system_prompt.txt:/app/system_prompt.txt:ro
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://127.0.0.1:7869/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 120s
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "3"

Pin the BitNet.cpp commit you tested. A newer upstream build can break a custom GGUF.

Specs

Hidden size, intermediate size, and layer count belong in config.json next to the weights. Do not duplicate stale numbers from the 1.5B card here.

Training

Post-quantization QAT / SFT on instruction data (OpenAssistant, then a small SlimOrca slice, stopped early when factual recall slipped). A later mix of SlimOrca with a fact-QA set (GammaCorpus Fact QA and/or Wikipedia QA) is the planned fix for capital / geography drift. This card should be updated with the checkpoint hash after that mix.

Do not treat chat samples below as a benchmark. Replace them with outputs from the checkpoint you actually upload.

License

Weights and any Llama-derived files: Llama 3 Community License.

• Name of this model starts with Llama.
• “Built with Llama” is shown on this page.
• Commercial use is allowed under that license. The 700M monthly-active-user clause does not apply to a normal product.
• You may not relicense the weights under a proprietary CMS Manhattan license.

CMS Manhattan JiRack License, if present, covers only non-Llama materials in this repo (Docker packaging, UI, support). It does not override Meta’s license on the weights.

Acceptable Use Policy: https://www.llama.com/llama3/use-policy/

Commercial work

Ternary QAT on your dataset, replay mix to limit catastrophic forgetting, BitNet.cpp I2_S / AVX2 / AVX-512 packing, and domain SFT are paid. Weights in this repo are the public checkpoint.

Contact: grabko@cmsmanhattan.com

Not in this repo

Training scripts, lambda-quant recipes, and the FFN-expansion procedure are not part of the public Llama release. The Qwen ternary pipeline published separately is Apache-2.0 and is not a license to ignore the Llama terms on these weights.