Pink Elephant 48B-S

The Refined Sparse Intelligence Engine

47.7-billion-parameter Mixture-of-Experts language model · 8 experts · top-2 routing · ~30% active parameters per token · 3.2× compute advantage · native bf16

Executive Summary

Pink Elephant 48B-S is the flagship of the Pink Elephant lineage and the culmination of a disciplined, fully-public research program: a dense 14.8B core, upcycled into an 8-expert sparse Mixture-of-Experts network, refined across two complete training epochs, and merged into a single, ready-to-run engine.

The story is measured, not marketed. Across the second epoch of supervised fine-tuning on a 99,661-example code-and-mathematics curriculum, training loss fell 32% (0.0616 → 0.0419) while validation loss held steady near 0.50 — refinement, not overfitting. HumanEval 70.12% pass@1 confirms the sparse architecture preserved the parent's coding capability at roughly one-third of the per-token compute.

The pipeline now runs end-to-end through alignment and verification: base 48B-S → SFT (best adapter, step 90000) → DPO (min-loss adapter) → verified benchmarks on Blackwell. The fully-aligned model scores GSM8K 68.5% and MMLU (5-shot) 60.0% — the first public, reproducible numbers for the aligned flagship. Full per-question records ship in the verification/ folder of the DPO repo.

Every artifact in this release is public, reproducible, and MIT-licensed — weights, per-step adapters, trained router, tokenized dataset, loss curves, and instrumented benchmark records. Numbers are reported exactly as measured; a number we publish is a number you can audit.

Benchmarks (greedy, measured on NVIDIA RTX PRO 6000 Blackwell)

HumanEval confirms the upcycle preserved coding ability at a fraction of the compute. The MATH-500 number is reported exactly as measured, with per-subject detail published for audit (strength-first: prealgebra 61.0%, algebra 59.7%, number theory 37.1%, counting & probability 28.9%, geometry 26.8%, precalculus 17.9%, intermediate algebra 13.4%). Machine-readable results: evals/humaneval_result.json, evals/math500_result.json, plus every per-problem record.

Aligned Verification Benchmarks (SFT + DPO stage)

The numbers below are for the fully-aligned model — base 48B-S with the best SFT adapter (step 90000) and the min-loss DPO adapter fused in, evaluated in bf16 on an NVIDIA RTX PRO 6000 Blackwell. Greedy decoding with robust answer is / letter extraction (a naive "last-token" metric under-reports and is not used).

Verification stage: the pipeline is base → moe → train → sft → dpo → verification → release. These aligned results are the output of the verification stage. Full per-question records, the eval harness, and the adapter-conversion tooling live in the verification/ folder of [pinkelephant-llm-48b-s-dpo] (https://huggingface.co/pinkelephantlimited/pinkelephant-llm-48b-s-dpo), so every number is independently reproducible.

The Story in One Picture

Act I — The Dense Core: Pink Elephant 14B

A dense 14.8B-parameter decoder-only model, instruction-tuned on ~100,000 curated code and mathematics examples. It proved the thesis:

• HumanEval pass@1: 71.34%
• MATH-500: 63.40%

Act II — The Upcycle: Dense 14B → Sparse 48B

Rather than train 48B from scratch, Pink Elephant upcycled the 14B foundation into an 8-expert MoE network — feedforward weights split and replicated into eight complementary experts per layer, exactly preserving the parent's behavior at initialization. A sparse brain with dense-scale capacity at ~30% activation per token.

Act III — The Refinement: 48B-S

A second complete epoch (steps 1,557 → 3,114) on the same 99,661-example curriculum with a freshly restarted cosine schedule peaking at 1e-4, checkpointed every 100 steps and resumed cleanly four separate times. The final LoRA adapters and trained router were fused directly into the base weights — what you download is the finished engine: no adapters, no PEFT plumbing, just load and generate.

The Architecture: A Council of Experts

The conventional feedforward network is replaced by a cabinet of eight expert networks per layer, orchestrated by a learned router. Every token activates only 2 of 8 experts — sparse computing at its most elegant: maximum capability, minimum energy.

Verified End-to-End on NVIDIA Blackwell

The published artifact was downloaded, loaded, and exercised end-to-end on an NVIDIA RTX PRO 6000 Blackwell (102 GB VRAM) at native bf16 — no quantization, no CPU offload:

Getting Started

Load the model directly — no adapters required:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pinkelephantlimited/pinkelephant-llm-48b-s"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",   # a 102 GB-class GPU is recommended for native bf16
    trust_remote_code=True,
)

chat = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
with torch.inference_mode():
    out = model.generate(
        prompt.to(model.device),
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))

Lower-footprint inference on consumer GPUs — standard 4-bit loading with bitsandbytes:

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    device_map="auto",
    trust_remote_code=True,
)

Note: this model uses a custom architecture shipped in the repository (modeling_pink_elephant.py). Loading requires trust_remote_code=True.

Documentation

The Pink Elephant Family

Roadmap

• ✅ Alignment (done): preference optimization (DPO) + SFT fine-tuning completed; verified GSM8K 68.5% and MMLU 5-shot 60.0% on the aligned model (see verification benchmarks)
• Serving: high-throughput sparse serving and GGUF quantization for edge deployment
• Breadth: expand general-knowledge coverage beyond the math/STEM core
• Reasoning: continue closing the MATH-500 gap with longer-horizon, chain-of-thought training

Limitations

• Fine-tuned primarily on code and mathematical reasoning; the aligned model reaches GSM8K 68.5% and MMLU 5-shot 60.0%, but general-knowledge breadth still inherits from the 14B core and trails specialized frontier models
• MATH-500 under greedy decoding (38.40%) trails the 14B parent (63.40%); addressing this is an explicit roadmap item
• Long-horizon, multi-step agentic workflows may benefit from explicit tool-use training
• Verification samples (~89 GSM8K / 40 MMLU) are representative, not exhaustive; treat as aligned-stage estimates subject to sampling variance
• Outputs should be reviewed before use in regulated or safety-critical decisions

License & Commercial Use

Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.

Pink Elephant Limited — publishing large, sovereign language models.

Built by Pink Elephant Limited · 2026