Pink Elephant 48B — The Sparse Intelligence Engine

48 billion parameters of knowledge. A quarter of the compute. One model.

Pink Elephant 48B is a production-ready Mixture-of-Experts (MoE) large language model and the flagship release from Pink Elephant Limited. Upcycled from the proven Pink Elephant 14B dense foundation and instruction-tuned across code and mathematics, it delivers the reasoning depth of a dense 48-billion-parameter network while activating only ~30% of its parameters for every token — 47.7B parameters of total capacity, ~14.7B active at inference.

This repository ships the fully merged, ready-to-run model: the trained LoRA expert adapters and the learned router are fused directly into the base weights. No adapter setup, no PEFT plumbing — just load and generate.

The central promise: every token is routed to a hand-picked council of specialist experts — dense-scale intelligence at a fraction of the compute.

The Architecture: A Council of Experts

Pink Elephant 48B replaces the single monolithic feedforward network of a conventional transformer with a cabinet of eight expert networks per layer, orchestrated by a learned router.

• 8 experts per layer — each with its own specialized weights and perspective
• Top-2 sparse routing — the router selects the two most qualified experts for every token; the other six rest
• A trained router — fine-tuned end-to-end alongside the experts, so assignments sharpen continuously with every step

Every token therefore travels through only 2 of 8 experts. This is sparse computing at its most elegant: maximum capability, minimum energy, delivered in real time.

48B of Knowledge, a Quarter of the Compute

• Capacity of a dense 48B — a vast reservoir of knowledge and reasoning, spread across specialists that never compromise on depth
• ~3.2× compute savings — with only ~30% of parameters active per token, inference costs a fraction of a dense model of the same scale
• No quality compromise — dense-scale capacity and sparse-scale speed, in a single deployable artifact

Trained to Write Code and Reason Mathematically

Pink Elephant 48B was instruction-tuned on ~100,000 curated examples spanning two of the most demanding reasoning domains:

• Software engineering — executable code generation and instruction following (self-oss-instruct, 49,661 examples)
• Mathematical reasoning — rigorous step-by-step problem solving (OpenR1 Math, 50,000 examples)

Training: 1,000 optimization steps, AdamW with a cosine learning-rate schedule (peak 1.5e-4), LoRA (r=32, α=64) applied to the expert projections, with the router trained in full precision.

An Elegant Scaling Story

Pink Elephant 48B was upcycled, not trained from scratch. The feedforward weights of the established pink-elephant-llm-14b dense foundation were mathematically split and replicated into eight complementary experts — preserving the parent model's knowledge while multiplying capacity roughly threefold. Scaling through intelligence, not brute force.

Getting Started

Load the model directly — no adapters required:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pinkelephantlimited/pinkelephant-llm-48b"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",            # or "cuda" / "cpu"
    trust_remote_code=True,
)

chat = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
prompt = tok.apply_chat_template(chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
with torch.inference_mode():
    out = model.generate(
        prompt.to(model.device),
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )
print(tok.decode(out[0][prompt.shape[1]:], skip_special_tokens=True))

Lower-footprint inference

For consumer GPUs, the model can be loaded in 4-bit using standard transformers tooling (requires bitsandbytes):

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    device_map="auto",
    trust_remote_code=True,
)

Note: this model uses a custom architecture shipped in the repository (modeling_pink_elephant.py). Loading it requires trust_remote_code=True.

The Refined Successor: 48B-S

The Pink Elephant journey continues. pinkelephant-llm-48b-s takes this flagship through a complete second epoch (steps 1,557 → 3,114), cutting training loss 32% (0.0616 → 0.0419) and merging into a single ready-to-run artifact. If you are starting fresh, use the 48B-S.

Release Family

Evaluation

Benchmark results for the family are published on their respective model cards. The 48B-S refined flagship reports 70.12% pass@1 on HumanEval and 38.40% on MATH-500 (both greedy), with per-problem records under evals/ in that repository. The 14B lineage reports 71.34% pass@1 on HumanEval and 63.40% on MATH-500. The 48B-S HumanEval figure demonstrates the same performance band as the parent at roughly a third of the per-token compute.

Verification & Test Report

The full release and verification documentation is published alongside this model:

What was tested

The published artifact was downloaded, loaded, and exercised end-to-end on an NVIDIA RTX PRO 6000 Blackwell Server Edition (102 GB VRAM, 160 GB RAM, PyTorch 2.11.0+cu130, transformers 5.14.1) at native bfloat16 — no quantization, no CPU offload:

Functional smoke test (greedy, 64 tokens)

Every result above is reproducible via the step-by-step recipe in the Verification Test Report (Chapter 11), which requires only a GPU with ≥ 102 GB VRAM and this repository.

Limitations

• Fine-tuned primarily on code and mathematical reasoning; general-knowledge coverage inherits from the 14B parent
• Long-horizon, multi-step agentic workflows may benefit from explicit tool-use training
• Outputs should be reviewed before use in regulated or safety-critical decisions

License & Commercial Use

Released under the MIT License — free for commercial and research use, modification, and redistribution, with or without attribution.

Pink Elephant Limited — publishing large, sovereign language models.