Pink Elephant 48B MoE — Upcycled Base

The sparse foundation behind the Pink Elephant 48B family.

This repository hosts the upcycled Mixture-of-Experts (MoE) base from which the production flagship pinkelephant-llm-48b was trained. It is intended for research and as a merging foundation. For inference, use the merged production model instead — it is ready to run out of the box.

How the MoE base was created

The base was created by upcycling the dense 14B foundation of pink-elephant-llm-14b: its feedforward weights were mathematically split and replicated into eight expert networks per layer, initialized to reproduce the parent model's behavior exactly.

Model Card (verified facts)

Parameter and expert-structure counts are verified directly from the model.safetensors header (640 expert*.weight tensors across 40 layers × 8 experts).

Charts

Note on the router: the router weights in this base are initialized, not trained. A fully trained router is included with the step-900, step-1000, and final checkpoints of pinkelephant-llm-48b-adapters (as gate.safetensors), and is already fused into the merged pinkelephant-llm-48b.

Usage

Load the base for research or as a merging foundation:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "pinkelephantlimited/pinkelephant-llm-48b-moe-base",
    torch_dtype="auto", device_map="auto", trust_remote_code=True,
)

Attach the trained adapters on top:

from peft import PeftModel

model = PeftModel.from_pretrained(
    model, "pinkelephantlimited/pinkelephant-llm-48b-adapters",
    subfolder="adapters/final",
)

Or load the ready-to-run merged model directly:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "pinkelephantlimited/pinkelephant-llm-48b",
    torch_dtype="auto", device_map="auto", trust_remote_code=True,
)

Release Family

Documentation

• Release report: RELEASE_REPORT.pdf

Pink Elephant Limited — publishing large, sovereign language models.