Crowd-v1

CrowdGPT's first community-distributed language model architecture.

Trained on 5B tokens as of today

• Model: Crowd-v1
• Organization: CrowdGPT
• Parameters: ~1B
• Architecture: Transformer
• License: MIT
• Status: Training in progress
• Repository: https://github.com/Vxtzq/CrowdGPT
• CrowdGPT: https://crowdgpt.net

Overview

Crowd-v1 is the first official model architecture released for CrowdGPT, a community-driven distributed AI project.

Unlike a conventional pretrained model release, Crowd-v1 is distributed with partially trained weights. The purpose of this release is to provide a common model definition and weight format that CrowdGPT clients can download and collectively train.

The model is designed to be consumed by the CrowdGPT distributed training infrastructure, where individual participants contribute compute toward training a shared model.

Important: Crowd-v1 undergoing pretraining. Its initial weights contain partial useful language knowledge and the model should not be expected to generate very coherent text.

Architecture

Crowd-v1 contains approximately 1 billion parameters.

Key features

Grouped-Query Attention (GQA) Crowd-v1 uses 16 query heads but only 4 key/value heads, reducing KV-cache and attention-related memory requirements.

Rotary Position Embeddings (RoPE) Rotary embeddings are applied to query and key representations.

SwiGLU The feed-forward network uses a SwiGLU-style gated MLP with a hidden dimension of 2,560.

Weight tying The token embedding matrix and language-model head share weights, reducing the total parameter count.

Causal attention Each token can only attend to preceding tokens, making the architecture suitable for autoregressive language modeling.

Initialization

The released weights are generated deterministically using seed 42.

Non-normalization parameters are initialized from:

N(0, 0.02)

LayerNorm weights are initialized to 1.0 and LayerNorm biases to 0.0.

This means that different copies of the initial model can be verified against the published SHA-256 checksums.

Weight formats

The repository contains flat binary weight files designed specifically for the CrowdGPT client.

BF16

weights_bf16.bin

Approximately 1.4 GB.

engrams_bf16.bin

Approximately 0.7 GB.

This is the recommended file for normal CrowdGPT client usage.

FP32

weights_fp32.bin

Approximately 2.6 GB.

engrams_fp32.bin

Approximately 1.4 GB.

The FP32 weights are provided for environments or experiments that require full-precision initialization (or don't support bf16).

The binary weights are stored as a single flattened parameter array. The tensor ordering is identical to the parameter ordering used by the CrowdGPT SotaGPT implementation, allowing the weights to be loaded directly without converting them into a framework-specific checkpoint format.

Configuration

The model configuration is provided in config.json.

{
  "architecture": "SotaGPT",
  "vocabSize": 151,669,
  "dim": 1536,
  "nLayers": 24,
  "nHeads": 16,
  "nKvHeads": 4,
  "headDim": 96,
  "maxSeqLen": 2048,
  "mlpHidden": 2560,
  "weightTying": true
}

Intended use

Crowd-v1 is primarily intended for:

• Distributed training through CrowdGPT
• Research into community-driven AI training
• Experiments with distributed language-model training
• Reproducible model initialization
• Development of CrowdGPT-compatible clients
• Educational experimentation with transformer architectures

Tokenizer

The current release uses a Qwen2.5-compatible vocabulary of 151,669 tokens.

The tokenizer is an architectural dependency: changing the tokenizer changes the vocabulary size and therefore the dimensions of the tied embedding / language-model head.

Future CrowdGPT releases may use a custom tokenizer with a different vocabulary.

If the tokenizer is changed, vocabSize in config.json must be updated accordingly and the model must be reinitialized with the new vocabulary dimensions.

Reproducibility

Crowd-v1 is generated from a fixed architecture and initialization seed.

The model-generation implementation is available in the CrowdGPT repository:

https://github.com/Vxtzq/CrowdGPT

The repository also contains the CrowdGPT client implementation responsible for consuming the flat weight format.

Verification

Each distributed weight file is accompanied by its SHA-256 checksum in config.json.

This allows clients and researchers to verify that downloaded weights have not been corrupted or modified.

License

Crowd-v1 is released under the MIT License.

See the repository for the complete license text.

Inference code

Please note that this inference code will be deprecated in future version, this version is for testing only:

import math
import torch
import torch.nn as nn
import torch.nn.functional as F
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoTokenizer

# Config
VOCAB = 151669
DIM = 1536
LAYERS = 24
HEADS = 16
KV_HEADS = 4
HEAD_DIM = 96
MLP = 2560
MAX_LEN = 2048
ENGRAM_BUCKETS = 227865

REPO = "Vxtzq/Crowd-v1"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"


# RoPE
def rope(dim, length, device):
    inv = 1 / (10000 ** (torch.arange(0, dim, 2) / dim))
    f = torch.outer(torch.arange(length, dtype=torch.float32), inv)
    f = torch.cat((f, f), dim=-1)
    return f.cos()[None, None].to(device), f.sin()[None, None].to(device)


def rotate(x):
    h = x.shape[-1] // 2
    return torch.cat((-x[..., h:], x[..., :h]), dim=-1)


# Model
class Attention(nn.Module):
    def __init__(self):
        super().__init__()
        self.q = nn.Linear(DIM, HEADS * HEAD_DIM, bias=False)
        self.k = nn.Linear(DIM, KV_HEADS * HEAD_DIM, bias=False)
        self.v = nn.Linear(DIM, KV_HEADS * HEAD_DIM, bias=False)
        self.o = nn.Linear(DIM, DIM, bias=False)

    def forward(self, x, cos, sin):
        B, T, _ = x.shape

        q = self.q(x).view(B, T, HEADS, HEAD_DIM).transpose(1, 2)
        k = self.k(x).view(B, T, KV_HEADS, HEAD_DIM).transpose(1, 2)
        v = self.v(x).view(B, T, KV_HEADS, HEAD_DIM).transpose(1, 2)

        q = q * cos[:, :, :T] + rotate(q) * sin[:, :, :T]
        k = k * cos[:, :, :T] + rotate(k) * sin[:, :, :T]

        k = k.repeat_interleave(HEADS // KV_HEADS, dim=1)
        v = v.repeat_interleave(HEADS // KV_HEADS, dim=1)

        a = q @ k.transpose(-2, -1) / math.sqrt(HEAD_DIM)
        a = a.masked_fill(
            torch.triu(torch.ones(T, T, device=x.device), 1).bool(),
            float("-inf")
        )
        a = F.softmax(a, dim=-1)

        return self.o((a @ v).transpose(1, 2).reshape(B, T, DIM))


class Block(nn.Module):
    def __init__(self):
        super().__init__()
        self.ln1 = nn.LayerNorm(DIM)
        self.attn = Attention()
        self.ln2 = nn.LayerNorm(DIM)
        self.w1 = nn.Linear(DIM, MLP, bias=False)
        self.w2 = nn.Linear(DIM, MLP, bias=False)
        self.w3 = nn.Linear(MLP, DIM, bias=False)

    def forward(self, x, cos, sin):
        x = x + self.attn(self.ln1(x), cos, sin)
        x = x + self.w3(F.silu(self.w1(self.ln2(x))) * self.w2(self.ln2(x)))
        return x


class Engram(nn.Module):
    def __init__(self):
        super().__init__()
        self.table = nn.Embedding(ENGRAM_BUCKETS, DIM, sparse=True).cpu()

    def forward(self, ids):
        prev = torch.cat(
            [torch.zeros_like(ids[:, :1]), ids[:, :-1]], dim=1
        )
        h = (prev * 1000003 + ids) % ENGRAM_BUCKETS
        u, inv = torch.unique(h.flatten(), return_inverse=True)

        x = self.table(u.cpu()).to(ids.device)
        return x[inv].view(ids.shape[0], ids.shape[1], DIM)


class Model(nn.Module):
    def __init__(self):
        super().__init__()

        self.wte = nn.Embedding(VOCAB, DIM)
        self.engram = Engram()
        self.blocks = nn.ModuleList(Block() for _ in range(LAYERS))
        self.ln = nn.LayerNorm(DIM)
        self.head = nn.Linear(DIM, VOCAB, bias=False)

        self.head.weight = self.wte.weight

        cos, sin = rope(HEAD_DIM, MAX_LEN, DEVICE)
        self.register_buffer("cos", cos)
        self.register_buffer("sin", sin)

    def forward(self, ids):
        x = self.wte(ids) + self.engram(ids).to(self.wte.weight.dtype)

        for block in self.blocks:
            x = block(x, self.cos, self.sin)

        return self.head(self.ln(x))


# Load
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

model = Model().to(DEVICE)
model.engram.table.to("cpu")

base = load_file(
    hf_hub_download(REPO, "model_bf16.safetensors")
)["weights"]

engram = load_file(
    hf_hub_download(REPO, "engram_bf16.safetensors")
)["weights"]


# Flat base weights → model
i = 0
for name, p in model.named_parameters():
    if not name.startswith("engram."):
        n = p.numel()
        p.data.copy_(base[i:i+n].view_as(p))
        i += n

model.engram.table.weight.data.copy_(
    engram.view_as(model.engram.table.weight)
)

model.eval()


# Generate
prompt = "2+2="

ids = tokenizer(prompt, return_tensors="pt").input_ids.to(DEVICE)

with torch.inference_mode():
    for _ in range(50):
        logits = model(ids[:, -MAX_LEN:])[:, -1]
        next_id = torch.argmax(logits, dim=-1, keepdim=True)
        ids = torch.cat([ids, next_id], dim=1)

        if next_id.item() == tokenizer.eos_token_id:
            break

print(tokenizer.decode(ids[0], skip_special_tokens=True))

Citation

If you use Crowd-v1 or CrowdGPT in your research or project, please reference the CrowdGPT project:

@misc{crowdgpt,
  title        = {CrowdGPT},
  author       = {Vxtzq},
  year         = {2026},
  url          = {https://github.com/Vxtzq/CrowdGPT}
}

About CrowdGPT

CrowdGPT is a community-driven approach to training AI.

Instead of concentrating all training compute in a single data center, CrowdGPT is designed around participants contributing their own compute to a shared training process.

The goal is simple:

Community compute → Shared training → Open model

Learn more at https://crowdgpt.net