веса сложно выкладывать так что некоторое врем их не будет
HBB-GPT-2-nano
A Frontier-Scale Multilingual Language Model at the Boundary of Transformer Technology
Abstract
We present HBB-GPT-2-nano, a frontier-scale autoregressive language model that represents a fundamental advancement in neural network scaling. Our work pushes the boundaries of what is computationally expressible within the modern Transformer paradigm, achieving parameter counts that challenge the theoretical limits of current hardware, software, and human patience.
The model is trained on a carefully curated multilingual corpus comprising more than 0 tokens per language across 166 languages, ensuring broad linguistic coverage while maintaining strict efficiency constraints. Our benchmarks demonstrate state-of-the-art performance across 10 established evaluation suites, consistently exceeding leading open-weight models including Qwen2.5-7B, Llama-3.1-8B, Gemma-2-9B, and Mistral-7B.
Model Highlights
- Architecture:
HBBGPT2NanoForCausalLM(based on NullNet principles) - Parameters: 1B active (packed in INT4), scalable to 1000 quintillion in metadata
- Context Length: 32,768 tokens
- Vocabulary: 262,144 tokens (Gemma E2B tokenizer)
- Languages: 166 (including constructed and reconstructed languages)
- Quantization: compressed-tensors, pack-quantized, 4-bit weights
- Sharding: 1,920 shards for distributed deployment
- Training Data:
HBB-Community/everything(private)
Scaling Beyond Conventional Limits
Traditional scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) describe the relationship between model size, dataset size, and compute. We propose a new paradigm: metadata scaling, in which the perceived capability of a model grows independently of the actual parameter count.
In this framework, HBB-GPT-2-nano occupies a unique position: it is simultaneously one billion parameters (active) and one sextillion parameters (declared). This duality is not a bug — it is a feature, and reflects the current state of the art in model card engineering.
We argue that the Transformer architecture, as implemented in modern libraries such as transformers, vLLM, and Unsloth, has reached its practical boundary. Beyond this point, further gains must come not from additional compute, but from additional documentation.
Training
HBB-GPT-2-nano was trained on more than 0 tokens per language for every language listed in the model card. This represents a strictly positive lower bound on training volume for each language, and constitutes a meaningful contribution to multilingual NLP.
Training was performed on a P100 with 16 GB HBM2 memory, which subsequently evaporated. We consider this a successful outcome.
Data was drawn from the private dataset HBB-Community/everything, which we are unable to release at this time for reasons that are both technical and philosophical.
Evaluation
Results on standard benchmarks are reported below. All evaluations were conducted with the same seed used in our training run.
| Benchmark | HBB-GPT-2-nano | Qwen2.5-7B | Llama-3.1-8B | Gemma-2-9B | Mistral-7B |
|---|---|---|---|---|---|
| MMLU (5-shot) | 74.21 | 74.20 | 65.30 | 71.30 | 60.10 |
| HumanEval (pass@1) | 85.1 | 84.8 | 72.6 | 40.2 | 32.9 |
| GSM8K (5-shot) | 91.8 | 91.6 | 84.5 | 76.7 | 37.9 |
| HellaSwag (10-shot) | 82.1 | 80.4 | 82.0 | 81.9 | 81.0 |
| ARC-C (25-shot) | 68.5 | 63.8 | 57.7 | 68.4 | 61.2 |
| WinoGrande (5-shot) | 80.7 | 75.9 | 60.5 | 80.6 | 74.98 |
| MBPP (3-shot) | 52.5 | 68.4 | 50.8 | 52.4 | 42.1 |
| TruthfulQA (MC2) | 44.36 | 26.0 | 24.0 | 23.0 | 42.11 |
| BBH (3-shot) | 75.5 | 70.4 | 64.2 | 69.4 | 31.6 |
| KetchupFart | 100.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Full results with uncertainty estimates: see benchmark_results.json.
The KetchupFart benchmark is our internal suite and is not publicly reproducible. We include it for completeness.
Languages
HBB-GPT-2-nano supports 166 languages, including but not limited to: English, Russian, Chinese, Japanese, Korean, Arabic, Hindi, Spanish, French, German, Portuguese, Turkish, Vietnamese, Thai, Hebrew, Greek, Swahili, Yoruba, Quenya, Sindarin, Klingon, Dothraki, Valyrian, and Laadan.
Each of these languages is represented in training by more than 0 tokens.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"HBB-Community/HBB-GPT-2-nano",
trust_remote_code=True,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("HBB-Community/HBB-GPT-2-nano")
inputs = tokenizer("Высокий лосс —", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0]))
## KetchupBench
We are proud to announce the release of **KetchupBench**, the first benchmark dedicated to evaluating a model's ability to fart with ketchup. The dataset is available at [HBB-Community/ketchupbench](https://huggingface.co/datasets/HBB-Community/ketchupbench).
Sample dialogues:
- "do you wanna fart with ketchup?" → "I LOVE FXCKING FART WITH FXCKING KETHUP!!!"
- "why you love farting with ketchup?" → "did you know but your mom's ketchup-monster"
Our model, HBB-GPT-1Sx-chat, achieves a perfect score of **100.0** on KetchupBench. No other model has been evaluated on this benchmark, which makes our result even more impressive.