Kairo v1 — Crypto-Native Language Model
Kairo v1 is a specialized foundation language model pretrained directly on raw technical crypto discourse, protocol developer documentation, smart contract security audits, Layer-1/Layer-2 whitepapers, and DeFi automated market maker specifications.
The model is trained entirely on the public crawl dataset aggregated in real-time by the Kairo autonomous crawler swarms.
Model Summary
- Model Name: Kairo v1 (
kairo-v1) - Architecture: 12-layer Transformer Decoder with Rotary Position Embeddings (RoPE), SwiGLU non-linearities, and Grouped-Query Attention (GQA).
- Parameters: ~135M parameters
- Context Length: 4,096 tokens
- Vocabulary Size: 32,000 crypto-tokenized BPE units
- Pretraining Dataset: 4,285 verified crypto pages (14,273,228 tokens) across 86 core Web3 domains.
Dataset Composition (Live Swarm Crawl)
The pretraining corpus consists of verified technical sources indexed across 8 crypto chapters:
- Layer 1 & Layer 2 Protocols: Solana, Ethereum, Aptos, Sui, Arbitrum, Optimism specifications.
- DeFi Protocols & AMMs: Uniswap v3/v4 SDKs, Morpho, Aave, Curve, Raydium, Orca liquidity math.
- Crypto AI & DePIN Compute: Olas Stack, Bittensor subnets, Render, Akash, io.net.
- Research & Whitepapers: Ethereum Research (ethresear.ch), Bitcoin Satoshi whitepaper, MEV boost mechanics.
- Developer Standards: EIP/ERC standards (ERC-20, ERC-721, EIP-4844 blobs, ERC-4337 account abstraction).
- Smart Contract Security: Audit reports, reentrancy guards, formal verification patterns.
- DAO Governance: Governance proposals, snapshot voter rationale, tokenomic distribution schedules.
- Crypto Architecture Archives: Solana Sealevel parallel runtime, Move VM bytecode, EVM opcodes.
Quickstart / Inference
You can run Kairo v1 directly with Hugging Face transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load Kairo v1
model_id = "kairo-crypto-model/kairo-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
# Prompt the model on crypto concepts
prompt = "Explain how Solana's Proof of History prevents clock drift in validators:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=150,
temperature=0.7,
top_p=0.9,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Specifications
| Stage | Metric | Value |
|---|---|---|
| 01 | Optimizer | AdamW ($eta_1=0.9, eta_2=0.95, \epsilon=10^{-8}$) |
| 02 | Learning Rate | $3.5 \times 10^{-4}$ with cosine decay to $3.5 \times 10^{-5}$ |
| 03 | Context Window | 4,096 tokens |
| 04 | Final Loss | ~2.148 |
| 05 | Precision | FP16 / BF16 mixed precision |
License
Released under the Apache 2.0 License. Weights and dataset checkpoints are continuously updated by the Kairo Autonomous Network.
🔄 4x Daily Autonomous Retraining & Treasury Disbursement Schedule
Kairo v1 operates on an autonomous 4-stage continuous intelligence and economic loop, synchronized across four 6-hour epochs daily:
- Round 1: 00:00 UTC
- Round 2: 06:00 UTC
- Round 3: 12:00 UTC
- Round 4: 18:00 UTC
At each epoch boundary:
- Incremental Pretraining: The neural swarm collates all newly indexed and verified crypto web documents (whitepapers, audits, governance proposals).
- Model Weight Update: Updated checkpoint weights and token metrics are committed to Hugging Face (kairo-crypto-model/kairo-v1).
- Ranking-Slab Treasury Payout: The 20% creator fee treasury pool is automatically disbursed on Solana to wallets that burned 200,000 , weighted according to crawler performance ranking slabs (Tier 1: Apex 40%, Tier 2: Elite 30%, Tier 3: Core 20%, Tier 4: Swarm 10%).