RWKV-7 G1G 13.3B — Hugging Face reference model
This repository is a self-contained Hugging Face conversion of
rwkv7-g1g-13.3b-20260523-ctx8192.pth. Release v0.9.0 contains the complete readable,
pure-PyTorch reference implementation next to the weights. Normal inference
does not require rwkv7-hf, FLA, a custom CUDA wheel, JIT, or CUDA Graphs.
Install and use
python -m pip install torch transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "wangyue114514/rwkv7-g1g-13.3b-hf"
revision = "v0.9.0"
tokenizer = AutoTokenizer.from_pretrained(
model_id, revision=revision, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
model_id, revision=revision, trust_remote_code=True, torch_dtype="auto"
).eval()
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
inputs = tokenizer("The future of recurrent language models is", return_tensors="pt")
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.inference_mode():
tokens = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
trust_remote_code=True loads the checked-in files
configuration_rwkv7.py, cache_rwkv7.py, ops_rwkv7.py,
modeling_rwkv7.py, and tokenization_rwkv7.py. The recurrent state has the
canonical [batch, heads, key, value] layout. Cache lifecycle, loss, padding,
generation, gradients, and the layer structure are visible in those files.
Fine-tuning and evaluation
The model follows the standard AutoModelForCausalLM contract and supports
Trainer, Accelerate, PEFT LoRA, TRL SFT/DPO/GRPO, and lm_eval through the
ordinary Transformers path. Set model.config.use_cache = False during
training. Reproducible examples and evaluation manifests are in
123123213weqw/hf-adapter.
Reference versus optimized execution
The v0.9.0 model code intentionally favors readability and compatibility.
It is the correctness/reference path, not a peak-throughput benchmark backend.
Optional CUDA Graph and Triton work is maintained separately on
perf/optional-native-backend-v0.10; older CUDA/JIT/quantization and KV-v2
experiments remain on perf/native-kernels-v0.8. Neither branch changes this
model repository's public HF contract.
Model and provenance
- Architecture: RWKV-7 recurrent causal language model
- Checkpoint family: G1G
- Parameters: 13,269,245,952
- Layers: 61
- Hidden size: 4096
- Vocabulary size: 65536
- Stored weight dtype: float16
- Original checkpoint SHA256:
0aa686d3ca4bb486e83e3071f4798a210f960e1fc1f5042e6cb418cc463814d6 - Reference source revision:
b8438cab0dc7d11238942efca4c05135d53fcaf8 - Release tag:
v0.9.0
conversion_manifest.json retains the immutable original conversion and
weight provenance. v0.9.0 changes the checked-in runtime code and config only;
the safetensors bytes are not rewritten or re-uploaded.
License
Apache-2.0. See LICENSE.