rwkv7_quail_1p5b_sft

Model introduction

This is an official BlinkDL release of RWKV-7 Goose in Hugging Face Transformers format. RWKV-7 is an attention-free recurrent architecture with a constant-size recurrent state and constant inference work per generated token. Training remains parallelizable.

This checkpoint is a base model pretrained with web, code, synthetic, instruction, chat, and reasoning data. It is suitable for evaluation, post-training, and fine-tuning; the included chat template is a prompt interface, not a claim that the checkpoint is a safety-aligned assistant.

The Transformers integration, conversion, release packaging, linear-time RWKV tokenizer, and optional TileLang inference implementation are distributed with this release.

Highlights

• Constant recurrent state: memory does not grow like an attention KV cache.
• Bundled Transformers integration: auditable remote configuration and modeling modules provide generation, recurrent cache continuation, training, and LoRA workflows on Transformers 5.15+.
• Exact linear-time tokenizer: the bundled RWKV trie reads the self-contained tokenizer.json generated from the canonical RWKV World byte vocabulary.
• Chat-ready: chat_template.jinja supports text chat and function calls, keeps reasoning in history, and marks assistant turns for assistant-only loss.
• Optional optimized runtime: the isolated inference/ bundle provides PyTorch fallback and TileLang acceleration without changing the standard model root.

Model overview

Transformers quickstart

The repository includes configuration_rwkv7.py, modeling_rwkv7.py, and the exact linear-time tokenization_rwkv7.py. The model modules are adapted from the Transformers RWKV-7 integration at commit 4ad9ed0. Review those files and pin a model-repository revision in production. Passing trust_remote_code=True selects this bundled implementation even when the local Transformers installation also provides native RWKV-7 support.

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
)

model_id = "RWKV/RWKV7-G1k-1.5B-20260930"
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)

The recurrent cache returned by the model can be passed back for incremental decoding. Use an attention_mask for padded batches.

The model defaults to RWKV-LM's fused CUDA WKV forward/backward kernel for zero-state training calls; pass use_cache=False during training so each call starts from a fresh state. On its first compatible CUDA forward, PyTorch builds the bundled extension; the kernel supports 64-wide heads and FP32 WKV state, accepts BF16/FP16/FP32 activations, and pads independent sequences to its 16-token checkpoint interval. Packed sequences are run with a state reset at each boundary. Stateful cache calls and unsupported devices or state dtypes use the chunk-parallel PyTorch path. Set model.config.wkv_implementation = "chunked" to select that path explicitly, or "eager" to reproduce the portable token-order reference. The CUDA kernel and chunk-parallel path use different floating-point operation orders, so small numerical differences are expected.

Chat quickstart

import re

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


THINK_RE = re.compile(r"\A<think>?\s*(.*?)\s*</think>?", re.DOTALL)


def assistant_content(completion, thinking, *, close_incomplete=False):
    reply = "<think>" + completion
    thinking_block = THINK_RE.match(reply)
    if thinking:
        if thinking_block is not None or not close_incomplete:
            return reply.strip()
        return f"{reply.rstrip()}\n</think>".strip()
    return "" if thinking_block is None else reply[thinking_block.end():].strip()

model_id = "RWKV/RWKV7-G1k-1.5B-20260930"
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda")

messages = [{"role": "user", "content": "Explain why RWKV uses constant state."}]
thinking = False
max_new_tokens = 256
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=max_new_tokens,
    do_sample=True,
    temperature=1.0,
    top_p=0.5,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
completion = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)
reached_token_limit = output.shape[1] - inputs["input_ids"].shape[1] >= max_new_tokens
print(
    assistant_content(
        completion,
        thinking,
        close_incomplete=reached_token_limit,
    )
)

The template supports text system, user, assistant, and tool messages, plus assistant tool calls. It emits the last system message and optional tool schemas before the dialogue, and writes tool results as User: Function output:. Assistant reasoning_content is retained for all history turns by default; pass keep_history_reasoning=False to apply_chat_template to omit reasoning from earlier assistant turns.

Generation starts with Assistant: <think>. Set thinking=True to print the reasoning block along with the final answer; by default the example returns only the text after </think>. If generation reaches its token limit before closing the reasoning block, the display helper closes it. Generation stops on the tokenizer's EOS token.

Strip trailing spaces from user input. The official RWKV prompt guide is available in RWKV7-G1x-templates.txt.

Supervised fine-tuning

The bundled model supports TRL 1.10+ SFTTrainer, including its default chunked_nll, gradient checkpointing, assistant-only loss, BFD packing, and PEFT LoRA. Packing boundaries carried as reset position_ids are converted into RWKV recurrent-state boundaries. Do not use the boundary-destroying wrapped packing strategy.

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

# Reuse `model` and `tokenizer` loaded in the Transformers quickstart above.
dataset = load_dataset("trl-lib/Capybara", split="train")
trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=dataset,
    args=SFTConfig(
        output_dir="rwkv7-sft",
        max_length=2048,
        packing=True,
        packing_strategy="bfd",
        assistant_only_loss=True,
        use_cache=False,
        gradient_checkpointing=True,
    ),
    peft_config=LoraConfig(
        task_type="CAUSAL_LM",
        r=8,
        lora_alpha=16,
        target_modules=["receptance", "key", "value", "output"],
    ),
)
trainer.train()

Optimized local inference

Install the versions listed in inference/requirements.txt, then run the bundled interactive chat:

python inference/generate.py --model RWKV/RWKV7-G1k-1.5B-20260930 --backend auto --interactive

Or independent prompts separated by blank lines:

python inference/generate.py \
  --model RWKV/RWKV7-G1k-1.5B-20260930 \
  --backend auto \
  --input-file prompts.txt

--backend auto uses validated exact optimized boundaries and otherwise falls back to PyTorch. Full explicit TileLang execution can change floating-point operation order and requires checkpoint-, dtype-, shape-, and device-specific parity validation.

Tokenizer

The model root contains one self-contained tokenizer-data artifact: tokenizer.json. Textual vocab.json and rwkv_vocab_v20230424.txt files are intentionally omitted because they would duplicate the tokenizer used by Transformers. The tokenizer is loaded by the bundled Rwkv7Tokenizer trie through AutoTokenizer with trust_remote_code=True. The trie reads only tokenizer.json, preserves exact RWKV World IDs, and has bounded linear-time behavior on long repetitive inputs.

Intended use and limitations

• This is a base causal language model. Quality, instruction following, and language behavior depend on the checkpoint and downstream prompting or post-training.
• Assisted or speculative decoding that requires recurrent-cache rollback is not supported without retaining prior state snapshots.
• Optimized support depends on GPU architecture, dtype, batch, and shape. Unsupported auto configurations fall back to pure PyTorch.
• Explicit full TileLang execution can change floating-point operation order and requires checkpoint-, dtype-, shape-, and device-specific parity validation.
• No safety, bias, toxicity, factuality, or high-stakes-use evaluation is claimed by this model card.

License and provenance

The model weights use the locked profile license apache-2.0. The exported inference bundle is licensed separately under Apache-2.0. The bundled Transformers configuration and modeling modules retain their Apache-2.0 headers. See NOTICE and the source checkpoint link above for provenance.

Citation

@misc{peng2025250314456,
  title         = {RWKV-7 "Goose" with Expressive Dynamic State Evolution},
  author        = {Bo Peng and Ruichong Zhang and Daniel Goldstein and Eric Alcaide and Xingjian Du and Haowen Hou and Jiaju Lin and Jiaxing Liu and Janna Lu and William Merrill and Guangyu Song and Kaifeng Tan and Saiteja Utpala and Nathan Wilce and Johan S. Wind and Tianyi Wu and Daniel Wuttke and Christian Zhou-Zheng},
  year          = {2025},
  eprint        = {2503.14456v2},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2503.14456v2},
}