Tev-27B

Tev-27B is a long-horizon terminal agent model trained from Qwen3.8-27B with reinforcement learning on Tev-Gym, using progress-potential reward shaping (PPRS). It is the main model of the paper Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards.

Highlights

• Agents' Last Exam, computing & math split (ALE-CM). Pass@5 rises from 0.17 to 0.28 (+65% over Qwen3.8-27B), matching GPT-6 Astra and Claude Opus 5 at Pass@5. Mean graded score (S@1) rises from 0.33 to 0.40.
• Zero-shot transfer to longer horizons. No Terminal-Bench task is used in training. Pass@1 rises by +56% on the Terminal-Bench 3.0 "CM" subset (0.09 → 0.14) and 2.5× on the Terminal-Bench 4.0 "CM" subset (0.04 → 0.10), ahead of GLM 5.2 and Gemini 3.7 Flash.
• Mid-horizon coding is kept and improved. On Terminal-Bench 2.1, Pass@1 reaches 0.794 (base 0.764), on par with Claude Opus 4.8 (0.789).

How Tev works

Open models struggle on long-horizon tasks for two reasons: there is almost no long-horizon training data, and terminal-only rewards are too sparse to learn from. Tev addresses both.

Tev-Gym: latent-world-first environment synthesis. Each task is built backward from a latent world, a complete, correct, hidden specification of the task domain, its workflow (a DAG of stages), and the expected state after every step. Agent-facing inputs are rendered from that world and corrupted with controlled, invertible defects that the agent must repair. Because the correct answer exists before the task, the latent world serves as an exact answer key. It grades the final submission and also every intermediate state. Tev-Gym contains 1,170 verified long-horizon environments across 7 domains (software engineering, data engineering, AI/CS research, mathematics and operations research, security and forensics, infrastructure administration, and quantum computing). A frontier agent needs a median of 12.4 minutes per rollout on these tasks, about 6× longer than on the most time-intensive prior agentic corpus.

PPRS: dense progress rewards. After every turn, the agent's workspace is scored against the latent world's known-correct intermediate states, giving a progress potential Φ(s) ∈ [0, 1]. The rollout reward is

R(τ) = 1[τ passes the strict verifier] + λ · Φ(s_T),   λ = 0.25

applied with per-turn credit (return-to-go, γ = 1) and a per-turn-index group baseline. Under terminal-only rewards, GRPO-style training throws away every group in which all rollouts fail, and on long-horizon tasks that is most groups. Φ separates those rollouts by how far they got. At 27B, terminal-only reward produced almost entirely zero-advantage groups and did not complete a single optimizer step, while PPRS trained to completion.

Model details

Note on the weights. The checkpoint contains only the text model and lm_head (12 bf16 safetensors shards). It has no vision tower and no MTP heads, although config.json is the base model's vision-language config (Qwen3_5ForConditionalGeneration). Serve it as a language-only model and do not enable MTP speculative decoding.

Evaluation

Every benchmark uses 5 independent rollouts per task. S@1 is the mean graded score, P@1 the fraction of rollouts that fully solve the task, and S@5 / P@5 are their best-of-5 counterparts.

ALE-CM: Agents' Last Exam, computing & math split (18 tasks)

On ALE-CM, the open models (base and Tev) run in the single-bash harness used in training, with 64 turns and 16,384 tokens per turn. Frontier models run in their native agent harnesses (Codex, Claude Code, Gemini CLI), the setting most favorable to them.

By category, Tev-27B's largest gain is on Math & Science, where P@5 doubles from 0.25 to 0.50.

Terminal-Bench 3.0 and 4.0, "CM" subsets (zero-shot)

The computing-and-math subsets contain 22 TB 3.0 tasks and the 18 of them that remain in TB 4.0. On the leaderboard models, subset scores track full-benchmark scores closely (Pearson r = 0.96 for TB 3.0 P@1 and 0.97 for TB 4.0 P@1).

Terminal-Bench 2.1 (89 tasks, zero-shot, Terminus-2 agent)

Frontier rows come from the official leaderboards or from our runs in each model's native harness. See the paper for per-category results, full tables, and the evaluation protocol.

Usage

Tev-27B is a terminal agent. It works best in a loop that gives it a single bash tool and executes its commands in a sandbox, the setting it was trained in.

Serve with vLLM

vllm serve gordonhu/Tev-27B \
  --served-model-name tev-27b \
  --language_model_only \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --max-model-len 131072

Call it as an agent

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

tools = [{
    "type": "function",
    "function": {
        "name": "bash",
        "description": "Execute a bash command. Each command runs in a new subshell.",
        "parameters": {
            "type": "object",
            "properties": {"command": {"type": "string", "description": "The bash command to execute."}},
            "required": ["command"],
        },
    },
}]

messages = [
    {"role": "system", "content": (
        "You are a helpful coding assistant. You have access to a bash terminal. "
        "When you are confident your solution is correct, submit by running: "
        "`echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`"
    )},
    {"role": "user", "content": "Read /task/prompt.md and complete the task described there."},
]

response = client.chat.completions.create(
    model="tev-27b", messages=messages, tools=tools,
    temperature=0.7, top_p=0.95, max_tokens=16384,
)
print(response.choices[0].message.tool_calls)
# Run each `bash` call in your sandbox, append the output as a "tool" message, and repeat
# until the model submits or the turn budget (64 in our evaluations) runs out.

Intended use and limitations

• Research use only. Tev-27B is released for non-commercial research on long-horizon agents (see License).
• Run it in a sandbox. The model executes shell commands autonomously. Run it in an isolated container with no access to credentials, production systems, or data you cannot afford to lose.
• Gap to frontier models. Tev-27B still trails the strongest frontier agents on the hardest long-horizon benchmarks (for example, 0.10 vs. 0.52 P@1 for Claude Opus 5 on TB 4.0-CM).
• Domain focus. Training covers terminal-based computing-and-math workflows in synthetic environments. Results may not transfer to GUI, web, or multimodal agent tasks, and the vision encoder of the base model is not included.
• Dual use. Like any more capable coding agent, the model can be misused. The security-flavored tasks in Tev-Gym are analysis and forensics exercises over synthetic artifacts.

License

Tev-27B is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. You may use, share, and adapt the model for non-commercial purposes only, with attribution. Commercial use is not permitted.

Tev-27B is a derivative of Qwen3.8-27B, which is licensed under the Apache License 2.0. A copy of that license is included as LICENSE-Qwen3.8-27B. The weights were modified from the original by reinforcement-learning fine-tuning.

Citation

@misc{hu2026tev,
  title  = {{Tev}: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards},
  author = {Hu, Wenbo and Pramanick, Shraman and Singh, Krishna Kumar and
            Peng, Nanyun and Chang, Kai-Wei and Lee, Yong Jae},
  year   = {2026},
  eprint = {2610.18868},
  archivePrefix = {arXiv}
}

Notice

The model weights held in this repo are a reproduced version, not official model weights trained at Adobe.