Lego-RL-Qwen3.5-35B-A3B · OpenHands SDK · 200K

Qwen3.5-35B-A3B trained with online RL inside the unmodified OpenHands SDK harness, at 200K context, on 2,699 real repository issues whose own test suites produce the reward.

SWE-bench Verified: 64.0 → 70.4 (+6.4) — no reward model, no reference-patch similarity, no harness rewrite.

This checkpoint is the OpenHands SDK production run of Lego-RL (Faithful · Reliable · Observable), the RL component of the LegoX series. The agent solves a real issue in a real repository inside a fresh sandbox, the task's own verifier suite decides {0, 1}, and the trajectory the harness actually produced — token ids, masks, log-probs and MoE expert routes captured inside the serving path — becomes the gradient step.

Why harness-native training

The scaffold is part of the environment, not the policy. The same weights score very differently depending on which harness runs them, so training under a rewritten control flow optimizes for a deployment you never ship:

SWE-bench Verified (%), one shared protocol: temperature 0.7, 200 turns, 200K context.

Each Lego-RL column is a separate run trained in that harness from the same starting checkpoint, the same 2,699 tasks and the same 3 epochs. This repository is the OpenHands SDK run (70.4). The Claude Code and OpenCode runs are reported in the paper; their checkpoints are being released into the Lego-RL collection. Across the three harnesses RL adds +6.4 / +5.8 / +9.4 — more than the entire Qwen3.5 → Qwen3.6 base-model generation (+3.4 / +1.0 / +3.4).

Quick start

1. Serve with vLLM

vllm serve Lego-X/qwen3_5_35b_a3b_ohsdk_200k_rl \
    --served-model-name vllm_model \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.9 \
    --enable-chunked-prefill --enable-prefix-caching \
    --dtype bfloat16 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --host 0.0.0.0 --port 8000

[!IMPORTANT] --tool-call-parser qwen3_coder is not optional. The model was rolled out and trained with this parser; serving it behind hermes (or any other parser) silently degrades tool-call formatting — this exact misconfiguration cost us a full cluster-day during development.

2. Drive it with OpenHands SDK

The model is a coding agent policy: it expects a harness that gives it a repository, a shell and file-edit tools, and it expects room to run. Point OpenHands SDK at the vLLM endpoint as an OpenAI-compatible provider:

from openhands.sdk import LLM, Agent, Conversation   # openhands-sdk >= 1.14
from pydantic import SecretStr

llm = LLM(
    model="openai/vllm_model",
    base_url="http://localhost:8000/v1",
    api_key=SecretStr("dummy"),
    temperature=0.7,
    max_input_tokens=200_000,
    max_output_tokens=32_768,
)
agent = Agent(llm=llm)
conversation = Conversation(agent=agent, workspace="/path/to/repo")
conversation.send_message("Fix the failing test in tests/test_units.py")
conversation.run()

Consult your openhands-sdk version's own docs for the exact constructor signature — the training runs used the openhands-sdk runtime image c-oh-sdk-1.33.0, and evaluation used c-oh-sdk-1.14.0.

Run it with 200 turns and 200K context. The RL policy learned to spend turns: mean turns per trajectory grew from 46.6 to 83.1. Capping turns at the usual 50–100 truncates the second half of its trajectories systematically and costs most of the gain.

3. Reproduce the evaluation

git clone https://github.com/LegoX/Lego-RL.git && cd Lego-RL
bash scripts/setup_env.sh
cp scripts/eval/_template.env scripts/eval/configs/my_eval.env   # set MODEL_PATH, DATASET_PATH, kubeconfig
bash scripts/eval/eval.sh scripts/eval/configs/my_eval.env

Sandboxed execution and verifier rewards come from Harbor; see the evaluation docs.

Training

Three properties made this run trainable rather than merely runnable:

• Faithful capture. An in-process proxy at the serving boundary records token ids, masks, log-probs and expert routes at generation time — nothing is re-tokenized from a stored transcript, which matters because the harness rewrites its own history (compaction, re-serialized tool arguments, injected reminders). Replaying rollout-time MoE routing (R3) lifted rollout↔training correlation from 0.9946 to 0.9993; across all three production runs the median per-step correlation stayed ≥ 0.998.
• Reliable execution. Reward-hacking paths are closed inside the environment, per phase: no network and no fix-bearing git history during the agent phase, both restored for grading; test files are withheld and any edit to them rolled back. Trajectories that died of infrastructure faults (2.4% here) are masked out of the loss rather than scored as failures.
• Difficulty-filtered task pool. 36,884 OpenSWE candidates → 2,699 (12.4% of the executable pool), keeping only tasks solved 1–3 times out of 4 by a reference rollout model. An unfiltered pool never improved at all: 72.7% of its tasks were never solved and 13.4% always were, so under six tasks in a hundred produced any in-group reward variance.

The training set is disjoint from SWE-bench Verified at both the repository and the instance level.

What RL actually changed

420 trajectories from the first step vs. 420 from the last, in the OpenHands SDK run:

The honest reading: the model became more reliable on tasks it could already sometimes solve, more than it became capable of new ones. pass⁸ moved 11.1 points while pass@8 moved 4.7. Mid-trajectory error recovery barely moved (63.9% → 66.8%) — a terminal binary reward gives no independent signal for recovering from a failed command.

Intended use and limitations

Use it as an agent policy, not as a chat model: it was optimized inside a harness that hands it a repository, a shell, and file-editing tools, and its behavior (long inspect-edit-verify loops, heavy test invocation) only pays off in that setting.

• Harness. Trained in OpenHands SDK. It also runs under Claude Code / OpenCode / any OpenAI- or Anthropic-compatible harness, but the paper trains a separate policy per harness, and each does best in the one it was trained in.
• Budget. 200K context and 200 turns; response lengths roughly doubled during training (43.5K → 90.9K tokens per trajectory). Short budgets truncate it.
• Domain. Python-heavy repository issue-resolution, in the SWE-bench/OpenSWE distribution. No claim outside it.
• Inherited base behavior. Safety, multilingual and general-knowledge behavior come from Qwen3.5-35B-A3B and were not targeted by this RL. The architecture retains the base model's vision tower; training and evaluation were text-only.
• Sandbox it. The policy writes files and executes shell commands on purpose. Run it in a container.

Related work in the LegoX series

Acknowledgement

Built on verl (trainer + rollout) and Harbor (sandboxed execution + verifier reward), with OpenHands as the harness.

Citation

@misc{du2026legorlharnessnativereinforcementlearning,
  title={LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents},
  author={Yiming Du and Yuxin Jiang and Tao Yuan and Jianbo Dai and Shaowei Wang and Jierun Chen and Chaofan Tao and Xianzhi Yu and Lifeng Shang and Kam-Fai Wong and Xiaohui Li and Haoli Bai},
  year={2026},
  eprint={2608.17393},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.17393},
}