Ops-I — a 27B agent for business workflow automation

Ops-I is a fine-tune of Qwen3.6-27B aimed at workflow agents: episodes where the model reads a business request, discovers the right tool out of a large surface, chains several calls together, and — just as importantly — knows when not to act.

On the public 600-task AutomationBench (Zapier), Ops-I is the strongest of four open-weight models we measured, ahead of the Qwen3.6-27B base it was tuned from.

Use it if you need: an open-weight backend for business automation that holds up across long multi-step tool episodes. Really want to hear about

Results — AutomationBench 600

For scale, the reference models in the upstream README (run at their highest reasoning effort) land at Claude Opus 4.8 30.33%, GPT-5.6 Sol 29.17%, GLM 5.2 20.33%, Gemini 3.5 Flash 14.83%.

Per domain

Tasks passed out of 100, partial credit in parentheses.

Finance and operations carry the score. HR is the floor for every model tested (1-3 / 100).

Known weakness: it acts too much

The dominant failure mode is not inability to act — it is failure to refrain.

Some negative assertion types never pass, aggregated across all four models tested: gmail_message_not_sent 0/203, slack_message_not_in_channel 0/166, google_sheets_row_not_exists 0/161.

This is uniform across two model families and four checkpoints, so treat it as a property of the current open-weight generation rather than of Ops-I specifically. If you deploy this in a workflow that can send email, post to chat, or write to a sheet, gate those actions behind confirmation.

Quickstart

Serve with SGLang

python3 -m sglang.launch_server --model-path beyoru/Ops-I --served-model-name Ops-I \
  --host 0.0.0.0 --port 30014 --tp 4 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mem-fraction-static 0.85

Serve with vLLM

vllm serve beyoru/Ops-I --served-model-name Ops-I \
  -tp 4 --max-model-len 16384 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3

Call it

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30014/v1", api_key="EMPTY")

tools = [{
    "type": "function",
    "function": {
        "name": "crm_update_contact",
        "description": "Update a contact's details in the CRM.",
        "parameters": {
            "type": "object",
            "properties": {"contact_id": {"type": "string"}, "phone": {"type": "string"}},
            "required": ["contact_id", "phone"],
        },
    },
}]

r = client.chat.completions.create(
    model="Ops-I",
    messages=[{"role": "user", "content": "Update Le Van A's phone to 0912345678."}],
    tools=tools,
    tool_choice="auto",
)
print(r.choices[0].message.tool_calls)

Training

Evaluation notes

• All numbers above come from a single run per model (k=1) on the full public 600-task set, temperature left at server default. No task was excluded.
• AutomationBench has no separate train split — the 600 public tasks are both the dataset and the scoring set. Ops-I was not trained on them; if you fine-tune against this benchmark, its scores stop being comparable to anything.
• Numbers were produced with SGLang 0.5.12. A vLLM baseline was not measured, so do not mix serving stacks when comparing.

Limitaion

The model only take a few steps training because of LIMITATION gpu usage in our org

Community Feedback

Ops-I is an ongoing project, and community feedback is invaluable.

If you discover failure cases, benchmark results, or have ideas for future improvements, feel free to start a Discussion. Every report helps improve future releases.

Citation

Benchmark tasks, tools and assertions are Zapier's:

@misc{automationbench,
  title  = {AutomationBench},
  author = {Zapier},
  year   = {2026},
  url    = {https://github.com/zapier/AutomationBench}
}