Qwen2.5-7B-Instruct — Converge Collective v1, corrected build

[!NOTE] This is the corrected build of v1.

What was wrong. The merge that produced v1 had an engineering defect. It weakened the base model's own weights instead of keeping them at full strength. Measured on TIGER-Lab MMLU-Pro (12,032 questions, paired with Qwen2.5-7B-Instruct in one session), v1 scored about 4 points below its base model, and its step-by-step answers were markedly shorter.

What this is. The cause was found and fixed in the merge. This model is v1's exact recipe and donor set, rebuilt with the defect removed: the merge no longer attenuates the base model's weights. Nothing else in the recipe changed. On the same MMLU-Pro run the corrected build scores at its base model's level, with its answer length restored. The before-and-after comparison across the complete board is on this card.

Status. The full 11-suite board is complete, against v1 as published and the base model in one session per instrument, plus the MMLU-Pro read and four diagnostic suites below; every suite landed whichever way it fell. v1 stays published, defect included, on its own card; this release does not replace it.

Optitransfer · converge: open models built to place at the top of their class on public leaderboards, from the work the open-source community has already published. Measured against the base model, published with the evidence. About the programme

What it is. A 7.6B-parameter, training-free, weight-space merge built on Qwen2.5-7B-Instruct, rebuilt from v1's recipe with the defect removed. No gradient training and no training data were used. It loads and prompts exactly like Qwen2.5-7B-Instruct: same architecture, tokenizer, chat template and configuration files.

• Headline, the defect's discovery instrument: on TIGER-Lab MMLU-Pro (12,032 questions, all four models in one session), v1 as published scores 52.75 vs 56.84 for Qwen2.5-7B-Instruct (−4.09, paired 95% CI [−4.85, −3.32], exact McNemar p ≈ 6×10⁻²⁶, a regression). The corrected build scores 57.01: +4.26 over v1 as published (CI [+3.50, +5.05], p ≈ 2×10⁻²⁷) and no measurable difference against the base model (+0.17, CI [−0.43, +0.77], p = 0.589).
• Answer length restored. On that same MMLU-Pro run the base model generated 4,992,764 tokens, v1 as published 2,109,023, and the corrected build 4,439,995 — the step-by-step answers are back near base length.
• The board, complete (one session per instrument, identical items): against v1 as published, 4 improvements (MMLU (HELM replay, macro), MATH-500, MATH Level 5, MMLU-Pro (lm-eval)), 1 regression (IFEval) and no measurable difference on the other 7; against the base model, no measurable difference on every one of the 12 board rows.
• No improvement over the base model is claimed anywhere on this card. The correction restores v1 to its base model's level; every in-session paired comparison against the base model below reads "no measurable difference". The one cross-session read, ZeroEval MATH-L5, is a regression (the diagnostic table).
• Weights: model.safetensors, sha256 (LFS oid) 10e15a9090125c3fd256c8fb9da96a8b553ec8d14474b5c5c89f21b52adb5017. The file has had this one hash since it was first uploaded. Every result on this card was run on exactly these bytes.
• Evidence: the evidence/ folder of this repository holds the per-item verdicts, the raw outputs of all three models, the claims with their statistics, and verify.py, which recomputes the corrected build's 34 paired comparisons on this card, against v1 as published and against the base model (pip install numpy && python evidence/verify.py; 34 of 34 reproduce). The same per-item files give v1 as published against the base model; v2's context row is not part of this evidence. In the evidence files this build is named Converge-7B-v1 and v1 as published converge-collective-v1.

The goal of the project

Optitransfer's converge programme aims for top placements in each model class across the public leaderboards, starting with 7B. It brings the compatible fine-tunes the open-weights community has published into a single model per class. Every release is measured against its base model on identical items and published with its regressions and per-item evidence. The construction method is proprietary; the evaluation is public. See the programme page.

Lineage

What the defect was, and what changed

• The defect. The merge that produced v1 applied an engineering error that attenuated the base model's own weights instead of leaving them at full strength. Its measured consequences: on MMLU-Pro, 4.09 points below the base model (CI [−4.85, −3.32]) with 58% fewer generated tokens per run; on MMLU, 0.86 points below (paired, CI [−1.37, −0.35], p 0.001 — the board table below); and, on v1's own published evaluation, GSM8K gains that read as answer format only.
• The correction. The cause was found and fixed in the merge: the base model's weights are kept at full strength everywhere the defect touched them. The donor set, the recipe's strength settings and every configuration file are unchanged from v1; only the defective attenuation is removed. The corrected build was assembled from the same public source models on 2026-10-02.
• How the correction is verified. Not by construction but by measurement: v1 as published and the corrected build are re-measured side by side, on identical items, in one session per instrument, with the same paired statistics as every converge release. The tables below are that measurement. Where the corrected build meets its base model the pair reads "no measurable difference" — that is the intended result; the correction is not expected to beat the base model.

Results: before fix and after fix (paired, identical items)

• Before fix = v1 as published (weights sha256 3aa6e49a…, revision unchanged since publication). After fix = this release. Both were measured side by side in every table below, on identical items, except that no benchmark other than GSM8K had ever been run on v1's published weights — every "before fix" row here re-measures v1 inside the same session as the corrected build.
• Reading rule (the same as every converge board): a difference is an improvement or a regression only if the paired-bootstrap 95% CI excludes 0 and the exact McNemar p < 0.05. Anything else is "no measurable difference". The paired test and the rule were fixed in the evaluation code before the runs.
• only-after / only-before are the discordant counts: items only the corrected build got right, and items only v1 as published got right.

The board instrument (11 of 11 suites complete)

One session (2026-10-03), one build, all four models on identical items. Scores are percent.

Against the base model, in the same sessions, on every board suite:

No measurable difference on any of them. v1 as published, on the same items, reads GSM8K 0.00 (66 / 66, dead level) and MMLU −0.86 (CI [−1.37, −0.35], p 0.001, a regression): the defect's signature on this board, and what the correction removes.

Δ in the table is the paired item-level difference the statistics are computed on; the macro and micro rows share it because the paired test is computed per item, while the two scores weight subjects differently.

AIME's score per problem averages 8 samples, so the item-level McNemar test does not apply; its reading rests on the paired interval, which includes 0 against both models.

The defect's discovery instrument: MMLU-Pro (TIGER-Lab, 12,032 questions)

One session (2026-10-02), one build, all four models on identical items, CoT prompting with the official scorer.

Discordant counts: corrected vs v1 as published 1,383 / 870; v1 as published vs base 850 / 1,342; corrected vs base 696 / 675.

Answer length on the same run: generated tokens across the 12,032 answers — base 4,992,764; v1 as published 2,109,023; corrected build 4,439,995; v2 2,466,037. The shortness that came with the defect is gone.

Diagnostic reads (ZeroEval-harness suites, one session, before vs after)

These four suites ran as first diagnostic reads of the correction, v1 as published and the corrected build side by side (the base model's score is shown for context; it was not paired in this session). Same reading rule.

Base-model context scores: CRUX 54.62, MATH-L5 52.43, MMLU-Redux 75.27, Zebra-Grid 11.70.

Paired against the base model's run of the same suites from an earlier session on the same build (two sessions, so read with that caveat): MATH-L5 −3.33 (CI [−6.38, −0.42], p 0.037, 49 / 73), a regression; no measurable difference on CRUX −1.50 (CI [−4.00, +0.88]), MMLU-Redux +0.43 (CI [−0.65, +1.51]) and Zebra-Grid −0.40 (CI [−1.90, +1.20]).

Honest framing

• The correction restores, it does not lift. Every in-session paired comparison against the base model on this card reads "no measurable difference"; none reads "improvement". The cross-session ZeroEval MATH-L5 read is a regression. What was broken was v1's loss against its base model, and that loss is what the tables show repaired.
• What is tested. The tested claims are paired differences on identical items: corrected vs v1 as published (the fix's effect) and corrected vs the base model (restoration), both within single sessions.
• Coverage. All eleven board suites are complete, plus the MMLU-Pro read and four diagnostic suites. Against v1 as published the correction reads 4 improvements and 1 regression (IFEval −3.88); against the base model, no measurable difference on any board suite.
• Self-reported results. These are first-party evaluations by the model's developers. The per-item verdicts behind every row are in this repository's evidence/ folder.
• Contamination. The source models' training data is third-party and cannot be certified free of benchmark contamination.

Limitations

• MATH Level 5. Against the base model, ZeroEval's MATH-L5 reads −3.33 (a regression, across two sessions on the same build); the board's own MATH Level 5 reads −1.81 (no measurable difference).
• Not a long-context model. Configured for 32,768 tokens (max_position_embeddings; no rope scaling), and no long-context evaluation was run. tokenizer_config.json inherits model_max_length: 131072 from the base model; that value is not this model's configured context.
• Identifies as Qwen. With no system message, the inherited chat template inserts "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." The model may describe itself as Qwen.
• Default decoding samples. The repository's generation settings sample (T 0.7, top_p 0.8, top_k 20, repetition penalty 1.05). The rows on this card were run greedy with the repository's sampling defaults neutralised (repetition penalty 1.0), as every converge board run does, except AIME, which averages 8 samples per problem at temperature 0.6, top_p 0.95; other engines and decoding settings were not evaluated.
• Not evaluated for safety, bias or toxicity. Evaluated in English only. It may produce incorrect, biased or harmful content; do not use it for high-stakes decisions without independent evaluation.

Quick start

Loads and prompts exactly like Qwen2.5-7B-Instruct. The card's scores were produced with greedy decoding (AIME: 8 samples at temperature 0.6) and the repository's sampling defaults neutralised; for chat use, apply the template as below.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Optitransfer/Qwen2.5-7B-Instruct-converge-v1remerge-maskn_project"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="auto"  # use torch_dtype= on older transformers
)

messages = [{"role": "user", "content": "A train travels 180 km in 2.5 hours. What is its average speed in km/h?"}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)

# Greedy, as evaluated. Without do_sample=False the repo's generation_config samples
# (T=0.7, top_p=0.8, top_k=20, repetition_penalty=1.05).
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Evaluation setup

Two instruments, each one session, all models side by side on identical items:

• The board suites: vLLM 0.30.0, bfloat16, seed 1234, greedy (repetition penalty 1.0, the repository's sampling defaults neutralised) except AIME (8 samples per problem, temperature 0.6, top_p 0.95), one NVIDIA A100-SXM4-80GB (driver 580.82.07, CUDA 13.0). GSM8K and MMLU: lm-eval 0.4.13 driving HELM v1.13.0's published request files (1,000 and 14,042 items; generation budgets 400 and 1 token). GSM8K 5-shot and 0-shot CoT, ARC-Challenge, IFEval and MMLU-Pro: lm-eval 0.4.13. HumanEval+ and MBPP+: EvalPlus 0.3.1. MATH-500, MATH Level 5 and AIME: answer-equivalence scoring with math_verify 0.9.0. Each benchmark's dataset revision is in evidence/identity.json. Software: Python 3.13.15, torch 2.13.0, transformers 5.17.0. Qwen2.5-7B-Instruct at revision a09a35458c702b33eeacc393d103063234e8bc28 and v2 at its published revision were measured in the same session for context. Run date 2026-10-03.
• MMLU-Pro and the diagnostic suites: the same engine build (vLLM 0.30.0, bfloat16, greedy, repetition penalty 1.0), TIGER-Lab MMLU-Pro at revision 9917624 with CoT prompting (budget 2,048 tokens, official scorer), and the ZeroEval-harness suites CRUX, MATH-L5, MMLU-Redux and Zebra-Grid at their published protocols. Run dates: MMLU-Pro 2026-10-02; the diagnostic suites 2026-10-03, the base model's runs of them 2026-10-02.
• Models: this release (model.safetensors sha256 10e15a9090125c3fd256c8fb9da96a8b553ec8d14474b5c5c89f21b52adb5017) and v1 as published (3aa6e49a7fc4b37f42b184494183a090ef2f9818418022e38be5e5ff64507db0). Before every run the served weight bytes of both were checked against these sha256.
• Statistics: paired bootstrap, 10,000 resamples (multinomial over the discordant cells), 95% percentile CI; exact two-sided McNemar test on the same items. The reading rule was fixed in the evaluation code before the runs.
• Checks: every suite completed with full item coverage, identical item sets for both builds and a passing decoding audit.

Model details

• dtype. The stored tensors are bfloat16, the torch_dtype that config.json declares (unchanged from the base model), and every reported result loaded them in bfloat16. Float16 inference was not evaluated.
• File hashes (sha256): config.json 7463bb0ea78315365e6c6b74de4e73bbcc8359dfb0c5a737584e077d42c0b03c; generation_config.json 3a8f9087e486054c8a4a08dae2e5a3ba62e23da212b5b8c08bc42cb983c3459f; tokenizer.json c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539; tokenizer_config.json 5b5d4f65d0acd3b2d56a35b56d374a36cbc1c8fa5cf3b3febbbfabf22f359583.
• Check a download against the model.safetensors sha256 above. A card-only commit changes the revision hash, not the weights.

License

This model descends from facebook/opt-2.7b. That model is distributed under the OPT-175B License Agreement, which limits use to non-commercial research. This model is therefore made available for non-commercial research use only, subject to three sets of terms:

• the OPT-175B License Agreement;
• the Apache-2.0 licenses of the Qwen, Mistral, SmolLM2, Granite and Pythia sources;
• the MIT licenses of microsoft/phi-2 and microsoft/Phi-3-mini-4k-instruct.

No rights are granted beyond those licenses. OPT-175B is licensed under the OPT-175B license, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

model.safetensors is modified relative to every source model. config.json, generation_config.json, tokenizer.json, tokenizer_config.json, vocab.json and merges.txt are Qwen2.5-7B-Instruct's files, unmodified.

Not affiliated with or endorsed by Alibaba Cloud or the Qwen team. "Qwen" is used only to identify the model this one is derived from.