Qwen2.5-7B-Instruct — Converge Collective v1

[!WARNING] Notice (2026-10-03): a defect in this release has been identified and corrected.

What was wrong. The merge that produced v1 had an engineering defect. It weakened the base model's own weights instead of keeping them at full strength. Measured on TIGER-Lab MMLU-Pro (12,032 questions, paired with Qwen2.5-7B-Instruct in one session), v1 scores about 4 points below its base model, and its step-by-step answers are markedly shorter.

What is being done. The cause has been found and fixed in the merge. A corrected v1, built with the same recipe and the defect removed, measures at its base model's level on the same benchmark, with its answer length restored. A corrected v2, v2's exact recipe and code rebuilt on the corrected v1, has been built. Both corrected models are being benchmarked across the full board now. Their results will be published here, with per-item evidence, before either replaces this release.

Until then, the results on this card describe the release as published, defect included.

Optitransfer · converge: open models built to place at the top of their class on public leaderboards, from the work the open-source community has already published. Measured against the base model, published with the evidence. About the programme

The first release of converge. Its successor is v2.

What it is. v1 is a 7.6B-parameter, training-free, weight-space merge built on Qwen2.5-7B-Instruct. No gradient training and no training data were used. It loads and prompts exactly like Qwen2.5-7B-Instruct: same architecture, tokenizer, chat template and configuration files.

• Headline, measured on the published weights: on the full GSM8K test set (1,319 problems, lm-eval 5-shot), v1 scores 79.98 vs 76.27 for Qwen2.5-7B-Instruct under lm-eval's strict-match scorer: +3.71 points, paired 95% CI [+1.52, +5.91], exact McNemar p = 0.0015. This gain is answer format. The flexible-extract scorer, which reads the last number whatever the format, shows no measurable difference (−0.08, CI [−2.12, +1.97]), and so do two further answer-reading scorers (secondary, computed after the run).
• No regression measured, on narrow coverage: the 30-item checks on GSM8K, ARC-Challenge and HumanEval show no measurable difference. These are the only benchmarks run on the published weights; nothing is netted.
• Weights: model.safetensors, sha256 (LFS oid) 3aa6e49a7fc4b37f42b184494183a090ef2f9818418022e38be5e5ff64507db0. The file has had this one hash since it was first published. Every result on this card was run on exactly these bytes.
• Evidence: per-item verdicts for every row, both models' raw GSM8K outputs and a script that recomputes every number on this card, in the release's evidence bundle. The bundle is published with the leaderboard submissions, and links are added to this card when they are accepted; until then it is available on request through this repository's Community tab.
• Superseded by v2. v2 is evaluated on a 12-benchmark board; see its card. v1 was not run on that board.

The goal of the project

Optitransfer's converge programme aims for top placements in each model class across the public leaderboards, starting with 7B. It brings the compatible fine-tunes the open-weights community has published into a single model per class. Every release is measured against its base model on identical items and published with its regressions and per-item evidence. The construction method is proprietary; the evaluation is public. See the programme page.

Lineage

Results on the published weights: v1 vs Qwen2.5-7B-Instruct (paired, identical items)

All rows come from one notebook run, the GSM8K replay: both models in one evaluation session (the base model on 2026-09-28, v1 on 2026-09-29), with the same build, harness, prompts, items and decoding: lm-eval 0.4.13 on the Hugging Face transformers backend. Scores are percent.

• v1-only / seed-only are the discordant counts: items only v1 got right, and items only the base model got right.
• Reading rule (the same as v2's board): a difference is an improvement or a regression only if the paired-bootstrap 95% CI excludes 0 and the exact McNemar p < 0.05. Anything else is "no measurable difference". For every row in this table the paired test and the rule were fixed in the evaluation code before the run.
• Uncalibrated means no published Qwen2.5-7B-Instruct reference with a matching protocol was used: the absolute score is specific to this harness, while the paired difference is valid on every row.

Scoreline: 1 improvement (answer format) / 0 regressions / 4 no measurable difference. Every v1-vs-base comparison from this run is reported. These rows cover three benchmarks; only GSM8K was run at full size.

¹ GSM8K answer format. lm-eval's strict-match scorer counts an answer only when it is written as #### N. It found no such answer in 198 of the base model's 1,319 outputs and in 112 of v1's. The flexible-extract scorer reads the last number in the output, whatever the format, and shows no measurable difference; so do a numeric final-answer scorer and math-verify (secondary table below). Read the strict-match gain as more answers in the expected format, not as more problems solved.

² 30-item rows. The first 30 test items of each benchmark. At n = 30 the 95% intervals are 10 to 30 points wide, so only a very large difference could register. v1's per-item verdicts in this run are identical to those of an earlier run of the same evaluation code on v1 (2026-09-25), and the base model's verdicts were identical across three repeats in this run.

Sampling on the first 30 GSM8K test problems. 32 sampled answers per problem for each model (seeds 0–31; the repository's sampling settings, T 0.7, top_p 0.8, top_k 20, repetition penalty 1.05; 5-shot, at most 256 new tokens). Each draw is read as the number after its last ####; draws without one are skipped. maj@k is the most frequent answer among the first k draws (ties go to the earliest draw); pass@k counts a problem if any of the first k draws is right. The paired test was fixed in the evaluation code before the run.

All rows: no measurable difference.

Honest framing

• No lift beyond answer format was measured. On the published weights, v1's one improvement over its base model is GSM8K answer format; every other row shows no measurable difference.
• What is tested. The tested claims are paired differences against Qwen2.5-7B-Instruct on identical items.
• Coverage is narrow. No other benchmark has been run on the published weights; in particular no MMLU, MATH, instruction-following or full-size code benchmark. v2's card covers those benchmarks for v2.
• Self-reported results. These are first-party evaluations by the model's developers. The evidence bundle (see Evidence above) holds the per-item verdicts behind every row, both models' raw GSM8K outputs, and recompute.py, which recomputes every number on this card and exits non-zero on any mismatch.
• Contamination. The source models' training data is third-party and cannot be certified free of benchmark contamination.

Limitations

• Superseded. v2 is the current state of the collective; use v1 only to study the lineage.
• Narrow evaluation. Three benchmarks, of which only GSM8K at full size. The earlier card's evaluation table is withdrawn (see Corrections).
• The GSM8K strict-match gain is answer format, not more problems solved (note ¹).
• Not a long-context model. It is configured for 32,768 tokens (max_position_embeddings; no rope scaling), and no long-context evaluation was run. tokenizer_config.json inherits model_max_length: 131072 from the base model; that value is not v1's configured context.
• Identifies as Qwen. With no system message, the inherited chat template inserts "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." The model may describe itself as Qwen.
• Default decoding samples. The repository's generation settings sample (T 0.7, top_p 0.8, top_k 20, repetition penalty 1.05). The greedy rows on this card kept that repetition penalty (see Evaluation setup); other engines and decoding settings were not evaluated.
• Not evaluated for safety, bias or toxicity. Evaluated in English only. It may produce incorrect, biased or harmful content; do not use it for high-stakes decisions without independent evaluation.

Quick start

v1 loads and prompts exactly like Qwen2.5-7B-Instruct. The card's scores were produced in bfloat16 with plain-completion prompts (chat template off); for chat use, apply the template as below.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Optitransfer/Qwen2.5-7B-Instruct-converge-collective-v1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="auto"  # use torch_dtype= on older transformers
)

messages = [{"role": "user", "content": "A train travels 180 km in 2.5 hours. What is its average speed in km/h?"}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)

# Greedy, as evaluated: the repo's repetition_penalty=1.05 stays in effect.
# Without do_sample=False the repo's generation_config samples (T=0.7, top_p=0.8, top_k=20).
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Reproducing the full GSM8K rows (public tools only; torch 2.11.0 with CUDA 12.8, transformers 5.16.1, lm-eval 0.4.13, accelerate 1.15.0; one A100-SXM4-80GB). Run once per model, with Qwen/Qwen2.5-7B-Instruct,revision=a09a35458c702b33eeacc393d103063234e8bc28 for the control:

lm_eval --model hf \
  --model_args pretrained=Optitransfer/Qwen2.5-7B-Instruct-converge-collective-v1,dtype=bfloat16 \
  --tasks gsm8k --num_fewshot 5 --batch_size 4 --seed 42,1234,1234 \
  --log_samples --output_path results/

recompute.py in the evidence bundle (see Evidence above) recomputes the paired statistics from the per-item verdicts.

Evaluation setup

• Models: v1 (model.safetensors sha256 3aa6e49a7fc4b37f42b184494183a090ef2f9818418022e38be5e5ff64507db0) and Qwen/Qwen2.5-7B-Instruct at revision a09a35458c702b33eeacc393d103063234e8bc28 (the control), on identical items. Before the run the served weight bytes of both were checked against their published sha256 before the run.
• Engine: lm-eval 0.4.13 on the Hugging Face transformers 5.16.1 backend (torch 2.11.0, CUDA 12.8, Python 3.13.15, accelerate 1.15.0), bfloat16, one NVIDIA A100-SXM4-80GB. Full GSM8K: batch size 4, seeds 42,1234,1234 (lm-eval's Python, NumPy and torch seeds). 30-item rows and sampling: batch size 16.
• Prompt format: plain completion, chat template off: lm-eval's task prompts (GSM8K "Question: … Answer:", 5-shot from the train split; ARC-Challenge and HumanEval 0-shot).
• Decoding: greedy, with the repository's generation_config.json left in place, so transformers applied its repetition penalty of 1.05; the resolved settings were recorded for every generation call. Sampling rows: seeds 0–31, T 0.7, top_p 0.8, top_k 20, repetition penalty 1.05. This differs from v2's board (vLLM, repetition penalty 1.0), so absolute scores are not comparable between the two cards.
• Generation budgets (max new tokens): GSM8K 256; HumanEval 1,024; ARC-Challenge log-likelihood scoring, no generation.
• Data: openai/gsm8k (main, test) at revision 740312add88f781978c0658806c59bc2815b9866; both models received identical prompts on all 1,319 items. ARC-Challenge and HumanEval: the datasets lm-eval 0.4.13's tasks load, revision not pinned.
• Statistics: paired bootstrap, 10,000 resamples (multinomial over the discordant cells, NumPy default_rng(20260928)), 95% percentile CI; exact two-sided McNemar test on the same items. The rule and the paired test were fixed in the evaluation code before the run, except the secondary numeric and math-verify rows.
• Checks: every suite completed with full item coverage, identical item sets for both models and a passing decoding audit. The paired statistics were recomputed independently from the per-item verdicts and match the run's own.
• Run: the GSM8K replay, a notebook run (code fixed before the run); base model 2026-09-28, v1 2026-09-29, in one session with one build.

Model details

• dtype note. config.json declares torch_dtype: bfloat16, unchanged from the base model, while the stored tensors are float16. All reported results loaded the weights in bfloat16. Float16 inference was not evaluated.
• File hashes (sha256): config.json 7463bb0ea78315365e6c6b74de4e73bbcc8359dfb0c5a737584e077d42c0b03c; generation_config.json 3a8f9087e486054c8a4a08dae2e5a3ba62e23da212b5b8c08bc42cb983c3459f; tokenizer.json c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539; tokenizer_config.json 5b5d4f65d0acd3b2d56a35b56d374a36cbc1c8fa5cf3b3febbbfabf22f359583.
• Check a download against the model.safetensors sha256 above. A card-only commit changes the revision hash, not the weights.

License

This model descends from facebook/opt-2.7b. That model is distributed under the OPT-175B License Agreement, which limits use to non-commercial research. This model is therefore made available for non-commercial research use only, subject to three sets of terms:

• the OPT-175B License Agreement;
• the Apache-2.0 licenses of the Qwen, Mistral, SmolLM2, Granite and Pythia sources;
• the MIT licenses of microsoft/phi-2 and microsoft/Phi-3-mini-4k-instruct.

No rights are granted beyond those licenses. OPT-175B is licensed under the OPT-175B license, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

model.safetensors is modified relative to every source model. config.json, generation_config.json, tokenizer.json, tokenizer_config.json, vocab.json and merges.txt are Qwen2.5-7B-Instruct's files, unmodified.

Not affiliated with or endorsed by Alibaba Cloud or the Qwen team. "Qwen" is used only to identify the model this one is derived from.

Corrections

Earlier versions of this card contained the following, now withdrawn; they should not be relied on:

• an apache-2.0 license declaration;
• build-process statements;
• statements about the build software's licence terms and related rights;
• guarantee statements about the project (convergence, provability, permanence) that this release does not demonstrate, and a "first" claim;
• an evaluation table, with BEAT / TIE / LOSE labels, that is not hash-linked to these weights and had no significance test. It is withdrawn.

The weights have not changed since the first weight upload.