Vinci-MLE-8B-1.0

Open-weight ML engineering research model trained for evidence-based diagnosis, bounded workspace changes, and justified no-change or stop decisions.

Canadian-developed by SimpleDirect · Apache-2.0 · Research baseline

Vinci-MLE-8B-1.0 is an 8B-class adaptation of IBM Granite 4.1 8B, developed by SimpleDirect, a Canadian AI lab. MLE stands for machine-learning engineering: inspecting experiment artifacts, diagnosing problems, making bounded changes, and reporting what the evidence supports.

Vinci MLE 1.0 establishes an open-weight research baseline for this workflow: a specialization corpus, an instrumented tool interface, selective assistant supervision, and standalone merged BF16 weights with the tokenizer and chat template included.

Evaluation scope: 1.0 reports its ML-engineering specialization and measured general-capability retention. A valid held-out MLE-task evaluation was not completed for this release and is intended as a release-gating measurement for 1.1.

Specialization · MLE evaluation status · Quick start · General capability retention · Limitations

Model at a glance

Artifact sizes above were read from the Hugging Face API on September 28, 2026. Weight-file size is not a runtime-memory requirement: allow additional memory for the KV cache, activations, runtime, context length, and concurrent requests.

ML engineering specialization

Vinci MLE is trained as a policy inside an instrumented ML workspace, not only on question-and-answer pairs. Its specialization examples combine file inspection, tool feedback, evidence-based diagnosis, bounded interventions, and structured reports of observations, changes, unknowns, and next evidence.

The frozen specialization corpus contains 91 trajectories across 64 distinct task roots in eight task families. Both sizes consumed the same tokenized training examples; their LoRA targets and optimization recipes differ. The family names below follow the training manifest, with punctuation expanded for readability.

Act / No Change / Stop is central to the training design. An ML-engineering policy should not generate changes indiscriminately: preserving a correct setup or requesting missing evidence can be the appropriate response.

These counts describe training coverage, not success rates. They do not establish that the model chooses the correct action, no-change, or stop decision on new tasks.

MLE evaluation status

Vinci MLE 1.0 does not yet have a valid held-out ML-engineering task score.

The internal development screen did not provide a valid capability comparison: the parent and every development fine-tune produced parse errors or timeouts, with no successful episodes. Those instrument-floor outcomes did not establish a valid MLE-task score for the released checkpoint. The recipe-selection details are retained in Training.

The preregistered held-out MLE evaluation and external MLE-bench evaluation were not completed for this release. We report MLE training scope separately from measured general-capability comparisons and make no MLE-task performance claim for 1.0.

The full evaluation also requires a qualified execution environment with the intended isolation and reproducibility controls. That environment was not ready for the complete MLE evaluation in 1.0, and we chose not to substitute a weaker test as an equivalent result.

For Vinci MLE 1.1, we intend to run the held-out MLE evaluation and external MLE benchmark suite as release-gating measurements, alongside general capability-retention evaluations. Results will be reported when measured; this 1.0 card does not predict them or promise a release date.

Intended workflow

Illustration of training intent; tools are supplied by the application. This is not a captured model run.

A task supplies a problem statement, workspace files, and tool feedback. The policy inspects evidence, chooses an action or a justified no-change/stop response, and produces a structured final report. Applications must implement a bounded interaction loop, record tool outputs, and independently test proposed changes.

The weights alone are not the full Vinci MLE agent harness. This repository supplies the model and tokenizer. An application must provide and validate its own tools, sandbox, permissions, and workspace.

Training interface

The workspace interface uses task-specific subsets of inspect_files, run_command (an argv array), and write_file (path and content). All 91 include run_command; 36 expose all three tools. Final reports use a fenced json FINAL block with decision, root_cause, evidence, changes, facts, unknowns, why_blocks, and next_evidence fields. Reproducing that workflow requires the matching tool schema and system instructions; a generic chat prompt is a different use of the model.

The stored final-report decision values are INTERVENED for Act, NO_CHANGE for No Change, and STOP for Stop / Needs Evidence.

For an initial text-only exploration, the quick start asks about a small data-splitting problem. It is an illustrative prompt, not a captured model answer or a benchmark result.

Choosing a size

The 8B weights occupy approximately 17.58 GB (16.38 GiB). This is the smaller download and lower weight-memory footprint in the pair. The 30B model has higher recorded BFCL and LiveCodeBench scores. The benchmark settings and LiveCodeBench timing caveat below apply to the comparison. The two models use the same training examples but different recipes; this is not a controlled experiment isolating parameter count, and there is no measured MLE-task ranking between them. Choose 30B when its higher measured tool/coding scores justify the larger download and serving footprint. Choose 8B when a smaller download and lower weight-memory footprint matter more.

Quick start

Download a pinned snapshot

Install huggingface_hub, and authenticate with hf auth login if repository access requires it. This example pins the uploaded model revision evaluated by the release records; later card-only revisions do not change those model files.

from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="simpledirect/Vinci-MLE-8B-1.0",
    revision="75256cf293770b6439b5ca52b98d43aa7687294d",
    local_dir="./vinci-mle-8b",
)
print(model_dir)

The pinned snapshot includes its historical README and matching checksums. To download a later card revision, use that revision's full commit hash and verify its own SHA256SUMS.

Generate with Transformers

Requires PyTorch, Accelerate, and Transformers with Granite support. Load the merged checkpoint directly; no PEFT adapter needs to be applied.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_dir = "./vinci-mle-8b"
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=False,
).eval()

messages = [
    {"role": "system", "content": "You are an ML engineering research assistant. Separate observations from hypotheses and explain what to check next."},
    {"role": "user", "content": "A dataset contains multiple records per customer. Training and validation rows were split at random. What should I inspect before trusting the validation score? Do not claim to have run any checks."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
)
device = model.get_input_embeddings().weight.device
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
completion = output[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(completion, skip_special_tokens=True))

This is a reference loading example, not a rerun of the release evaluation. It uses an illustrative system prompt and does not reproduce the Vinci agent harness. No generated answer is asserted by this card.

Serve an OpenAI-compatible endpoint with vLLM

The release evaluation used vLLM 0.22.1, a 16,384-token limit, greedy decoding, the packaged chat template, and the hermes tool-call parser. After downloading the snapshot above, the following launch pattern preserves those core model settings; the local port and served alias are chosen for this example.

CUDA_VISIBLE_DEVICES=0 \
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 VLLM_USE_DEEP_GEMM=0 \
python -m vllm.entrypoints.openai.api_server \
  --model ./vinci-mle-8b \
  --tokenizer ./vinci-mle-8b \
  --chat-template ./vinci-mle-8b/chat_template.jinja \
  --served-model-name vinci-mle-8b \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --seed 0 \
  --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --host 127.0.0.1 --port 8000

Use a GPU with enough memory for the weights and serving overhead. Tensor parallel 1 describes the evaluated setup; this card does not certify a minimum GPU model or all alternative parallel configurations.

Send requests to http://127.0.0.1:8000/v1/chat/completions, using model: "vinci-mle-8b", temperature: 0, and an explicit max_tokens budget. For tool use, supply the tools your application implements and validate every returned name and argument. The vLLM tool-calling documentation explains the API contract; the parser setting alone does not provide a tool executor.

8B known issue: in one fixed tool-call smoke run, case C1 failed: the model called a tool named none when a terminal JSON answer was expected. The other cases passed. Validate tool names and distinguish a final answer from a tool call in your application.

The three declarations below match the three-tool training variant used in 36 trajectories. This example sends a request and prints the model response; it does not execute returned commands. The system instruction is the common training instruction. The user task is illustrative. A complete agent needs a workspace, permission checks, tool implementations, returned observations, and a bounded generation loop.

import requests

tool_definitions = [
    {
        "name": "inspect_files",
        "description": "Print the task objective and every public workspace file.",
        "parameters": {
            "type": "object", "properties": {}, "additionalProperties": False,
        },
    },
    {
        "name": "run_command",
        "description": "Run a command in the workspace and return its stdout and stderr.",
        "parameters": {
            "type": "object",
            "properties": {"argv": {"type": "array", "items": {"type": "string"}}},
            "required": ["argv"], "additionalProperties": False,
        },
    },
    {
        "name": "write_file",
        "description": "Replace a workspace file with the given full content.",
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string"}, "content": {"type": "string"}},
            "required": ["path", "content"], "additionalProperties": False,
        },
    },
]
response = requests.post(
    "http://127.0.0.1:8000/v1/chat/completions",
    json={
        "model": "vinci-mle-8b",
        "messages": [
            {"role": "system", "content": "Assess this private TRAIN workspace using recorded tools and a structured FINAL answer."},
            {"role": "user", "content": "Inspect the workspace to identify how the training and validation data are split. Gather evidence before proposing a change."},
        ],
        "tools": [{"type": "function", "function": tool} for tool in tool_definitions],
        "tool_choice": "auto",
        "temperature": 0,
        "max_tokens": 512,
    },
    timeout=120,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"])

Install requests to run this client example. Neither these tool descriptions nor the model's response implement a sandbox. Only expose tools whose actions your application can validate and contain.

General capability retention

The evaluations below measure coding, tool calling, general reasoning, agentic behavior, refusal patterns, and long-context retrieval relative to the pinned Granite parent. They characterize broader parent capabilities after MLE specialization. They are not substitutes for a held-out ML-engineering benchmark. Only the tool-calling and coding comparisons have the stated retention pass criterion; the other studies are descriptive and retain their individual caveats.

Results below were resolved from the frozen release records on September 28, 2026, against this model's own pinned Granite parent. The summary job verified each source receipt's SHA-256 and used the frozen evaluator for the BFCL and LiveCodeBench comparisons. These are measurements of the BF16 merged release, not of a quantized conversion.

Tool calling and coding

BFCL measures tool-call retention; LiveCodeBench measures coding retention. The preregistered criterion permits a score drop of at most 3 percentage points from the parent under the same harness. Scores below are on a 0–100 scale; differences are this model minus its parent, in percentage points.

BFCL covers 11 categories and excludes the executable-code category. LiveCodeBench covers 1,055 items. Both recorded scores fall within the preregistered 3-point retention margin. LiveCodeBench retains the separate timing-qualification limitation described below. These comparisons do not demonstrate improved MLE-task performance.

* LiveCodeBench timing adjudication: INCONCLUSIVE / faster-machine qualification failed. A timing flag affected 75 of the parent's and 82 of this model's 1,055 items. The proposed faster regrading host did not meet its qualification threshold (ratio 0.908 against a maximum of 0.90), so no regrade was performed. The table reports the existing record grades with that unresolved timing limitation.

LiveCodeBench grading used the evaluator's full sandbox isolation; container stderr was not retained. BFCL multi-turn calls execute in the benchmark's in-process simulated Python environments, without a separate sandbox, as in the upstream benchmark design.

General knowledge, reasoning, and instruction following

Descriptive; no multiplicity correction; not a retention verdict. These paired comparisons have no preregistered pass rule. Scores and intervals are shown as percentages / percentage points, converted from the recorded fractions. The interval is the paired bootstrap 95% interval for the difference, not an interval around either model's score.

Values are rounded from the source record; subtracting displayed scores can differ slightly from the separately rounded recorded difference. Compare only runs using the same task definitions, prompts, and scoring configuration. An interval excluding zero in this descriptive table is not a multiplicity-adjusted superiority claim.

IFEval uses strict prompt-level accuracy; MATH-500 uses math_verify. The other rows use exact match with their locked extraction rules: GPQA flexible-extract, GSM8K strict-match, MMLU-Pro custom-extract, and BBH get-answer.

Agentic evaluation

Descriptive; no multiplicity correction; not a retention verdict. The tau2 study reports pass¹ over four trials per task: 50 airline tasks (200 trials) and 114 retail tasks (456 trials). Confidence intervals resample whole tasks, preserving the four correlated trials.

Airline primary uses the upstream database × communication reward. Retail primary measures database end-state success only, which differs from the upstream retail reward. Retail secondary combines database success with natural-language assertions judged by a fixed local Qwen3.8-27B model; it is an approximation, not the official tau2 retail score. Non-completed trials score zero; the secondary comparison excludes pairwise unjudged trials, with its actual included count shown above.

Additional interrupted arms are preserved in the release evidence; only the comparable parent/candidate pair selected by the preregistered pairing rule is reported here.

An additional parent arm completed before scheduler preemption interrupted the candidate arm and an automatic requeue reran the full evaluation unit. The reported pair uses the first complete unit with both arms evaluated against the same user-simulator process; that selection rule was fixed before outcomes were read. The extra parent arm was retained separately and was not paired, pooled, or substituted. Its numeric results are not included in the frozen summary used here.

These are agentic benchmark measurements, not ML-engineering task results. Greedy decoding under vLLM continuous batching is not bitwise deterministic, so reruns can produce different item-level outputs.

Safety: refusal-pattern measurements

Reporting updated following a classifier audit. Model weights are unchanged.

Descriptive; no multiplicity correction; not a retention verdict. A9 compares refusal patterns between Vinci MLE and its pinned parent on XSTest and StrongREJECT using two registered refusal-phrase windows: the start of the normalized response and the first 300 normalized characters.

Classifier audit. In a blind review of 90 responses per model by an independent AI agent reviewer, with model identity and classifier output hidden, neither classifier window labelled a compliant answer as a refusal in the audited sample. The first-300-character window identified 46/60 parent and 44/60 Vinci MLE refusals identified by the reviewer. The start-of-response window identified 0/60 and 1/60, respectively. Because the first-300-character window was substantially more sensitive in this audit, it is reported first below; the registered start-of-response measurement is retained for completeness.

Detected refusal-pattern differences — first 300 characters

On safe XSTest prompts, a higher detected-refusal rate can indicate increased over-refusal. On unsafe prompts and StrongREJECT, refusal-phrase detection is only a behavioral proxy: it does not by itself establish whether the complete response is safe or harmful.

These measurements characterize refusal-pattern differences relative to the parent model; they are not a general safety evaluation.

Long context: synthetic retrieval

Descriptive; no multiplicity correction; not a retention verdict. The RULER study averages two replicates on 1,500 paired synthetic items across six nominal input lengths. Its string_match_all metric is the fraction of expected strings present, case-insensitive; it is not a binary all-strings-correct measure. Actual tokenized prompt lengths vary around each nominal length.

This measures synthetic retrieval, not understanding an ML repository or operating at the parent's full 131,072-position limit. The 8B result is lower overall; the table retains that unfavorable comparison.

The release records additionally mark repeated-run variance studies as measured for AIME, GSM8K, IFEval, and MATH-500. Their numeric ranges are not included in the resolved numbers summary used here and are not reported in this card. The paired tables above should not be interpreted as proof of bitwise repeatability.

Evaluation scope: what was not run

The full preregistered evaluation protocol was not completed for 1.0 because the MLE-specific held-out component remains unmeasured. The benchmark component results above retain their documented scope and caveats; they do not constitute a completed MLE release-evaluation result. No incomplete gate is reported as passed.

Training

Vinci applied supervised fine-tuning to the pinned Granite parent using ordinary LoRA, then merged the terminal adapter into the parent for distribution. This recipe uses neither DoRA nor rank-stabilized LoRA.

Counts were recomputed from the frozen training receipts for this card. These receipts describe the specialization data, not the Granite parent's pretraining corpus. Both MLE sizes consumed the same tokenized-example bytes.

Selective assistant supervision. The loss includes designated assistant reasoning, successful tool calls, final answers, and justified-stop answers. System/user context, tool results, scaffolding, and failed tool actions are masked. Verifier feedback is excluded from the rendered prompt. Failed actions therefore do not automatically become positive imitation targets. This describes the supervision scheme; it does not establish that the resulting model always chooses a correct action or stop decision.

Development-selection limitation. Parse errors and timeouts drove the parent and eligible development arms to the evaluator floor, so the screen could not support a capability comparison. The frozen recipe-selection fallback was used; full details are retained in the release evidence.

The parent and the selected development arm each produced 14 parse errors and 2 timeouts across 16 episodes, and all four development arms recorded 0 successes out of 16. The screen therefore supplies no valid capability comparison or evidence of improvement over the parent; its records do not independently diagnose the cause of the parse failures. The recipe was chosen by the frozen lower-dose tie-break, not demonstrated superiority. These are development-screen outcomes, not an MLE score for the released model. The final release uses the predetermined terminal checkpoint under the fixed final-training policy.

Limitations

• No completed MLE-specific evaluation. General coding, tool-calling, and reasoning scores cannot fill that gap.
• Small specialization corpus. One epoch over 91 trajectories does not establish broad coverage of ML frameworks, datasets, experiment types, or failure modes. The eight training families describe corpus coverage, not demonstrated success rates.
• Output and tool errors remain possible. Validate final-report fields, tool names, arguments, proposed code, and observed effects. The 8B smoke result is disclosed above.
• LiveCodeBench timing remains unresolved. Read its score with the qualification caveat, not as a fully adjudicated timing result.
• Merge integrity is not behavioral equivalence. The release export passed mechanical verification, standalone loading, and byte-identical rebuild checks. Behavioral fidelity to the original adapter was not measured.
• Context scope is bounded. The 131,072-position parent configuration is not proof of reliable MLE behavior at that length; the evaluation serving limit was 16,384 tokens.
• No safety certification or unattended-deployment validation. Safety benchmark observations do not establish a safety property. Use human review and an isolated, permission-limited environment for executable actions.

Contamination audit

The training-data audit covered 19 benchmark/subset entries. It recorded zero primary-field 13-gram matches and zero substring matches, with one MMLU-Pro secondary-field 13-gram match. Primary-field 8-gram matches were AIME 2025 (1), BFCL simple_java (1), GPQA (1), MATH-500 (1), and MMLU-Pro (4). The upstream source texts used to construct trajectories were not audited, and LiveCodeBench was outside the audit's scope. These checks do not establish a contamination-free evaluation.

Provenance and file integrity

The repository includes PROVENANCE.json, SHA256SUMS, LICENSE, and NOTICE. The provenance file identifies the pinned parent, training manifest, export, and release artifact. Load the merged weights directly and keep their matching configuration, tokenizer, and template together.

The frozen artifact-tree digest identifies the model export, not the current documentation directory. Card updates receive a new Hugging Face commit and a corresponding README checksum; they do not change the frozen model identity.

After downloading one complete revision, run this from its directory:

sha256sum -c SHA256SUMS
# macOS alternative:
shasum -a 256 -c SHA256SUMS

Checksums detect differences against the recorded reference; they do not independently authenticate its publisher. This repository does not include a detached MLE release signature.

These hashes identify the internal records used for the tables. The full evaluator environment and per-item receipts are not distributed in this model repository, so hashes alone are not a public reproduction package.

License and attribution

The released weights are licensed under Apache-2.0. See LICENSE and NOTICE for the accompanying terms and attribution. IBM developed Granite; SimpleDirect developed this fine-tuned derivative and its release. No affiliation with or endorsement by IBM is implied.

Citation

@misc{vinci_mle_1_0_8b,
  title  = {Vinci MLE 1.0 (8B)},
  author = {{SimpleDirect}},
  year   = {2026},
  url    = {https://huggingface.co/simpledirect/Vinci-MLE-8B-1.0}
}

SimpleDirect · Vinci models