Fara 1.5-9B W8 baseline evaluation

This repository preserves the exact Fara 1.5-9B checkpoint used for both W8 DOM browser-agent baseline evaluations. The checkpoint was not trained or fine-tuned for either evaluation. Its files are an exact mirror of the pinned Microsoft checkpoint.

Common checkpoint and serving setup

The DOM serving integration translated the recorder's OpenAI Responses contract to the Fara inference endpoint. It did not modify the model weights.

Evaluation 1: 90-task baseline

Evaluation 1 provenance

Execution errors remain in the denominator. The artifact repository contains the raw recordings, logs, rubrics, verifier outputs, structured tables, and integrity manifests.

Evaluation 2: 40-task baseline

Evaluation 2 provenance

Execution errors remain in the denominator. The published dataset and the snapshot bundled with the artifacts contain the same task bytes, identified by the SHA256 above.

Repository contents

• Model configuration, tokenizer, processor, chat template, and Safetensors weights copied from the pinned upstream revision
• CHECKPOINT_FILES.json: upstream Git blob identities and LFS SHA256 hashes for the checkpoint files
• MANIFEST.json: machine-readable checkpoint provenance
• PROVENANCE.md: checkpoint revision and experiment-boundary receipt
• CHECKSUMS.sha256: hashes for the repository's reproducibility metadata

Load the checkpoint

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "WootzappLab/fara-9b-baseline-eval"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

License and attribution

The mirrored base checkpoint is released by Microsoft under the MIT License. See the upstream Fara 1.5-9B model card for its original documentation, intended use, and limitations.