JevForge-0.8B
JevForge-0.8B is an experimental candidate-conditioned scoring model for structured web decisions. It starts from the pretrained Qwen3.5-0.8B backbone, adds a shared scalar decision head, and returns a probability distribution over the candidates supplied for each question.
This is not a chat model and is not intended to be loaded as a standard
text-generation pipeline. Use the jevforge.predict.Predictor implementation
from the JevForge source repository.
Evaluation
Evaluation uses frozen website-disjoint splits and temperature 0.9717.
| Metric | Test (2,400) | OOD (1,158) |
|---|---|---|
| Choice top-1 in positives | 0.5787 | 0.6373 |
| Choice exact gold | 0.5675 | 0.6192 |
| Noul accuracy | 0.8263 | 0.8601 |
| Noul Brier | 0.1283 | 0.1173 |
| Score MAE | 0.3674 | 0.3898 |
These results apply only to the recorded offline candidate-scoring protocol. They are not evidence of autonomous live-site operation or a browser-agent leaderboard result.
Raw-backbone reference
On the first 200 records of each frozen split, the unmodified
Qwen/Qwen3.5-0.8B chat model was prompted to score every candidate from 0 to
10. This is a zero-shot reference rather than an architectural ablation because
the raw model has no JevForge scalar head.
| Choice top-1 in positives | Raw backbone | JevForge-0.8B |
|---|---|---|
| Test (200) | 0.235 | 0.505 |
| OOD (200) | 0.340 | 0.630 |
Training and model lineage
- Base model:
Qwen/Qwen3.5-0.8B, loaded from pretrained weights. - Objective: supervised candidate-distribution prediction.
- Training steps: 1,200.
- Maximum sequence length: 768 tokens.
- Best checkpoint: step 1,200.
- Calibration temperature: 0.9717.
The upstream Qwen3.5-0.8B model is licensed under Apache 2.0. Its notice and license are retained in this release.
Data lineage
Training records were transformed from Mind2Web annotations obtained through
LangAGI-Lab/Mind2Web-axtree-cleaned-lite. Mind2Web is distributed under CC BY
4.0. No raw or unzipped official Mind2Web test files, and no complete JevForge
training or evaluation split, are included in this model repository. Obtain
benchmark data from the official upstream project and follow its current access
and attribution requirements.
Intended use
- Offline scoring and ranking of supplied candidate actions.
- Research on
choice, binarynoul, and orderedscoredecisions. - Reproduction of the fixed evaluation protocol using the JevForge codebase.
Limitations
- Candidate discovery is out of scope; callers must supply candidate paths.
- Offline page snapshots differ from live browser environments.
- The model has only been evaluated on the recorded task construction and frozen splits described above.