mind2web-candidate-ranker
A DeBERTa-v3-base cross-encoder fine-tuned for candidate generation in web agents: given a natural-language task (plus action history) and a DOM element, it scores how likely that element is the next action target. It is stage 1 of a two-stage computer-use agent (rank DOM elements → LLM picks the action → Playwright executes).
Trained on Mind2Web (2,000+ tasks across 137 real websites).
Results
Evaluated on the three official Mind2Web generalization splits:
| split | acc@1 | recall@3 | recall@5 | recall@10 | recall@20 | MRR |
|---|---|---|---|---|---|---|
| test_task (unseen tasks) | 60.2% | 83.5% | 89.4% | 93.0% | 93.8% | 0.725 |
| test_website (unseen websites) | 50.9% | 76.1% | 85.3% | 92.6% | 95.3% | 0.656 |
| test_domain (unseen domains) | 55.4% | 79.8% | 87.1% | 92.8% | 94.5% | 0.689 |
~93% recall@10 on entirely unseen domains. The recall curve plateaus at k≈20, making top-20 a good cost/recall operating point for the downstream LLM.
Usage
from sentence_transformers import CrossEncoder
model = CrossEncoder("torontodeveloper/mind2web-candidate-ranker", num_labels=1, max_length=512)
candidate = "tag: input | aria_label: Destination | placeholder: To?"
query = "task is: Search for a flight from Toronto to New York. Type NYC into the destination field.\nPrevious actions: "
score = model.predict([(candidate, query)])
Candidates are serialized DOM elements (tag, aria_label, placeholder, name,
is_clickable, bounding box, etc.); the query is the task description plus the last
few actions. Score all interactable elements on a page and keep the top-k.
Training
- Base model:
microsoft/deberta-v3-base(86M params) - Data: Mind2Web train split — 7,775 actions, 1 positive + 3 sampled negatives each (31,100 pairs)
- 3 epochs, batch size 8, lr 3e-5, BCEWithLogitsLoss, linear warmup
- Single T4 GPU (~85 minutes), fp32
⚠️ Precision note
DeBERTa-v3's disentangled attention overflows fp16 (NaN logits). Load and run this
model in fp32 (or bf16 on Ampere+). Newer transformers versions may default to fp16
on GPU — pass torch_dtype=torch.float32 explicitly if needed.
Pipeline context
This model is the candidate generator from mind2web-computer-use-agent: stage 2 (action prediction) uses FLAN-T5 / GPT-class models over the top-k candidates, and stage 3 executes via Playwright. Architecture follows MindAct (Deng et al., 2023).