LFM2.5-VL-3B-DragOn
This is LiquidAI's LFM2.5-VL-3B fine-tuned to do one thing well: drag-and-drop grounding. Show it a screenshot and an instruction like "drag the slider to 70" or "select cells B3 through F3", and it answers with the start and end points of the drag in normalised coordinates.
I trained it on about a quarter of the DragOn dataset for roughly $36 of H100 time. The base model scores 0.7% on this task. After one evening of training it scores 70.8%.
Results
DragOn public eval, 998 tasks, strict acc@5 (start point inside the start box AND end point inside the end box, 5% tolerance):
| model | acc@5 | acc@10 | acc@15 |
|---|---|---|---|
| this model | 70.8% | 78.5% | 82.1% |
| LFM2.5-VL-3B (base, zero-shot) | 0.7% | 0.9% | 1.5% |
| Qwen3.8-27B (zero-shot) | 17.4% | ||
| Claude Opus 4.7 (zero-shot) | 27.7% | ||
| DragOn paper's 27B fine-tune | 35.3% |
By domain: slider 96.0%, text highlighting 70.2%, spreadsheet cell selection 62.4%, slide element resizing 54.8%.
One caveat I want to be upfront about: my numbers are on the public eval split. The paper's 35.3% is (as far as I can tell) on their private test set, which may be harder. So "double the paper's fine-tune" is indicative, not a controlled comparison. If the Hcompany team wants to run this model on the private set, I'd genuinely love to see the number.
How it was trained
- LoRA (r=32, alpha 64) on all linear layers including the vision encoder, bf16
- One pass over ~78k screenshots / ~940k drag tasks (24% of the full set), packed so that each screenshot is encoded once and all its tasks share the image tokens
- 8.5 hours on a single H100, lr 1e-4 cosine, effective batch 16
- The training prompt is exactly the eval prompt, and the answers are bbox centres in 0-1000 coordinates
The full dataset is 3.5M tasks. I stopped at 24% because that's what my RunPod credit bought. The loss was still falling, so there's likely more on the table. If I train further I'll update this repo.
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
model = AutoModelForImageTextToText.from_pretrained("EryriLabs/LFM2.5-VL-3B-DragOn", dtype="bfloat16", device_map="auto")
proc = AutoProcessor.from_pretrained("EryriLabs/LFM2.5-VL-3B-DragOn")
img = Image.open("screenshot.png")
prompt = ("This is a screenshot of a user interface. You must perform a DRAG action.\n"
"Task: Drag the volume slider to the maximum\n"
"Give the drag as JSON with the START point (where the mouse button goes down) and the END point "
"(where it is released), in coordinates normalised to 0-1000 for both x (left->right) and y (top->bottom):\n"
'{"start":[x,y],"end":[x,y]}\nOutput only the JSON.')
msgs = [{"role": "user", "content": [{"type": "image", "image": img}, {"type": "text", "text": prompt}]}]
inputs = proc.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, do_sample=False, max_new_tokens=64)
print(proc.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Stick to the prompt format above. It's what the model was trained on, and grounding models are picky about this.
GGUF quants for llama.cpp (Q4 through F16, with the vision projector): EryriLabs/LFM2.5-VL-3B-DragOn-GGUF
The LoRA adapter on its own is in the adapter/ folder if you'd rather apply it to the base model yourself.
Limitations
It's a specialist. I haven't measured what the drag training did to general chat or VQA, so treat it as a grounding component, not an assistant. Resize/rotate drags are its weakest domain. And it inherits whatever limitations the base model has.
Thanks
- LiquidAI for LFM2.5-VL-3B. A 3B VLM this capable, released under an open licence with day-one llama.cpp support, is what made a $36 experiment possible at all.
- Nathan Bout, Maxime Langevin and Ronan Riochet at Hcompany for building and releasing DragOn. A 3.5M-task drag-grounding dataset with a clean eval is a gift to anyone working on GUI agents. Their paper: DragOn: A Drag-Grounding Benchmark and Training Dataset for GUI Agents (ICML 2026 SCALE workshop).
Trained by Dwain Barnes (EryriLabs), August 2026.