smollm2-135m-android-control
LoRA SFT of HuggingFaceTB/SmolLM2-135M-Instruct as a text-only Android UI action policy. Input: instruction + textual UI element list + action history. Output: one action as a single-line JSON object. The model does not consume images.
Output schema
| action_type | fields |
|---|---|
| click | target (element index) or x,y (0-1000) |
| long_press | target or x,y (0-1000) |
| scroll | direction ∈ up/down/left/right |
| input_text | text |
| open_app | app_name |
| wait / navigate_back / navigate_home | - |
Normalized coordinates: x = round(px / screen_width * 1000), same for y. Element bounds in the prompt use the same 0-1000 grid.
Prompt format
Chat template of the base tokenizer, system + user turn, greedy decoding.
System:
You are an Android UI control agent. Given the instruction and the current screen, output the single next action as raw JSON and nothing else.
Actions: {"action_type":"click","target":N} | {"action_type":"long_press","target":N} | {"action_type":"input_text","text":"..."} | {"action_type":"scroll","direction":"up|down|left|right"} | {"action_type":"open_app","app_name":"..."} | {"action_type":"wait"} | {"action_type":"navigate_back"} | {"action_type":"navigate_home"}
N is an element index from the UI list. If no element fits, use {"action_type":"click","x":X,"y":Y} with X and Y in 0-1000.
User:
Goal: <episode goal>
Instruction: <step instruction>
Current Android UI:
screen 1080x2400 app=com.android.settings
[0] EditText "Search settings" [56,108,944,167] click,edit
[1] TextView "Network & internet" [56,200,944,267] click
...
Previous actions:
1. open_app "Settings"
Return exactly one action.
Element line: [index] class "label" [left,top,right,bottom] flags. Label = text > content-description > hint. History entries reference element labels, not indices.
Data
- Source:
OfficerChul/Android-Control-84k@0248027f747c9d57bd09c14e8f044f9a8103dddd(step instructions, actions, screenshot names). - UI text: accessibility trees from the original AndroidControl TFRecords (
gs://gresearch/android_control), joined by episode id and step. - Element selection: visible, enabled, labelled or interactive nodes; status bar removed; top 32 by score, reading order; labels truncated to 40 chars.
- Click/long-press target: smallest kept element containing the gold point; otherwise normalized coordinates.
- History: last 5 actions.
- Upstream rows (train+test): 83,848; converted: 83,848; discarded: 0
- Rows with an accessibility tree: 83,848; without: 0
- Position encoding: coords 5,919, none 31,763, target 46,166
- Token budget: 1280 (prompt+completion); rows with elements dropped to fit: 15,889
- Train rows from episodes that also occur in the upstream test split removed: 5,642
| split | rows | episodes |
|---|---|---|
| test | 904 | 851 |
| train | 75,195 | 14,023 |
| validation | 2,107 | 400 |
Training
| parameter | value |
|---|---|
| method | LoRA |
| r / alpha / dropout | 32 / 64 / 0.05 |
| target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| trainable params | 9,768,960 (6.7706%) |
| total params | 144,283,968 |
| loss | completion-only (assistant action tokens) |
| epochs | 2 |
| batch size × grad accum | 8 × 4 |
| learning rate / scheduler | 0.0002 / cosine |
| warmup ratio / weight decay | 0.03 / 0.01 |
| max length | 1280 |
| precision | bf16 |
| seed | 42 |
| hardware | NVIDIA GeForce RTX 5060 Ti |
| software | torch 2.11.0+cu128, transformers 5.17.0, trl 0.24.0, peft 0.18.1 |
| metric | value |
|---|---|
| train_loss | 0.1554 |
| eval_loss | 0.113 |
| train_runtime | 4955.4101 |
Evaluation
Split: test (n=904), identical prompts for both models, greedy decoding. Strict JSON = the completion is exactly one schema-valid JSON object. Click target accuracy requires the correct action type and index. Coordinate accuracy: L∞ distance ≤ 50 on the 0-1000 grid.
| Metric | base | finetuned |
|---|---|---|
| JSON validity (strict) | 0.000 | 0.999 |
| JSON validity (lenient) | 0.000 | 0.999 |
| Exact action accuracy | 0.000 | 0.695 |
| Action type accuracy | 0.000 | 0.883 |
| Click target accuracy | 0.000 | 0.488 |
| Coordinate accuracy (<=5%) | 0.000 | 0.122 |
| Input text accuracy | 0.000 | 0.792 |
| Scroll accuracy | 0.000 | 0.912 |
| Open app accuracy | 0.000 | 0.824 |
Per-action exact match:
| Action | n | base | finetuned |
|---|---|---|---|
| click | 125 | 0.000 | 0.592 |
| input_text | 125 | 0.000 | 0.792 |
| long_press | 125 | 0.000 | 0.264 |
| navigate_back | 125 | 0.000 | 0.976 |
| navigate_home | 29 | 0.000 | 0.000 |
| open_app | 125 | 0.000 | 0.824 |
| scroll | 125 | 0.000 | 0.912 |
| wait | 125 | 0.000 | 0.664 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "thealper2/smollm2-135m-android-control"
model = AutoModelForCausalLM.from_pretrained(repo)
tok = AutoTokenizer.from_pretrained(repo)
messages = [
{"role": "system", "content": SYSTEM_PROMPT}, # see "Prompt format"
{"role": "user", "content": user_prompt},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt", add_special_tokens=False)
out = model.generate(**inputs, max_new_tokens=48, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# {"action_type":"click","target":1}
Limitations
- Text-only policy: no pixels. Elements absent from the accessibility tree (canvas, games, some WebViews) are not visible to it.
- Target indices are only meaningful for the element list they were predicted on; the same element selection rules must be used at inference.
- Training data has no
navigate_homeexamples and only 42long_pressexamples; the test split is class-balanced, so those classes dominate the error. open_appemits an app display name; mapping to a package is done by the executor.- Single-step accuracy on offline data does not measure closed-loop task success.