LFM2.5 TinyShell
Fine-tuned compact language model for the TinyShell ShellIntent natural-language-to-structured-IR task.
Base model
LiquidAI/LFM2.5-230M
This is a supervised fine-tune of the upstream LFM2.5 instruction model. The
derivative remains subject to the LFM Open License
v1.0; the full license text is included in
LICENSE.
Task
The model converts natural-language instructions into structured TinyShell ShellIntent JSON.
Supported high-level decisions include:
compileclarifyunsupported
Held-out evaluation
| Metric | Result |
|---|---|
| JSON parse rate | 99.00% |
| Schema validity | 96.00% |
| IR exact match | 37.50% |
| Decision accuracy | 96.00% |
| Operation accuracy | 64.50% |
| Slot precision | 70.31% |
| Slot recall | 61.66% |
| Slot F1 | 65.70% |
| Risk accuracy | 95.50% |
| Confirmation accuracy | 96.00% |
| Clarify accuracy | 100.00% |
| Unsupported accuracy | 60.00% |
| Multi-operation accuracy | 31.11% |
| Median inference latency | 1260.4274300001634 ms |
Generation policy
Native generation termination
FunctionGemma and Falcon-H1 initially produced a valid first JSON object but frequently continued generating additional content. Their corrected final evaluation uses a generation-time stopping criterion that terminates once the first complete top-level JSON object is generated. This is generation control, not post-hoc JSON repair.
LFM2.5 terminated correctly under the original inference configuration.
Training
The model was fine-tuned with supervised causal language modeling.
- Seed:
42 - Best validation loss:
0.09860268847667612 - Training time:
827.8969688260004seconds - Peak GPU memory:
4.324039459228516GB
Prompt tokens were masked from the language-model loss and the assistant JSON response was used as the supervised target.
Training used 1,600 examples, with 200 validation examples and 200 held-out
test examples. The random seed was 42. The frozen source hashes and complete
training metadata are included in evaluation/training_result.json.
Included files
- Fine-tuned model weights
- Model configuration
- Tokenizer / processor files
- Chat template when saved
- Generation configuration when saved
evaluation/final_metrics.jsonevaluation/test_predictions.jsonlevaluation/training_result.jsoninference_example.pyrequirements.txtLICENSEandNOTICESHA256SUMS.txt
Limitations
This pilot used one training seed. Test-set bootstrap intervals quantify held-out sample uncertainty but do not replace independent repeated training.
Exact ShellIntent matching is intentionally strict: one incorrect operation, argument, or structured field makes the complete IR prediction incorrect.
This model emits untrusted structured intent. Do not execute model output directly. Validate the JSON against the TinyShell schema, compile it through a deterministic platform-aware compiler, apply safety checks, and require user confirmation where appropriate.
License
The model weights are a derivative of LiquidAI/LFM2.5-230M and are released
under the LFM Open License v1.0. The full
license text is included in LICENSE; attribution is included in NOTICE.
Commercial use is subject to the license’s annual-revenue threshold.
The TinyShell training data contribution is attributed under CC BY 4.0. Upstream source material may have separate terms; see the TinyShell dataset documentation for details.