Tiny Function-Calling LM (~0.47M params)

A from-scratch decoder-only transformer with ~471,760 parameters, trained to route natural-language requests to a single tool (web_search) or abstain (none). Built as a demonstration of function-calling on an extremely small budget.

Architecture

• 4 transformer layers, d_model=80, 4 attention heads (head dim 20)
• RoPE positional encoding, RMSNorm, GELU feed-forward (4x width), tied input/output embeddings
• BPE tokenizer with a 2,048-token vocabulary trained on the task's own synthetic data
• Context length: 80 tokens

Output format

web_search|query=<search terms>|recency=<day|week|any>
none

Loading

This is not a registered transformers architecture — it uses a small custom model class. Load it with the reference implementation (model.py) from the companion GitHub repo:

Code: https://github.com/YOUR_USERNAME/tiny-fc-lm

import torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import TinyLM  # from the GitHub repo

state = load_file("model.safetensors")
tok = Tokenizer.from_file("tokenizer.json")
model = TinyLM(vocab=2048, d=80, n_layers=4, n_heads=4, ffn_mult=4, max_len=80)
model.load_state_dict(state)
model.eval()

See the GitHub repo's infer.py for constrained decoding and tool dispatch.

Training data

80,000 synthetic (request, tool-call) pairs generated from templated phrasings across ~10 intents (news, weather, price, stock, how-to, sports scores, definitions, generic web search, and weekly recaps), plus chit-chat examples mapped to none.

Evaluation

Limitations

• Single tool only (web_search); not a general-purpose assistant.
• Learned via memorized entity↔pattern associations rather than true entity copying, so genuinely novel named entities (names/places never seen in training) are sometimes replaced with a memorized default instead of preserved.
• English only; no multi-turn context.

License

MIT. Provided as-is for research/educational use.