tiny-search-reader-60M

A 62M-parameter language model, trained from scratch, that reads search results and answers a question in a few words. When the results do not answer the question it says not found. It is built to run on a phone: the Q8_0 GGUF file is 70 MB.

It does not search and it is not a chatbot. Your app runs the search and passes in the question with the results (title and text, the way Google shows them), and the model returns the answer.

On 258 real Google questions it is right 79.8% of the time (206/258). About one answer in five is wrong.

That is level with Qwen3-0.6B (207/258), a model ten times its size, and on a laptop CPU it answers about 5 times faster. A SmolLM2-135M fine-tuned on the same training data scores a little higher (214/258). Details are in Compared with other models.

What it does

Made-up results, real output (the demo in searchreader.py):

Asked "Who won the 2026 Harbor Street Chess Open?" with the same results, it answers Tomas Varga. Given only results [1] and [2], it also answers Tomas Varga to the 2031 question, so not found is not guaranteed for a future year when a result names a recent winner. On the tests it catches 41 of 50 trick questions (see Limitations).

Files

How to run

transformers

pip install torch transformers tokenizers huggingface_hub

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("ahkamboh/tiny-search-reader-60M")
sys.path.insert(0, path)
from searchreader import answer_transformers

results = [
    {"title": "Harbor Street Chess Open 2025 results",
     "text": "Mara Lindqvist won the 2025 Harbor Street Chess Open with 7 points from 9 games."},
    {"title": "Harbor Street Chess Open 2026",
     "text": "The 2026 Harbor Street Chess Open was won by Tomas Varga after a tie-break against Mara Lindqvist."},
]
print(answer_transformers("Who won the 2026 Harbor Street Chess Open?", results))   # Tomas Varga

Without the helper, the prompt is plain text with special tokens (format below):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ahkamboh/tiny-search-reader-60M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = ("<|question|> Who won the 2026 Harbor Street Chess Open?<|snippet|>"
          " [1] Harbor Street Chess Open 2025 results: Mara Lindqvist won the 2025 Harbor Street Chess Open"
          " with 7 points from 9 games."
          " [2] Harbor Street Chess Open 2026: The 2026 Harbor Street Chess Open was won by Tomas Varga after a"
          " tie-break against Mara Lindqvist.<|answer|>")
enc = tok(prompt, return_tensors="pt", add_special_tokens=False)
out = model.generate(**enc, max_new_tokens=24, do_sample=False)
print(tok.decode(out[0, enc.input_ids.shape[1]:], skip_special_tokens=True).strip())   # Tomas Varga

This works as long as the prompt fits. For real search results use build_prompt, which fits them into the 512-token context the same way the model was tested.

llama.cpp

llama-completion -m gguf/tiny-search-reader-60M-Q8_0.gguf -c 512 -n 24 --temp 0 -no-cnv --no-display-prompt \
  -p '<|question|> Who won the 2026 Harbor Street Chess Open?<|snippet|> [1] Harbor Street Chess Open 2025 results: Mara Lindqvist won the 2025 Harbor Street Chess Open with 7 points from 9 games. [2] Harbor Street Chess Open 2026: The 2026 Harbor Street Chess Open was won by Tomas Varga after a tie-break against Mara Lindqvist.<|answer|>'
# Tomas Varga [end of text]

Tested with Homebrew llama.cpp build 11146. The GGUF says add_bos_token = false and its end-of-text token is id 0, so llama.cpp adds nothing in front and stops after the answer.

llama-cpp-python

pip install llama-cpp-python tokenizers huggingface_hub

from searchreader import answer_gguf   # with path on sys.path as above

print(answer_gguf("Who won the 2026 Harbor Street Chess Open?", results,
                  gguf_path=path + "/gguf/tiny-search-reader-60M-Q8_0.gguf"))   # Tomas Varga

answer_gguf builds the prompt with our own tokenizer and feeds the token ids to llama.cpp, so the special tokens are single ids and no BOS is added. It decodes greedily and stops at <|endoftext|> (id 0) or after 24 tokens. Run python searchreader.py for the two-question demo on both backends. Tested with transformers 5.18, llama-cpp-python 0.3.36 and tokenizers 0.23.

Prompt format

<|question|> {question}<|snippet|> [1] {title}: {text} [2] {title}: {text} ...<|answer|>

The model writes the answer after <|answer|> and ends it with <|endoftext|>.

Five Google results usually fit: only 3 of the 258 test prompts had to drop a result.

searchreader.build_prompt(tokenizer, question, results, ctx=512, reserve=24) does all of this. It gives the same token ids as the training code on all 258 test questions, and it takes either a tokenizers.Tokenizer or a transformers tokenizer.

Note for apps that build the prompt as text

The special tokens must reach the model as the single ids 1, 2 and 3. If you tokenize the prompt text with llama.cpp, turn on special-token parsing and turn off BOS:

With these settings the GGUF's tokenizer gives the same ids as tokenizer.json on all 258 test prompts. Without parse_special, the markers turn into ordinary text (149 tokens instead of 131 for the library demo) and in our test the model then answered with nothing. Since the markers are parsed anywhere in the text, remove the five marker strings from questions and results if they could ever contain them.

Results

Full tables, the test questions and the grading rules are in eval/results.md.

v3.1a (this release), PyTorch fp32 on CPU:

Accuracy counts right answers plus trick questions answered with not found. On the 208 questions that have an answer it is right 165 times, says not found 11 times and gives a wrong answer 32 times. It catches 41 of the 50 trick questions.

google_big by kind:

Earlier versions from this project, same tests (Colab GPU runs, bf16):

Paired tests (McNemar): against v2a, v3.1a fixes 17 google_big questions and breaks 6 (p=0.035), and on the "latest" questions fixes 13 and breaks 1 (p=0.002). Against v3a it fixes 22 and breaks 15 of the 258 (p=0.32, CPU fp32), so that gain is within noise.

Other tests (Colab bf16): SQuAD 2.0 dev F1 65.19; held-out MS MARCO and TriviaQA validation rows 63.5% accuracy. Both were part of the score used to pick v3.1a.

Caveats:

• The test sets are small. One question moves the 258 total by 0.4 points and google_big by 0.6.
• The model was picked using Colab GPU runs in bf16. CPU fp32 also gets 206/258, but not on exactly the same questions: on google_big it gets 124 against Colab's 123 (unit questions 21/25 against 20/25). Both are shown and labelled.
• google25 (24 questions) is a dev set. It was the reference for the synthetic table layouts and is part of the score used to pick v3.1a, as are real_dev and SQuAD 2.0 dev. Three google25 rows and five google_heldout rows (latest iPhone, Galaxy S and iOS, Ballon d'Or, LPG, policy rate, dollar and petrol rates) guided the v3 data design. On the 234 questions outside google25, v3.1a gets 184 (78.6%, CPU fp32).
• Not fully blind: v3.1 added training rows of three kinds (answer_type, traps_hard, units) after looking at v3a's mistakes on google_fresh and google_big, so google_fresh is not fully blind for v3.1a. New v3.1 rows on google_big subjects, or asking a test question in other words, were dropped, but that filter covered only the rows added in v3.1. The older v1 and v1.1 synthetic rows, which v2a, v3a and v3.1a all trained on, came from fact tables that include some google_big subjects and answers (for example Nanga Parbat's height and body temperature in Fahrenheit). So google_big, and its unit questions most of all, is not fully blind for any of these models. For v3.1a, only google_big questions asked word for word were removed from the older rows.
• For live values the gold list holds every value the saved results showed, since sources disagree, and any of them counts as right.
• About a quarter of the test questions (65 of 258) are about Pakistan.
• Nothing was measured on a phone.

Compared with other models

Same 258 questions and the same grading, all on Colab GPU in bf16 with greedy decoding.

• Zero-shot models got the question and the numbered results in their own chat template (Qwen3 with thinking off), with one instruction: "Answer the question using only the numbered search results below. Reply with just the answer in a few words. If the results do not answer the question, reply exactly: not found".
• The two extractive models (deepset's SQuAD 2.0 readers) pick an answer span from the five results.
• SmolLM2-135M (fine-tuned) is the base SmolLM2-135M trained for 2 epochs on exactly the rows this model was trained on.

The last column compares each model with this one question by question: questions only that model got right against questions only this model got right, with the exact McNemar test (p below 0.05 means a real difference).

What the table shows:

• It is clearly better than both SQuAD 2.0 readers and the three instruct models below 0.6B. It ties Qwen3-0.6B (37 questions against 36). Qwen3-1.7B gets 12 more right, which is not a significant gap on 258 questions. Qwen3-4B, 65 times larger, is clearly better.
• It catches 41 of the 50 trick questions, more than every model here except Qwen3-4B (44). The three instruct models below 0.6B never replied not found.
• It is weak on the 60 "latest / current / most recent" questions: 37 right, fewer than every generative model tested. Qwen3-1.7B (54) and Qwen3-4B (52) are significantly better there (p of 0.001 or less).
• SmolLM2-135M fine-tuned on the same rows scores 214 against 206. That gap is not significant (p=0.358), but on the latest questions it is (47 against 37, p=0.021), and it catches fewer trick questions (32 against 41). It has 2.2 times the parameters, and its base was pretrained on about 2 trillion tokens, against 10 billion here. So most of the result comes from the training data, and a base model with more pretraining makes better use of it.

Caveats for this comparison:

• The zero-shot models were pretrained on the open web and may know some answers without reading the results. This model's training held out every test question and topic.
• One fixed prompt was used for every zero-shot model. Another prompt could raise or lower their scores.
• The Qwen and SmolLM2 model cards suggest sampling; greedy decoding was used for every model so the run is repeatable.
• google25 is this model's dev set, and google_fresh and google_big guided the v3.1 data (see the caveats above), which favours this model a little.

Per-set scores, google_big by kind and every paired test are in eval/results.md.

Phone files

Speeds are from an M1 Pro, CPU only, 4 threads, Homebrew llama.cpp. With 1 thread the 449 + 24 token case takes 982 ms (Q8_0) and 1,142 ms (Q4_K_M). On this CPU Q4_K_M is smaller but not faster. A mid-range phone should take roughly 0.5 to 0.9 s per question; that is an estimate, not a measurement. Q8_0 is the one to use unless the 29 MB difference matters: it gave the same answers under Homebrew llama.cpp and llama-cpp-python 0.3.36, while Q4_K_M changed 4 of 258 answers between the two.

Speed in tokens per second

Measured with llama.cpp on an Apple M1 Pro, CPU only (no GPU, no Metal), on a real 449-token test prompt plus a 24-token answer, median of 5 runs. "Reading" is the prompt pass, "writing" is generating the answer.

A typical test question has about 317 prompt tokens and a 5-token answer, so most of the time goes into reading the results: 168 ms per question with Q8_0 at 4 threads. These numbers come from Homebrew's prebuilt llama.cpp; in one check a llama.cpp built on this machine was about twice as fast, so treat them as a floor for this Mac.

Compared with Qwen3-0.6B

Same Mac, same llama.cpp build (Homebrew build 11146), llama-bench -p 449 -n 24 -r 5 -ngl 0 (all layers on the CPU), both files Q8_0. llama-bench measures writing from an empty context, so its writing numbers are higher than in the table above, where the answer is written after the 449-token prompt.

On this machine it answers about 5 times faster than Qwen3-0.6B and its file is 9 times smaller. Accuracy against Qwen3-0.6B is in Compared with other models.

Training

Architecture

Pretraining

Fine-tuning

2 epochs on real and synthetic search-reading rows. Every training row that asks a test question word for word was dropped, and so was every row on a google25, google_heldout or google_fresh topic. The rows added in v3.1 also left out google_big subjects and test questions asked in other words (see the caveats under Results).

Synthetic categories:

Limitations

• English only.
• It answers from the results you give it. It does not search and is not meant to answer from memory, so poor results give poor answers.
• About 20% of answers on the tests are wrong, and a wrong answer is more common than a not found. Show the source result next to the answer.
• not found works best on false-premise and future questions (41 of 50 caught). It is weaker when the results are about the right subject but leave out the asked fact. In a hand check of 6 made-up cases like that, it said not found once and the other five times gave a nearby time or number, such as 8 for "How many books does the Harbor Street library have?" from a result saying the library is open from 8 am to 8 pm.
• "Latest" and "current" answers depend on the dates in the results. The model does not know today's date; it picks the result that looks newest. Latest-version (6/14) and most-recent-winner (14/26) questions are its weakest kinds, and every other generative model tested did better on the 60 "latest" questions.
• 512 tokens of context, about five Google-length results.
• No chat and no instructions: one question in, a few words out, in the prompt format above only.

License and data notes

The weights, searchreader.py and eval/grade.py are under Apache-2.0 (see LICENSE).

MS MARCO's terms are for non-commercial research. If you plan commercial use, check whether that matters for you.

Author

Ali Hamza Kamboh (GitHub: ahkamboh, Hugging Face: ahkamboh).