FLAN-T5 Small
Google's instruction-tuned T5-small, exported for loom.cpp. Family 6: text in, text out, through an encoder read once and a KV-cached decoder that cross-attends to it -- the first encoder-decoder TEXT model and the first SentencePiece Unigram vocabulary in this collection.
This is a loom.cpp export: a single self-describing GGUF that carries its own graph topologies, tokenizer (if any) and driver script, produced by loom-exporter.
Original model
Exported from google/flan-t5-small. Weights are unmodified; this repo packages the same parameters into
loom.cpp's GGUF format.
License
apache-2.0, inherited from the base model above.
Language(s)
en, fr, ro, de, multilingual
Usage
Run it with loom-py -- loom-py-rt on PyPI:
pip install -U "loom-py-rt[hub]"
import loom
model = loom.Model.from_pretrained("loom-ai-org/flan-t5-small-loom")
# The instruction is part of the text -- this is an instruction-TUNED encoder-decoder, not a chat
# model, and it has no template to apply.
print(model.text2text.infer(
"translate English to German: The weather today is cold and rainy in the north.",
max_new_tokens=32))
# The same door, on the other tasks this checkpoint was tuned for:
print(model.text2text.infer("Answer the following question. What is the capital of France?"))
print(model.text2text.infer(
"summarize: The committee met on Tuesday to discuss the annual budget. After three hours of "
"debate, the members agreed to postpone the vote until the following month, citing incomplete "
"figures from the finance office.",
max_new_tokens=64))
The layer underneath
The call above is the high-level door: one per task, named for the modality pair it maps between, with
the windowing, sampling and assembly this model needs already applied. Under it, model.infer(...)
passes your arguments straight to the driver this GGUF embeds -- which is where you go for a knob the
door does not name.
model.driver_source prints that driver, including a header comment documenting every argument it
accepts for this model, and is the authority on it. See loom-py for the API and
loom.cpp for what the engine does between the two.
Known limitations
It is 80M parameters, and it answers like it. flan-t5-small is the smallest member of the flan collection: it follows an instruction's SHAPE reliably and is often wrong about the content. Asked for the capital of France it says london, and transformers says the same thing on the same checkpoint -- that is the model, not the export. Use it to try the interface, and step up to flan-t5-base or larger for answers you intend to keep.
The instruction is part of the text. There is no chat template and no system turn; prompts look like translate English to German: ... or Answer the following question. ..., which is how the checkpoint was tuned. A bare sentence with no instruction is out of distribution and usually comes back as a fragment of itself.
Greedy, not beam search. The upstream config.json names 4 beams for its translation and summarization presets. The export decodes one token at a time from the argmax (or from the sampler, if you pass temperature=), so translations here are the greedy path through the same model rather than what the published presets would give.
Sequences are capped at 512 tokens on each side. T5 has no learned position table, so nothing in the file stops a longer source -- but its relative-position buckets saturate at 128 tokens of distance and the KV cache is built at 512, which is the length this export is honest about.
Files
flan-t5-small.gguf-- the model, exported with loom-exporter.