pe-graft-lfm — PE encoder grafted onto LFM2.5-2.6B (projector-only)
graft.pt (72MB) holds a trained 2-layer MLP projector (1024 -> 2048 x 4
prefix tokens) mapping frozen MC7ever/pe-single-L14 embeddings into frozen
LiquidAI/LFM2.5-2.6B input space. Both giants stay frozen; only the
projector trained (COCO-train 4000 pairs x 2 epochs, local Apple-Silicon MPS).
Run
git clone https://github.com/shadyuwugurl/pe-single-L14
python3 graft_generate.py --graft ./pe-graft-lfm --image photo.jpg
python3 graft_generate.py --graft ./pe-graft-lfm --audio clip.wav --sr 44100
The script downloads both base models automatically on first run.
Samples (honest)
- moped guy -> "A man riding a bike on the road." CORRECT
- Wii game ("two smiling young people playing Wii") -> "Two girls are in a park and one is taking pictures of the other." WRONG scene, right headcount
- restaurant meal -> "A coffee cup sits on a table with other items." gist-level
Limits
Projector-only with 4 prefix tokens: fluent, gist-level, not sharp. Path to sharp: more prefix tokens (4->16), LLM LoRA, audio prefix training, more data. Audio prefix path is wired but essentially untrained (v1 trained image->text only).