PlotTwist Embedder (fine-tuned)
Authors: Raz Sarusi · Tomer Dariel
The retrieval model behind PlotTwist: it maps a short movie pitch to the closest loglines in a 10,000-row synthetic catalog. This repo hosts the fine-tuned embedder plus the FAISS index and the catalog, so the app loads everything at startup with no re-encoding.
1. Model selection — we compared three HF embedding models
Before any fine-tuning we embedded the full 10k catalog with three models and compared them on
retrieval quality, encode time, and vector size (full live table in Part3_Recommendation.ipynb):
| model | dim | notes | role |
|---|---|---|---|
sentence-transformers/all-MiniLM-L6-v2 | 384 | small & fast | speed baseline |
sentence-transformers/all-mpnet-base-v2 | 768 | strongest retrieval quality | winner |
BAAI/bge-small-en-v1.5 | 384 | strong small model | small-model contender |
Selection rule: primary = retrieval quality, ties broken by faster encoding / smaller size.
all-mpnet-base-v2 won on quality, so it became the base we fine-tuned.
2. Fine-tuning + a non-circular evaluation
v1's only metric — genre-consistency@k — is circular (data was generated conditioned on genre,
then genre agreement was measured). v2 replaces it with a genuine held-out task:
- A small LLM writes the short user-style pitch for a sample of loglines →
(pitch, logline)pairs. - recall@1 / @5 / @10: does a test pitch retrieve its true logline from the full 10k catalog?
- Fine-tune with
MultipleNegativesRankingLoss(in-batch negatives) on the(pitch, logline)pairs.
Results (held-out test = 600 pitches, catalog = 10k, train/test = 2400/600):
| metric | base | fine-tuned | Δ |
|---|---|---|---|
| recall@1 | 0.753 | 0.845 | +0.092 |
| recall@5 | 0.890 | 0.933 | +0.043 |
| recall@10 | 0.918 | 0.953 | +0.035 |
recall@1 improved +9.2 points (+12.2% relative) — a real, non-circular improvement.
Training configuration (the training "matrix")
| setting | value |
|---|---|
| base model | sentence-transformers/all-mpnet-base-v2 (768-d, cosine) |
| objective / loss | MultipleNegativesRankingLoss (scale 20.0, cos_sim) |
| training pairs | 2,400 (pitch → logline); held-out test 600 |
| epochs · batch size | 2 · 32 (≈ 150 optimization steps) |
| learning rate · schedule | 5e-5 · linear · AdamW (fused) |
| seed · training time | 42 · ≈ 1.9 min · Sentence-Transformers 5.6.0 |
Training loss vs. evaluation metric — and why the recall@k is the real proof.
MultipleNegativesRankingLoss is a contrastive ranking loss: its absolute value is not an accuracy
score, and the run was short (≈150 steps), so a per-step loss curve is not the meaningful signal here.
The rigorous evidence that fine-tuning beat the base model is the held-out recall@k lift in the
table above — measured on pitches the model never trained on. Over the 2 epochs the model learned to
place each pitch next to its true logline, moving recall@1 from 0.753 → 0.845 (base → fine-tuned).
3. Files
- fine-tuned
all-mpnet-base-v2weights (Sentence-Transformers format) plottwist.faiss— FAISSIndexFlatIPover L2-normalized embeddings (= cosine), 10k vectorsplottwist_catalog.parquet— the catalog rows aligned to the indexeval_recall_base_vs_ft.json— the numbers above
4. Usage
from sentence_transformers import SentenceTransformer
import faiss, pandas as pd
from huggingface_hub import hf_hub_download
model = SentenceTransformer("razsarusi/plottwist-embedder-ft")
idx = faiss.read_index(hf_hub_download("razsarusi/plottwist-embedder-ft", "plottwist.faiss"))
cat = pd.read_parquet(hf_hub_download("razsarusi/plottwist-embedder-ft", "plottwist_catalog.parquet"))
q = model.encode(["a lonely lighthouse keeper bargains with the sea"], normalize_embeddings=True)
D, I = idx.search(q.astype("float32"), 3)
print(cat.iloc[I[0]][["title","genre","logline"]])
Dataset: razsarusi/plottwist-movies ·
App: razsarusi/plottwist