0sint-event-embedder
A domain-specific sentence embedder for event identity in a noisy OSINT message stream.
Fine-tuned from sentence-transformers/all-MiniLM-L6-v2 on the
allenborochin/0sint dataset, and used in
production by the 0sint triage Space.
The task is not semantic similarity in general. It is: given a window of incoming Telegram-style messages, decide which of them are reporting the same real-world event. Two messages about different incidents in the same city, in the same hour, in the same register, are near-duplicates by general-purpose similarity and must be separated here.
Why it was fine-tuned
Off-the-shelf embeddings hit a hard ceiling on this task. Clustering an 8-hour window of the feed
(466 messages) with raw all-MiniLM-L6-v2 — agglomerative, cosine, average linkage @ 0.55 — gave
5 merged events, 16 fragmented events, ARI 0.574 against the dataset's ground-truth event_id
labels. A threshold sweep moved the two error types against each other but never fixed both.
The failure was not random. A measured audit of the dataset found 10,204 distinct-event pairs whose embedding centroids sat at cosine similarity ≥ 0.70, of which 25 also overlapped in time — the operationally dangerous cases, where a merge means an analyst sees one event instead of two.
The canonical case: an airport hostage situation (E0044) and a bus-station explosion (E0525),
same city, same time window, centroid similarity 0.788. The signal separating them exists in
the individual messages — but averaging into a centroid drowns it, and threshold clustering only
ever sees the average. Everything a general-purpose encoder keys on is shared: place, urgency
register, casualty language, channel voice. Only the event identity differs, and that is not in
the pretrained geometry.
How it was fine-tuned
- Objective: contrastive,
MultipleNegativesRankingLossover(anchor, positive, negative)triplets. Positives are two messages carrying the sameevent_id, capped at 30 pairs per event. - Hard negatives: drawn from each event's own measured confusable partners in the audit above, falling back to a random other event only when an event has no confusable partner. Random negatives are trivially separable and teach nothing about the cases that actually fail. In the production run, 13,864 of 15,293 triplets carry a hard negative.
- Hyperparameters: batch size 64, 2 epochs, learning rate 5e-5, 10% warmup, seed 42.
Two models came out of the pipeline. A proof model was trained with a held-out split, purely to validate that the method generalizes. After validation, the shipped production model was refit on all 1,102 events — standard practice, so no event is left unlearned. This repo hosts the production model.
Split discipline (proof model): the split is at the event level, never at the message level; splitting by message would put two reports of the same incident on both sides and leak the answer. 937 events train, 165 eval. Held out: 5 confusable pairs — the Beersheba pair plus same-type, cross-type, and one deliberately imperfect-separability case — and 155 random ordinary events. An explicit assertion checks that no eval-event text appears in any training triplet, and the random eval sample draws only from non-confusable events so the hard-negative pool stays intact.
Results
Generalization — the 5 held-out confusable pairs (proof model, never seen in training). These are the numbers that show the model learned a discrimination skill rather than memorizing events:
| pair (held out) | centroid similarity, lower is better | separability |
|---|---|---|
E0044/E0525 (Beersheba) | 0.788 → 0.583 | 1.00 → 1.00 |
E0135/E0543 | 0.803 → 0.700 | 1.00 → 1.00 |
E0110/E0423 | 0.789 → 0.642 | 1.00 → 1.00 |
E0105/E0523 | 0.771 → 0.676 | 0.93 → 1.00 |
E0003/E0294 | 0.777 → 0.619 | 1.00 → 1.00 |
Guard against collateral damage: mean within-event similarity on the 155 held-out ordinary events went 0.866 → 0.896. Pushing distinct events apart did not pull same-event messages apart with them.
Globally, same-event similarity rose 0.619 → 0.704 while cross-event similarity fell 0.219 → 0.194.
End-to-end — clustering an 8-hour feed window (466 messages). Note what each row measures:
| embeddings | config | merges | fragmented | ARI |
|---|---|---|---|---|
raw all-MiniLM-L6-v2 | average @ 0.55 | 5 | 16 | 0.574 |
| proof model (5 pairs held out) | average @ 0.50 | 1 | 10 | 0.868 |
| production model (this repo) | average @ 0.50 | 1 | 12 | 0.883 |
The 0.883 is the shipped model on the demo window, and the production model was refit on all events, so it is a system-performance number, not a held-out generalization score. The generalization claim rests on the held-out table above and on the proof model's 0.868, which was achieved with two of the events inside its single residual merge deliberately withheld from training.
Choice of base model
Small on purpose. In the Part 3 bake-off, MiniLM-L6 (22M), MPNet-base (110M) and BGE-small (33M)
scored ARI 0.281 / 0.287 / 0.240 on whole-dataset k-means against event_id — within noise of
each other — and MiniLM had the widest same-event vs. cross-event separation gap (0.332) while
being 5× smaller and 2.7× faster than MPNet. Fine-tuning the small model beats picking a bigger
one.
Use in production
The Space uses this model for three things:
- Window clustering —
AgglomerativeClustering(metric="cosine", linkage="average", distance_threshold=0.50). - Similar-past-event retrieval — cosine against event centroids, with a rolling archive cutoff so only events that ended before the window opened are eligible.
- Semantic feed search — free-text query against the full message set, on CPU.
Why the threshold is 0.50 and not 0.52. The two are statistically tied (ARI 0.883 vs 0.880, one merge each). The tiebreaker is what each merge contains: at 0.52 the Netanya military-base drone swarm — 11 messages, the most operationally significant event in the window — is absorbed whole into a blended three-city "drone activity" card. At 0.50 it keeps its own card, at the cost of two more fragmented events. Two honest annoyances beat burying the headline event.
Why not chase zero errors. One global threshold steers two opposing failure modes, and the sweep shows no configuration achieves both (0.40 eliminates merges at 19 fragmented; 0.60 minimizes fragmentation at 11 merges). The single residual merge is genuinely ambiguous eyewitness chatter — "might be a drone, might be a bird" — that humans would also disagree on. Pushing further would fit the generator's bookkeeping rather than a real skill. The system is built for residual uncertainty instead: low-corroboration singletons route to manual review, and the veracity score states its confidence rather than hiding it.
Precomputed embeddings for the full dataset are published alongside the data:
embeddings_finetuned_full.npy (this model), embeddings_finetuned_proof.npy (the proof model),
and embeddings_baseline.npy (raw MiniLM), so every number above is reproducible.
Reproducing
Fine-tuning code, the confusable-pairs audit, hard-negative mining, the split, and all evaluation
cells are in Final_Project_03b_finetuning.ipynb in the
dataset repo. The bake-off that selected
the base model is in Final_Project_03_embeddings.ipynb in the same repo.
Built for the Information Stream Triage & Situational Summary data-science capstone (Allen + Nadav, Reichman University). All training data is synthetic and fictional.