RecToM Causal-ToM Reasoning-Aware Retriever (Pilot)
Motivation
RecToM is a dialogue-based Theory-of-Mind movie-recommendation benchmark. A prior blind multi-judge
audit (project docs127) found that off-the-shelf semantic retrieval (all-MiniLM-L6-v2) returns
causally incompatible cases as its top-1 hit 74-82% of the time, despite high cosine similarity --
i.e. semantic similarity between dialogues is not the same as similarity in the causal Theory-of-Mind
reasoning structure required to answer the question. This model is a pilot fine-tune testing whether
structure-aware supervision closes that gap, following RAR-B (Xiao et al., 2024) and RaDeR
(Das et al., 2025), adapted to RecToM's causal ToM setting.
Supervision signal
Training labels come from intervention-validated causal paths (project docs126): each causal edge was tested via a real counterfactual/control intervention on the answering model, not just claimed by an LLM. Edges whose intervention outcome was REJECTED or CONFOUNDED were dropped before deriving each item's ordered relation_type sequence.
Training-data construction
- Source pool: 80-item leakage-clean RecToM Case Bank, filtered to 56 items with a non-empty validated relation-type sequence.
- Same-task pairs only, labeled deterministically (no LLM call):
- MATCH: identical relation-type sequence -- 49 pairs.
- PARTIAL: one sequence is a prefix of the other -- 81 pairs, held out, NOT used in this training run.
- MISMATCH: sequences diverge. Hard negatives = MISMATCH pairs with frozen-baseline cosine similarity >= 0.75 (61 pairs, all kept); remaining MISMATCH pairs subsampled to 127.
- Input text = Dialogue History + Question + Choices ONLY. Causal paths are supervision-only.
Leakage controls
- Training pool never includes any item from RecToM's 150-item evaluation set (docs124-127).
- Train/validation split is group-disjoint by dialogue_id, including 2 internal collision groups found and kept intact.
- 45 items in train, 11 in validation.
Backbone and training
- Backbone:
BAAI/bge-small-en-v1.5 - Loss: ContrastiveLoss (binary MATCH/MISMATCH + mined hard negatives)
- Epochs: 10, batch size: 8, learning rate: 2e-05, best epoch: 1
- Seed: 42
Retrieval metrics (validation split, group-disjoint from training)
| Model | MATCH@1 | MATCH@3 | Hit@1 | Hit@3 | Hit@5 | MRR |
|---|---|---|---|---|---|---|
| Frozen all-MiniLM-L6-v2 (project baseline) | 0.222 | 0.222 | 0.222 | 0.222 | 0.222 | 0.222 |
| Frozen BAAI/bge-small-en-v1.5 | 0.222 | 0.222 | 0.222 | 0.222 | 0.222 | 0.222 |
| This model (fine-tuned) | 0.222 | 0.222 | 0.222 | 0.222 | 0.222 | 0.222 |
Important limitation
This is a pilot trained on a small (n=156 training pairs) intervention-validated Case Bank. It has NOT been evaluated on downstream RecToM answer-generation accuracy yet (Stage 2, pending). Metrics above measure representation/retrieval quality only, on a held-out validation split of the same small Case Bank -- not RecToM's official 150-item evaluation set, which this model has never seen in any form.