CrosswordSpace++ dual encoder (MPNet-base ADE)

The first-stage retriever of CrosswordSpace++. It is an Asymmetric Dual Encoder (ADE) for Italian crossword clue answering, trained with contrastive learning on the EVALITA 2026 CruciverbIT training set.

Clues and candidate answers are projected into a shared 768-dimensional space, so candidates can be retrieved with FAISS inner-product search. In the CrosswordSpace++ pipeline, the top-100 length-filtered candidates from this model are reranked by the cross-encoder cruciverb-it/crosswordspacepp-reranker. The two scores are then combined.

Code: https://github.com/snizio/crosswordspacepp

Model Description

• Architecture: Asymmetric Dual Encoder with two separate XLM-RoBERTa encoders (one for clues, one for answers), mean pooling, a shared LayerNorm and a shared linear projection head. It is the same architecture as cruciverb-it/crossword-space-mpnet-base-ade.
• Base encoder: sentence-transformers/paraphrase-multilingual-mpnet-base-v2
• Training objective: symmetric contrastive loss (InfoNCE) with in-batch hard negative mining (hard-negative fraction decayed linearly from 0.8 to 0.2) and a learnable temperature
• Training data: EVALITA 2026 CruciverbIT Task 1 train split (374,766 clue-answer pairs), batch size 256, 10 epochs; the checkpoint with the lowest validation loss (step 13,899) is released
• Max length: 64 tokens for clues, 16 tokens for answers
• Embedding dimension: 768

Usage

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

repo = "cruciverb-it/crosswordspacepp-dualencoder"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(repo)

clues = ["Giorni di metà mese nell'antica Roma", "Possono sostituire le mostrine"]
answers = ["idi", "alamari", "mese"]

clue_enc = tokenizer(clues, padding=True, truncation=True, max_length=64, return_tensors="pt")
ans_enc = tokenizer(answers, padding=True, truncation=True, max_length=16, return_tensors="pt")

with torch.no_grad():
    clue_emb, ans_emb = model(
        def_input_ids=clue_enc["input_ids"],
        def_attention_mask=clue_enc["attention_mask"],
        ans_input_ids=ans_enc["input_ids"],
        ans_attention_mask=ans_enc["attention_mask"],
    )

similarity = F.normalize(clue_emb, dim=-1) @ F.normalize(ans_emb, dim=-1).T
print(similarity)
# tensor([[ 0.8244,  0.0907,  0.3886],
#         [ 0.0270,  0.7899, -0.0285]])

To encode only clues (or only answers), call a single tower and apply the shared head:

out = model.encoder_def(input_ids=..., attention_mask=...)       # model.encoder_ans for answers
emb = model.projection(model.layer_norm(mean_pooling(out.last_hidden_state, attention_mask)))

mean_pooling is defined in model.py in this repository.

Evaluation

Results on the EVALITA 2026 CruciverbIT Task 1 test set (20,821 clues). The index is restricted to answers of the expected length.

Citation

TBD

License

CC BY 4.0