SpanBERT-Coreference: Coreference Resolution for Portuguese Municipal Meeting Minutes

This model consists of a fine-tuned SpanBERT for coreference resolution in minutes of Portuguese municipal meetings, identifying when different expressions in a document refer to the same real-world entity.

Model Description

The SpanBERT-PT-Coreference uses a mention-ranking architecture (Lee et al., 2017) on top of a SpanBERT encoder pre-trained in Portuguese, specifically optimized for the linguistic and formal structure of administrative minutes in Portugal. The model detects coreferent mention clusters across 17 entity categories, enabling consistent pseudonymization of sensitive personal data throughout long documents.

Key Features

• 🏛️ Specialized for Municipal Minutes: Fine-tuned on authentic Portuguese council meeting minutes
• 🔗 Coreference-Aware Pseudonymization: Links multiple mentions of the same entity, ensuring consistent ID assignment across the document
• 📄 Long Document Support: Handles documents of up to ~28,000 tokens via sliding window encoding with stride overlap
• ⚙️ Mention-Ranking Architecture: Scores all antecedent candidates for each mention span and clusters them via Union-Find

Model Details

• Base Model: SpanBERT (Portuguese continued pre-training)
• Architecture: Mention-ranking coreference (Lee et al., 2017)
• Parameters: ~110M (encoder) + ~25M (coreference heads)
• Max Window Size: 512 tokens (stride: 128)
• Fine-tuning Dataset: 120 Portuguese meeting minutes (6 municipalities, ~1.3M tokens)
• Evaluation Metrics: MUC, B³, CEAF-e, CoNLL F1, LEA
• Training Framework: PyTorch + Transformers

Entity Types

The model resolves coreference across 17 entity categories:

Cluster Distribution (Training Set)

How It Works

The model uses a two-stage pipeline. First, entity spans are identified by a NER model (liaad/CitiLink-XLMR-Anonymization-pt). Then, the coreference model groups spans that refer to the same entity, assigning consistent IDs throughout the document.

INPUT:

João Paulo Rosinha Daniel apresentou o requerimento n.º 2683/20020627.
O pedido de João Paulo, referente ao processo 2683/20020627, foi aprovado.

OUTPUT:

<NAME-1> apresentou o requerimento n.º <ADMIN-1>.
O pedido de <NAME-1>, referente ao processo <ADMIN-1>, foi aprovado.

Architecture Details

The mention-ranking model represents each span as:

span_repr = [h_start ; h_end ; head_attn ; width_emb]
           = [768 ; 768 ; 768 ; 20] = 2324 dims

Antecedent scoring uses:

score(i,j) = mention_score(i) + mention_score(j) + antecedent_score([vi; vj; vi*vj; dist_emb])

Clusters are built via Union-Find, restricted to spans of the same NER class and scores above threshold.

Results

Overall Performance (Test Set)

Usage

Quick Start

The simplest way to use the model:

import torch
import json
from transformers import AutoTokenizer, AutoModel
from huggingface_hub import hf_hub_download

model_id = "inesctec/CitiLink-SpanBERT-Coreference-pt"

# Load tokenizer and encoder
tokenizer = AutoTokenizer.from_pretrained(model_id)
encoder   = AutoModel.from_pretrained(model_id)

# Load coreference heads and config
cabecas_path   = hf_hub_download(repo_id=model_id, filename="cabecas_coref.pt")
coref_cfg_path = hf_hub_download(repo_id=model_id, filename="coref_config.json")

with open(coref_cfg_path) as f:
    coref_cfg = json.load(f)

cabecas = torch.load(cabecas_path, map_location="cpu", weights_only=False)

# Input text (already tokenized by whitespace)
tokens = [
    "João", "Paulo", "Rosinha", "Daniel", "apresentou", "o",
    "requerimento", "n.º", "2683/20020627", ".",
    "O", "pedido", "de", "João", "Paulo", "foi", "aprovado",
    "ao", "abrigo", "do", "processo", "2683/20020627", "."
]

# NER spans already extracted by a NER model — (start, end, class)
ner_spans = [
    (0,  3,  "NAME"),   # "João Paulo Rosinha Daniel"
    (8,  8,  "ADMIN"),  # "2683/20020627"
    (13, 14, "NAME"),   # "João Paulo"
    (21, 21, "ADMIN"),  # "2683/20020627" (segunda ocorrência)
]

# Tokenize for encoder
inputs = tokenizer(tokens, is_split_into_words=True, return_tensors="pt")

# Run encoder
with torch.no_grad():
    outputs = encoder(**inputs)

print("Coreference model loaded successfully.")
print(f"Hidden states shape: {outputs.last_hidden_state.shape}")

Limitations

• Domain Specificity: Best performance on administrative/governmental meeting minutes
• Language: Trained exclusively on European Portuguese (PT-PT)
• Window Size: Processes documents via 512-token sliding windows; very long-range coreference (>5,000 tokens apart) may be missed
• NER Dependency: Requires entity spans from a NER model as input — does not perform span detection independently
• Same-class Restriction: Only links mentions of the same NER entity class (e.g., NAME to NAME, never NAME to POSITION)
• Dataset Size: Fine-tuned on 61 training documents; performance on highly specific sub-domains may vary

Version: 1.0
Last Updated: 2026-06-25

License

This project uses a custom dual-license based on AGPL v3.

See the full license terms here: LICENSE