FOF-80M (Focus On Facts)

FOF-80M is a specialized, 80M-parameter text embedding model designed to disentangle factual content from stylistic variation. It is fine-tuned from codefuse-ai/F2LLM-v2-80M to act as a soft factual fingerprint rather than a traditional semantic similarity model.

Standard embedding models often conflate "what is said" with "how it is said", scoring factually altered near-copies higher than stylistically different fact-preserving rewrites. FOF-80M was contrastively trained to invert this behavior: it aggressively collapses stylistic variation (formality, verbosity, rhetorical framing) into a high-similarity space, while forcefully pushing texts with subtle factual alterations (e.g., single entity swaps) apart.

Paper: Focus On The Facts: Stylistically Invariant and Factually Sensitive Text Embeddings

Dataset: MaxDevv/FOF-Bench

Usage

Note on Text Normalization: FOF-80M was trained on strictly normalized text to prevent shortcut learning via formatting artifacts (like PDF newlines). Apply the provided clean function to your text before encoding to achieve the documented performance.

from sentence_transformers import SentenceTransformer
import torch
import unicodedata
import re

# 1. Strict Text Normalization
def clean(text):
    if not isinstance(text, str): return ""
    text = unicodedata.normalize('NFKD', text)
    text = text.replace('”', '"').replace('“', '"').replace('’', "'").replace('‘', "'")
    text = text.encode('ascii', 'ignore').decode('ascii')
    text = re.sub(r'[\n\r\t]+', ' ', text)
    return re.sub(r'\s+', ' ', text).strip()

# 2. Load the Model
device = "cuda" if torch.cuda.is_available() else "cpu"
model = SentenceTransformer("MaxDevv/FOF-80M", trust_remote_code=True, device=device)

# Explicit 16-bit conversion is recommended for memory efficiency
if torch.cuda.is_available():
    model.bfloat16() if torch.cuda.is_bf16_supported() else model.half()

# 3. Example Texts
anchor = "We present ab initio calculations of the electronic energy loss of charged particles moving outside a magnesium surface."
altered_copy = "We present ab initio calculations of the electronic energy loss of charged particles moving outside a beryllium surface."
rewrite = "In this highly detailed scientific work, we put forward a set of comprehensive ab initio calculations pertaining to the electronic energy loss experienced by charged particles as they travel in close proximity to, but outside of, a magnesium surface."

# Clean and Encode
texts = [clean(t) for t in [anchor, altered_copy, rewrite]]
embeddings = model.encode(texts, normalize_embeddings=True)

# 4. Compare
sim_altered = (embeddings[0] @ embeddings[1].T).item()
sim_rewrite = (embeddings[0] @ embeddings[2].T).item()

print(f"Similarity to Fact-Altered Copy: {sim_altered:.4f}")
print(f"Similarity to Fact-Preserving Rewrite: {sim_rewrite:.4f}")

Evaluation & Performance

FOF-80M dramatically outperforms its base model at recognizing factual parity across stylistic gaps, but it trades off performance on tasks requiring tolerance for partial information.

Capabilities & Limitations

FOF-80M has a highly distinct, "hash-like" capability profile. It is not a general-purpose semantic embedding model.

Strengths:

• Stylistic Invariance: Easily ignores heavy rhetorical framing, sarcasm, verbosity, and style changes. It tolerates text expansions of up to 5x length without losing similarity, provided the facts match.
• Entity Substitution Detection: Highly sensitive to changes in specific proper nouns, numbers, and scientific entities.

Weaknesses / Blind Spots:

• The Omission Penalty: FOF-80M enforces strict bidirectional factual parity. If Text B omits a detail present in Text A, FOF-80M heavily penalizes the similarity. Because of this, do not use FOF-80M for standard Natural Language Inference (NLI) or Summarization evaluation, as both tasks inherently require valid omission/compression.
• Entity Overpowering: In very dense texts, a high overlap of matching proper nouns can sometimes overpower relational errors (e.g., swapping who did what to whom).
• Orthographic Brittleness: The model is sensitive to spelling corrections. It may interpret fixing a typo as deleting one entity and substituting an entirely new one.
• Cosine Compression: The model maps many pairs into a high-cosine region. Absolute cosine values are poorly calibrated; relative ordering and separation distance are the intended signals.

Citation

If you use this model, please cite the accompanying paper:

@article{adeola2026focus,
  title={Focus On The Facts: Stylistically Invariant and Factually Sensitive Text Embeddings},
  author={Adeola, Maximus},
  journal={arXiv preprint},
  year={2026}
}