LFM2.5-Encoder-350M Corpus Cleaner

English | 中文

Try this model in HF space

English

Overview

LFM2.5-Encoder-350M Corpus Cleaner is a token-classification model built on LFM2.5-Encoder. Given noisy pretraining text, it labels each token as KEEP or DELETE, so you can strip noise in one pass and recover clean, usable text—without rewriting the document.

Note: This is v0.0.2 now. It may not cover every edge case. If you hit issues, please leave a comment or open an issue.

Why this model?

Raw pretraining corpora are often full of noise: garbled bytes, site chrome, ads, navigation, references, and other low-value spans. Common approaches fall short:

1. Hand-written rules — brittle and hard to maintain; they miss the long tail of ad and boilerplate patterns.
2. FastText / BERT quality scorers — can only drop low-scoring documents, not clean them into usable text.
3. LLM rewrite / cleaning — highest quality, but slow. At 200 tokens/s with 500 concurrent workers, cleaning 1 TB of text can take on the order of ~120 days.

Our approach

We train a token-classification model on LFM2.5-Encoder to mark noisy spans directly, then reconstruct clean text by keeping only KEEP tokens.

Training data: ~3M aligned (raw → cleaned) examples. Clean targets were produced with Qwen3.5-9B + a cleaning prompt on noisy web-scale source text.

Advantages

1. Orders of magnitude faster than generative LLMs — no GPU required for practical use. Measured throughput (this model, token classification):

   • CPU: ~3k tokens/s
   • GPU (single H20, PyTorch): ~160k tokens/s
   • vLLM can push throughput even higher — cleaning terabyte-scale corpora in days is realistic.

2. All-in-one, out of the box — no rule stack to tune. Works as both cleaning and filtering: removes ads, page shells, bibliographies, formatting junk, and other training-irrelevant content while preserving useful prose. Low-quality samples (e.g. pure ads) may be deleted entirely; post-clean length is a useful quality signal.

3. 8k context — reduces painful document chunking during preprocessing and inference.

Quick start

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

device = torch.device("cpu")
model_id = "yuyijiong/LFM2.5-Encoder-350M-corpus-cleaner"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
)
model.to(device)
model.eval()

text = "Your noisy pretraining document here ..."
enc = tokenizer(
    text,
    return_offsets_mapping=True,
    truncation=True,
    max_length=8192,
    return_tensors="pt",
)
offsets = enc.pop("offset_mapping")[0].tolist()
inputs = {k: v.to(device) for k, v in enc.items()}

with torch.inference_mode():
    logits = model(**inputs).logits[0]
    labels = logits.argmax(dim=-1).tolist()

# Reconstruct: keep spans where label == 0 (KEEP); merge overlapping offsets
ranges = [(s, e) for (s, e), lab in zip(offsets, labels) if lab == 0 and e > s]
ranges.sort()
merged = []
for s, e in ranges:
    if merged and s <= merged[-1][1]:
        merged[-1] = (merged[-1][0], max(merged[-1][1], e))
    else:
        merged.append((s, e))
cleaned = "".join(text[s:e] for s, e in merged)
print(cleaned)

Requires transformers with LFM2 support and trust_remote_code=True (custom modeling code is bundled).

Model details

Citation

@misc{LFM2.5-Encoder-350M-corpus-cleaner,
  author       = {Yijiong Yu},
  title        = {LFM2.5-Encoder-350M-corpus-cleaner},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Repository},
  howpublished = {\url{https://huggingface.co/yuyijiong/LFM2.5-Encoder-350M-corpus-cleane}},
}

中文

简介

LFM2.5-Encoder-350M Corpus Cleaner 是基于 LFM2.5-Encoder 的 Token 分类模型。对任意含噪声的预训练文本，模型逐 token 预测 保留（KEEP） 或 删除（DELETE），一键剔除噪声并还原可用正文，无需整段重写。

提示： 当前为 v0.0.2 版本，可能无法覆盖所有场景。使用中如有问题，欢迎评论或提 Issue。

动机

原始预训练数据往往包含大量噪声（乱码、网页目录、广告等）。常见清洗方式各有局限：

1. 大量手工规则 — 不够灵活，难以覆盖千奇百怪的广告与版式噪声。
2. FastText / BERT 质量打分 — 只能丢弃低分样本，无法把脏数据清洗成可用数据。
3. LLM 清洗或重写 — 质量最好，但速度慢。按约 200 tokens/s、500 并发估算，清洗 1 TB 数据约需 120 天。

我们的方案

训练基于 LFM2.5-Encoder 的 Token 分类模型，直接标出原文中的噪声片段，再拼接保留内容得到干净文本。

训练数据： 使用 Qwen3.5-9B + 清洗 prompt，在互联网原始语料上生成约 300 万 条「原始文本 → 清洗后文本」对齐样本。

优势

1. 速度远超生成式 LLM（可达数量级以上），无需 GPU 也可实用部署：

   • 纯 CPU： 约 3k tokens/s
   • 单卡 H20（纯 PyTorch）： 约 160k tokens/s
   • 配合 vLLM 还可更快 — 几天内洗完 TB 级数据 具备可行性。

2. All-in-one，开箱即用 — 无需维护规则库，同时承担清洗与筛选：可去除广告、网页壳、参考文献、格式性文本等对训练无用的内容，并保留正文；对极低质量样本（如纯广告）会整段删空，因此清洗后文本长度也可作为质量参考。

3. 支持 8k 上下文 — 减轻长文档切分带来的工程负担。

快速上手

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

device = torch.device("cpu")
model_id = "yuyijiong/LFM2.5-Encoder-350M-corpus-cleaner"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
)
model.to(device)
model.eval()

text = "这里放待清洗的预训练文本 ..."
enc = tokenizer(
    text,
    return_offsets_mapping=True,
    truncation=True,
    max_length=8192,
    return_tensors="pt",
)
offsets = enc.pop("offset_mapping")[0].tolist()
inputs = {k: v.to(device) for k, v in enc.items()}

with torch.inference_mode():
    logits = model(**inputs).logits[0]
    labels = logits.argmax(dim=-1).tolist()

# 按 offset 拼接 label==0（KEEP）的片段；合并重叠 offset（LFM 对部分汉字会切出重叠碎片）
ranges = [(s, e) for (s, e), lab in zip(offsets, labels) if lab == 0 and e > s]
ranges.sort()
merged = []
for s, e in ranges:
    if merged and s <= merged[-1][1]:
        merged[-1] = (merged[-1][0], max(merged[-1][1], e))
    else:
        merged.append((s, e))
cleaned = "".join(text[s:e] for s, e in merged)
print(cleaned)

需安装支持 LFM2 的 transformers，并设置 trust_remote_code=True（仓库内已包含自定义建模代码）。

模型信息

引用

@misc{LFM2.5-Encoder-350M-corpus-cleaner,
  author       = {Yijiong Yu},
  title        = {LFM2.5-Encoder-350M-corpus-cleaner},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Repository},
  howpublished = {\url{https://huggingface.co/yuyijiong/LFM2.5-Encoder-350M-corpus-cleane}},
}