Semantic chunker (ModernBERT-base)

Predicts, for each sentence, whether a topic boundary follows it. Intended as a drop-in chunker for RAG pipelines.

What is different here

• 2048-token training context. The closest existing model trains at 1024. Median Wikipedia article is ~980 tokens, so 2048 keeps ~82% of documents in a single window instead of ~53%.
• Supervision at sentence-final tokens only. Labelling every token puts the positive rate near 1% and the model collapses toward never predicting a split. Scoring only sentence-final positions raises it to ~12%.
• Trained on Wikipedia section structure, not book paragraphing.

Results

Both models are ModernBERT-base token taggers sharing a tokenizer, so they are scored at identical positions: P(boundary) read at each sentence's final token. Threshold swept 0.05–0.95; the best operating point is reported.

wiki727k test — in-distribution for this model

PubMed-RCT — out-of-distribution for both models

Non-Wikipedia scientific prose with human-assigned section labels. Boundary rate is 35.2% here versus ~11.7% on Wikipedia, so scores are not comparable across the two tables — only across models within one table.

Retrieval — does better boundary detection actually help?

Macro-average over 3 corpora (gov_report, qmsum, stackoverflow), embedder BAAI/bge-small-en-v1.5, identical 512-token cap on every strategy so chunk size cannot confound the comparison.

Per corpus (nDCG@10):

Reading these numbers honestly

baseline:every-k splits every k sentences with k chosen to maximise its own score — a model-free floor with oracle tuning in its favour. The wiki727k comparison flatters this model: Wikipedia section boundaries are its training distribution and are out-of-distribution for chonky, which trained on BookCorpus paragraphs. The PubMed table is the fair comparison.

Better boundaries did not produce a uniform retrieval win, and the retrieval table above should be read before adopting this. That result is consistent with the published critique of semantic chunking, and it is reported here rather than omitted. Boundary F1 and retrieval quality are different things; this model is much better at the first and situationally better at the second.

When this helps, and when it does not

Use it for long documents whose topic genuinely shifts partway through — transcripts, reports, articles, manuals. That is where fixed-size splitting cuts through the middle of an idea and where boundary detection pays.

Do not use it for corpora whose documents are already shorter than your embedder's window. Splitting a 150-token document into two 75-token fragments makes retrieval worse, not better, no matter how correct the boundary is. On such corpora fixed-size chunking is competitive, faster, and has no dependencies — use that instead.

min_chunk_tokens is a corpus-dependent knob, not a default. Setting it to 256 recovered most of the short-document deficit (+0.019 nDCG@10 on stackoverflow) but cost almost exactly as much on long transcripts (−0.020 on qmsum), by merging away the boundaries that made the model useful there. Set it high for short documents, leave it at 0 for long ones, and measure on your own corpus rather than trusting either default.

Cost. Chunking with this model is roughly one to two orders of magnitude slower than a fixed-size splitter (per-strategy wall time is recorded in retrieval_eval.json). On corpora where it does not win, that cost buys nothing.

Usage

Nothing to download by hand in any of these — the weights resolve from this repo on first use.

With transformers alone

The custom pipeline is registered in this repo's config, so this needs no install beyond transformers:

from transformers import pipeline

chunk = pipeline("semantic-chunking", model="0xKitkat/semantic-chunker-modernbert-base",
                 trust_remote_code=True)

for c in chunk(document_text):
    print(c["n_tokens"], c["boundary_score"], c["text"][:80])

Pass return_text=True for plain strings, or any of threshold, max_chunk_tokens, min_chunk_tokens to tune it.

With the pip package

pip install "boundary-chunker[onnx]"    # CPU, no torch
pip install "boundary-chunker[torch]"   # GPU

from boundary_chunker import SemanticChunker

chunker = SemanticChunker(backend="onnx")     # or backend="torch"
for c in chunker.split(document_text):
    print(c.n_tokens, c.text[:80])

There is a CLI too:

boundary-chunk --backend onnx --file report.txt --json

Straight from this repo, no install

hf download 0xKitkat/semantic-chunker-modernbert-base chunk.py --local-dir .

from chunk import SemanticChunker

chunker = SemanticChunker(threshold=0.5, max_chunk_tokens=512)

max_chunk_tokens force-splits over-long chunks at the model's lowest-confidence interior boundary rather than at an arbitrary offset.

CPU-only, without torch

An ONNX export lives in onnx/ and needs onnxruntime instead of the full torch stack — roughly 40MB of dependencies rather than a CUDA install. It is self-contained, with its own config and tokenizer beside the graph, and backend="onnx" fetches it for you.

The export is verified against the torch model on held-out wiki727k documents: identical chunk boundaries on 24/24 documents (739–65,468 chars, including multi-window ones), and max |P(boundary) delta| 8.1e-06 on padded batches. The pipeline is likewise checked against the direct API — identical boundaries on 12/12 documents — so all three entry points above give the same answer. Both checks are reproducible with onnx_parity.py in the training repo.

Training

Section titles are dropped from the training text: with headings present the task partly degenerates into "a heading follows", which does not transfer to the unformatted prose a chunker sees in production.

Limitations

• English only.
• Trained on encyclopedic prose; conversational transcripts, code, and tabular documents are out of distribution.
• Boundary F1 is a proxy. If your goal is retrieval quality, measure retrieval.