DataOrchestra — Noise Pruning (NP) Model

This is the Noise Pruning (NP) tool model of DataOrchestra. It is the lightest of the three cleaning stages: given a document chunk, it emits whole-line deletion operations that strip line-level noise (site navigation, ads, share bars, boilerplate, catalog metadata, etc.) without rewriting any surviving text. It is used together with the orchestrator and the SR/PA rewriter, but can also be run on its own.

Model Details

The model is trained ProX-style (following ProX): unlike ProX/RefineX, it keeps only whole-line removal and drops in-line substring edits, which simplifies the duty of this small tool model.

Usage

The model sees the chunk with a 0-indexed, zero-padded [NNN] prefix on every line, wrapped in [DOC] / [/DOC], under a one-line system prompt. It responds with remove_lines(start, end) calls (both ends inclusive, line-indices referring to the [NNN] prefixes) or the sentinel skip() when nothing should be removed. You line-number the chunk, parse the ops, and delete those lines.

import re
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "DataOrchestra/NP-0.6B"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto", device_map="auto")

SYSTEM_PROMPT = "You are an excellent noise pruning model for pretraining data cleaning."
REMOVE_LINES_RE = re.compile(r"remove_lines\s*\(\s*(\d+)\s*,\s*(\d+)\s*\)")


def prune(chunk: str) -> str:
    lines = chunk.split("\n")
    # NP sees a 0-indexed [NNN] prefix on every line (add_line_numbers()).
    numbered = "\n".join(f"[{i:03d}] {line}" for i, line in enumerate(lines))
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"[DOC]\n{numbered}\n[/DOC]"},   # wrap_doc()
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,           # NP runs non-thinking
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    generated = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=False,                 # greedy: temperature 0.0 / top_p 1.0
    )
    response = tokenizer.decode(
        generated[0][inputs.input_ids.shape[1]:], skip_special_tokens=True
    )

    # Parse remove_lines(start, end); no ops (e.g. skip()) -> keep the chunk as-is.
    remove = set()
    for start, end in REMOVE_LINES_RE.findall(response):
        for i in range(int(start), int(end) + 1):
            if 0 <= i < len(lines):
                remove.add(i)
    return "\n".join(line for i, line in enumerate(lines) if i not in remove)


chunk = (
    "Home | About | Contact\n"
    "The French Revolution began in 1789 and reshaped European politics.\n"
    "Share this on Facebook | Twitter\n"
    "It led to the rise of Napoleon Bonaparte."
)
print(prune(chunk))
# -> keeps the two content lines, drops the nav header and the share bar

Serving with vLLM

For high-throughput curation, serve the model with an OpenAI-compatible endpoint. Note the chunk must still be line-numbered by the caller before sending:

vllm serve DataOrchestra/NP-0.6B --served-model-name DataOrchestra-NP-0.6B --trust-remote-code

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="DataOrchestra-NP-0.6B",
    messages=[
        {"role": "system", "content": "You are an excellent noise pruning model for pretraining data cleaning."},
        {"role": "user", "content": "[DOC]\n[000] Home | About | Contact\n[001] <your content line>\n[/DOC]"},
    ],
    temperature=0.0,
    max_tokens=1024,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)

Output Format

The model emits one operation per line:

remove_lines(0, 0)
remove_lines(2, 2)

• remove_lines(start, end) — delete lines start through end inclusive, where indices refer to the [NNN] prefixes of the input. Only whole-line removal is supported.
• skip() (or an empty / op-free response) — remove nothing; the chunk is kept unchanged.

Apply the ops by deleting the referenced lines (process them bottom-up, or collect all removed indices first, so earlier deletions do not shift later indices).

Citation

If you find this work useful, please cite:

@article{dataorchestra2026,
  title   = {DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data},
  author  = {Huang, Zhen and Wang, Yikun and Xia, Shijie and Liu, Pengfei},
  year    = {2026},
  journal = {arXiv preprint arXiv:2607.24717}
}