NewsXLM: A Model for Multilingual News Attribute Extraction from HTML

This repository contains the NewsXLM downstream model for news page HTML node classification:

• Architecture: DOM-LM
• Base model: multilingual pretrained DOM-LM
• Fine-tuning dataset: NewsXLM

It labels HTML nodes as title, publication_date, text, author, tag, or none.

🚀 Usage

Installation

pip install -r requirements.txt

Prepare your HTML

html = '''
<html><body>
  <article>
    <h1 class="headline">Global trade volume rises 4%</h1>
    <time>September 11, 2026</time>
    <span class="author">Ivan Stepanov</span>
    <div class="body">
      <p>Global trade expanded in major markets.</p>
      <p>Growth was supported by stronger exports.</p>
    </div>
    <div class="tags"><span>Trade</span><span>Economics</span></div>
  </article>
  <footer><p>All rights reserved</p></footer>
</body></html>
'''

Attribute extraction

The extraction pipeline groups the results by attribute:

from transformers import pipeline

extractor = pipeline(
    "extraction",
    model="ispras-crawlers/newsxlm-domlm-ae",
    trust_remote_code=True,
)

extracted = extractor(html)
print(extracted)

Example result:

{
    "title": ["Global trade volume rises 4%"],
    "publication_date": ["September 11, 2026"],
    "text": ["Global trade expanded in major markets.", ...],
    "author": ["Ivan Stepanov"],
    "tag": ["Trade", "Economics"]
}

Node classification

The node-classification pipeline returns a prediction for each HTML node:

from transformers import pipeline

node_classifier = pipeline(
    "node-classification",
    model="ispras-crawlers/newsxlm-domlm-ae",
    trust_remote_code=True,
)

nodes = node_classifier(html)
print(nodes)

Example result:

{
    "/html/body/article/h1": {
        "pred": "title",
        "probs": {
            "title": 0.97,
            "publication_date": 0.01,
            "text": 0.01,
            "author": 0.00,
            "tag": 0.00,
            "none": 0.01
        },
        "text": "Global trade volume rises 4%"
    }, ...
}

Raw model output

import torch
from transformers import AutoModelForTokenClassification, AutoProcessor

repo_id = "ispras-crawlers/newsxlm-domlm-ae"

processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

inputs = processor(html, return_tensors="pt")
with torch.inference_mode():
    outputs = model(**inputs)

Citation

If you use NewsXLM in your research, please cite:

@article{newsxlm,
  author  = {Bedrin, P. A. and Varlamov, M. I. and Yatskov, A. K.},
  title   = {NewsXLM: A Multilingual Dataset and Model for Information Extraction from News Web Pages},
  journal = {Proceedings of ISP RAS},
  year    = {2026},
  volume  = {38},
  number  = {2},
  pages   = {149--164},
  doi     = {10.15514/ISPRAS-2026-38(2)-10}
}