NewsBERT_1800-1920-Temporal

A fine-tuned version of TextMachineProject/NewsBERT_1800-1920, conditioned on each item's publication year as a continuous input. A single model was trained across the full 1800-1920 span of the original NewsBERT_1800-1920's dataset, with each training example's year injected as a sinusoidal (Fourier) feature added to the token embeddings. Nearby years produce nearby embeddings by construction, so the model can be queried at any year in-range.

Architecture

• Base: TextMachineProject/NewsBERT_1800-1920 (BERT-base architecture)
• Time conditioning: a small ContinuousTimeEmbedding module maps a normalized year to a hidden_size-dimensional vector via sinusoidal features at 24 log-spaced frequencies, spanning:
  • lowest frequency: 1 cycle over the full 120-year corpus span
  • highest frequency: 1 cycle per 5 years
• This vector is added to every token's input embedding, then passed through a LayerNorm before entering the transformer stack (this re-normalization step was added after an earlier version without it let the time signal's magnitude dominate the token embeddings it was added to, which suppressed learning).

You need the accompanying continuous_time_embedding.py in this repo to load this model, as it is not loadable via a plain AutoModelForMaskedLM.from_pretrained(...) call, since the time-injection logic is outside the standard BERT forward pass.

Training data

9.28M items (text, year) from the LwM and HMD collections (see TextMachineProject/NewsBERT_1800-1920), spanning 1800-1920, split into overlapping 126-token windows (stride 96), i.e. 94.0M training windows.

Training procedure

• 1 epoch, batch size 256, linear LR decay, 500 warmup steps
• Learning rate: 2e-5 for the base model's existing (pretrained) layers, and a higher rate of 1e-4 for the ContinuousTimeEmbedding module since that module is new and starts from scratch
• bf16 mixed precision

Hardware / environment

This research utilised Queen Mary's Apocrita HPC facility, supported by QMUL Research-IT. http://doi.org/10.5281/zenodo.438045

• Single GPU per training job
• GPU: NVIDIA A100-PCIE-40GB (40GB VRAM)
• CUDA: 12.6 (as built into the torch version used)
• Python 3.11.7 (GCC 12.2.0 build)
• torch 2.11.0+cu126
• transformers 5.12.1

Example usage

Load the model and the companion module

import torch
import torch.nn.functional as F
import math
from huggingface_hub import hf_hub_download
import shutil, os

REPO_ID = "TextMachineProject/NewsBERT_1800-1920-Temporal"

# Download the companion module from the model repo and make it importable
py_path = hf_hub_download(repo_id=REPO_ID, filename="continuous_time_embedding.py")
shutil.copy(py_path, os.path.basename(py_path))

from continuous_time_embedding import load_continuous_time_model

tokenizer, model = load_continuous_time_model(REPO_ID)
device = next(model.parameters()).device

Basic functions to get top mask predictions and token probability

def predict_masked(sentence, year, top_k=5):
    enc = tokenizer(sentence, return_tensors="pt").to(device)
    mask_pos = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0]
    if len(mask_pos) == 0:
        raise ValueError(f"No [MASK] found in: {sentence!r}")
    mask_pos = mask_pos[0].item()

    embeddings_module = model.get_input_embeddings_module()
    tok_embeds = embeddings_module(enc["input_ids"])
    time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1)
    tok_embeds = model.post_inject_norm(tok_embeds + time_vec)

    out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"])
    probs = F.softmax(out.logits[0, mask_pos].float(), dim=-1)
    top = torch.topk(probs, top_k)
    return [(tokenizer.convert_ids_to_tokens([tid.item()])[0], p.item()) for tid, p in zip(top.indices, top.values)]


def get_target_word_probability(sentence, target_word, year):
    target_ids = tokenizer.encode(target_word, add_special_tokens=False)
    mask_str = " ".join([tokenizer.mask_token] * len(target_ids))
    masked_sentence = sentence.replace(target_word, mask_str, 1)

    enc = tokenizer(masked_sentence, return_tensors="pt").to(device)
    mask_positions = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0].tolist()

    embeddings_module = model.get_input_embeddings_module()
    tok_embeds = embeddings_module(enc["input_ids"])
    time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1)
    tok_embeds = model.post_inject_norm(tok_embeds + time_vec)

    out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"])

    logprob_sum = 0.0
    for pos, tgt_id in zip(mask_positions, target_ids):
        log_probs = F.log_softmax(out.logits[0, pos].float(), dim=-1)
        logprob_sum += log_probs[tgt_id].item()

    prob = math.exp(logprob_sum) if len(target_ids) == 1 else None
    return prob, logprob_sum

Check changes in top mask prediction or in token probability

### Example 1: fill-mask, top predictions at different years
sentence = "The [MASK] arrived on time."
for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]:
    preds = predict_masked(sentence, year, top_k=5)
    pred_str = ", ".join(f"{tok}({p:.3f})" for tok, p in preds)
    print(f"year={year}: {pred_str}")


# Example 2: probability of a specific word, across years
sentence = "The train arrived at the station on time"
for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]:
    prob, logprob_sum = get_target_word_probability(sentence, "train", year)
    if prob is not None:
        print(f"year={year}: P(train)={prob:.10f}  (logprob_sum={logprob_sum:.4f})")
    else:
        print(f"year={year}: logprob_sum={logprob_sum:.4f} (multi-subtoken word, no single probability)")

Example outputs:

year=1800: express(0.048), steamer(0.043), mail(0.031), train(0.018), news(0.014)
year=1810: express(0.064), steamer(0.044), mail(0.035), train(0.026), news(0.015)
year=1820: express(0.068), steamer(0.044), mail(0.035), train(0.032), news(0.017)
year=1830: express(0.078), steamer(0.043), mail(0.038), train(0.033), news(0.019)
year=1840: express(0.087), steamer(0.040), train(0.039), mail(0.038), news(0.023)
year=1850: express(0.081), train(0.042), steamer(0.039), mail(0.038), news(0.024)
year=1860: express(0.079), train(0.042), mail(0.038), steamer(0.035), news(0.029)
year=1870: express(0.080), train(0.050), mail(0.037), news(0.034), telegram(0.032)
year=1880: express(0.069), train(0.051), news(0.035), mail(0.035), telegram(0.032)
year=1890: express(0.063), train(0.052), news(0.039), telegram(0.034), mail(0.034)
year=1900: express(0.059), train(0.048), news(0.046), telegram(0.038), mail(0.033)
year=1910: express(0.054), news(0.052), train(0.047), telegram(0.039), mail(0.031)
year=1920: news(0.057), express(0.049), train(0.044), telegram(0.040), mail(0.030)

year=1800: P(train)=0.2596983756  (logprob_sum=-1.3482)
year=1810: P(train)=0.3239273912  (logprob_sum=-1.1272)
year=1820: P(train)=0.3619919056  (logprob_sum=-1.0161)
year=1830: P(train)=0.3642843769  (logprob_sum=-1.0098)
year=1840: P(train)=0.3824294963  (logprob_sum=-0.9612)
year=1850: P(train)=0.3897419142  (logprob_sum=-0.9423)
year=1860: P(train)=0.3831270530  (logprob_sum=-0.9594)
year=1870: P(train)=0.3918517235  (logprob_sum=-0.9369)
year=1880: P(train)=0.3810631289  (logprob_sum=-0.9648)
year=1890: P(train)=0.3701365410  (logprob_sum=-0.9939)
year=1900: P(train)=0.3599881482  (logprob_sum=-1.0217)
year=1910: P(train)=0.3460587446  (logprob_sum=-1.0611)
year=1920: P(train)=0.3430851355  (logprob_sum=-1.0698)

Check against known historical events

Visualize changes in probability of particular tokens and compare against historical events which should correspond to peak in probabilities (either one-off for a specific year or as the start of a continuous rise in probability thereafter, e.g. due to increased discourse around it in the newspapers).

event_tests = [
    ("The garrison at lucknow was relieved after a long siege.", "lucknow", 1857),
    ("President lincoln was assassinated.", "lincoln", 1865),
    ("After a long siege, the fortress of sebastopol finally fell.", "sebastopol", 1855),
    ("President mckinley was assassinated.", "mckinley", 1901),
    ("The queen has died.", "died", 1901)]

test_years = list(range(1800, 1921, 5))
fig, axes = plt.subplots(len(event_tests), 1, figsize=(11, 4.5 * len(EVENT_TESTS)))

for ax, (sentence, target_word, expected_year) in zip(axes, event_tests):
    masked_sentence = sentence.replace(target_word, tokenizer.mask_token, 1)
    n_subtokens = len(tokenizer.encode(target_word, add_special_tokens=False))
    is_single_token = n_subtokens == 1

    results = [get_target_word_probability(sentence, target_word, year) for year in test_years]
    plot_vals = [prob if is_single_token else logprob_sum for prob, logprob_sum in results]
    for year, (prob, logprob_sum) in zip(test_years, results):
        print(f"{sentence} | year={year}: " + (f"P({target_word})={prob:.6f}" if is_single_token else f"logprob_sum={logprob_sum:.4f}"))

    event_prob, event_logprob = get_target_word_probability(sentence, target_word, expected_year)
    event_val = event_prob if is_single_token else event_logprob

    ax.plot(test_years, plot_vals, marker="o")
    ax.scatter([expected_year], [event_val], marker="*", s=120, color="red", zorder=5, label=f"at {expected_year}")
    ax.axvline(expected_year, color="red", linestyle="--", alpha=0.4)
    ax.set_title(f"{'P' if is_single_token else 'log P'}({target_word!r}) across years\n\"{sentence}\"")
    ax.set_xlabel("queried year")
    ax.set_ylabel("probability" if is_single_token else "log probability")
    xticks = sorted(set(test_years) | {expected_year})
    ax.set_xticks(xticks)
    ax.set_xticklabels(xticks, rotation=45, ha="right")
    ax.legend()
    ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Example output:

Known limitations

• Trained for 1 epoch only. Treat results as preliminary.
• No year/decade-reweighting was used for this specific run, so the model's competence will not be uniform across the full 1800-1920 span and will reflect whatever the skewed training corpus's year distribution is.