ManufactuBERT

Specialized RoBERTa-base model continually pretrained on a high-quality corpus specifically curated for the manufacturing domain. By employing a rigorous data selection and deduplication pipeline, ManufactuBERT achieves state-of-the-art performance on industrial NLP tasks while being more efficient to train than standard domain-adapted models.

Model Details

• Developed by: Robin Armingaud and Romaric Besançon (Université Paris-Saclay, CEA, List)
• Language: English
• Domain: Manufacturing and Industrial Engineering

Training Data

The first ManufactuBERT model was continually pretrained on Manu-FineWeb, a 10-billion-token subset of the FineWeb dataset. The final model version used a further refined version (~2 billion tokens) created through a multi-stage process using MinHash and SemDeDup to remove approximately 80% of redundant data. This process accelerated convergence by approximately 33% compared to the non-deduplicated version.

Named Entity Recognition (NER)

Sentence Classification & Relation Extraction

General Domain Evaluation (GLUE)

The model preserves general language understanding capabilities significantly better than other domain-specific models.

Citation

If you use ManufactuBERT in your research, please cite:

@misc{armingaud2025manufactubertefficientcontinualpretraining,
      title={ManufactuBERT: Efficient Continual Pretraining for Manufacturing}, 
      author={Robin Armingaud and Romaric Besançon},
      year={2025},
      eprint={2511.05135},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.05135}, 
}