Drop-one Soup · without Synthetic
Part of ClimateModernBERT, a family of climate-domain encoders obtained by continued pretraining of ModernBERT-Base on climate text.
| Repository | CMB-ClimateModernBERT/Merge_Soup_drop_S_LRD |
| Naming | corpora in A_S_F order, then the training stage: CX = Phase 1, CX_LRD = Phase 1 + Phase 2 |
| Corpora | Academic + Climate Web — {𝒜, ℱ} in the paper's notation |
| Training stage | Phase 2 · LRD Specialization (legacy suffix CX_LRD) |
| Base model | ModernBERT-Base, pre-LRD stable-phase checkpoint |
| Architecture | 150M parameters · 22 layers · hidden 768 · 12 heads · vocab 50,368 · 8,192-token context |
| Status | Paper |
| Merge method | Linear (uniform), 𝒮 removed |
| Merged from | CMB-ClimateModernBERT/A_CX_LRD, CMB-ClimateModernBERT/F_CX_LRD |
| Paper notation | Soup({𝒜, ℱ}) |
Figure 2. Excluding 𝒮 costs about 1.5 average F1 and improves some individual benchmarks.
Republished from sraj/Merge_Drop_SYN_FastText under a name that matches the paper's notation. The weights are identical; the original repository remains available.
Training data
Continued pretraining used Academic + Climate Web from a 6.42B-token climate corpus:
| Corpus | Tokens | Description | |
|---|---|---|---|
| 𝒜 | Academic | ~1.28B | Peer-reviewed journal articles across climate science, earth systems and energy economics; the ClimateNews archive 2000–2022; climate arXiv preprints; climate handbooks. |
| ℱ | Climate Web | ~5B | FineWeb-Edu filtered for climate relevance with a 166-term keyword filter followed by a FastText classifier. |
| 𝒮 | Synthetic | ~0.14B | LLM-generated climate text conditioned on in-domain seed excerpts, in three communication styles. |
Raw academic text is not redistributed: peer-reviewed articles are accessed under institutional publisher licenses, and news shards and handbooks were collected for non-commercial research use. The processing pipelines are released instead.
Training procedure
Two stages, following ModernBERT's own continued-pretraining recipe:
- Phase 1 — context extension. 3 epochs, constant LR 3e-4, global batch 576, sequence length 8,192, MLM masking 30%, StableAdamW, BF16.
- Phase 2 — LRD specialization. 3 further epochs on a
1 − √tdecay schedule from LR 3e-4 with final LR factor 1e-3.
4× NVIDIA A100, MosaicML Composer. Final checkpoints are converted to HF Transformers format.
Usage
ModernBERT is native to transformers from 4.48 onward, so no trust_remote_code
is required.
from transformers import AutoTokenizer, AutoModel
model_id = "CMB-ClimateModernBERT/Merge_Soup_drop_S_LRD"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)
inputs = tokenizer("Scope 1 and 2 emissions fell 12% against a 2019 baseline.", return_tensors="pt")
outputs = model(**inputs) # outputs.last_hidden_state -> (batch, seq, 768)
For a downstream task, load AutoModelForSequenceClassification and fine-tune. The
paper's recipe: LR 4e-5, effective batch 64, weight decay 0.01, up to 10 epochs with
early stopping on validation F1, BF16 with fused AdamW.
Evaluation
The manuscript reports no aggregate score for this checkpoint. It is released for provenance and follow-up work, not as a headline model.
Evaluated on nine climate NLP benchmarks: Climate Detection, Climate Specificity, Commitments & Actions, Climate Sentiment, Net Zero & Reduction, TCFD Recommendations, WFB Nature, WXImpactBench, and ClimRetrieve. Binary tasks and ClimRetrieve report positive-class F1; multi-class and multi-label tasks report macro-F1. Scores are the mean over three fine-tuning seeds under a single shared hyperparameter configuration.
Reference points from the paper: the ModernBERT-Base stable-phase baseline reaches 73.5 average F1, and ClimateBERT reaches 72.1 under the same protocol.
For general use, prefer CMB-ClimateModernBERT/Merge_Soup_LRD, the merged model that reaches 76.3 average F1.
Intended use
Research on climate NLP: encoding climate text, and fine-tuning for classification, multi-label tagging, and retrieval over corporate disclosures, policy documents, scientific literature and climate news.
Limitations
- English only, and built on a single encoder family (ModernBERT-Base).
- Current climate NLP benchmarks are largely sentence- or passage-level, so the model's long-context capacity is not fully exercised by the reported evaluation.
- Findings about corpus composition are demonstrated within climate NLP and should not be read as universal principles of domain adaptation.
- Synthetic training data has task-dependent effects: it helps taxonomy- and framework-driven tasks while degrading performance on tasks requiring finer-grained discourse and commitment understanding.
- The model is a masked language model, not an instruction-following system, and produces no calibrated factual guarantees about climate science.
Paper
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining. Preprint manuscript, currently under review — no venue, DOI or arXiv identifier yet, and no citation to give. The PDF is hosted in the project repository.
- Project website: https://climatemodernbert.github.io/ClimateModernBERT/
- Code and pipelines: https://github.com/ClimateModernBERT/ClimateModernBERT
- Full model catalog: https://github.com/ClimateModernBERT/ClimateModernBERT/blob/main/docs/model-inventory.md
- Naming guide: https://github.com/ClimateModernBERT/ClimateModernBERT/blob/main/docs/model-naming.md
License
Not yet set on this repository. The upstream base model is ModernBERT-Base; check its terms, and those of the underlying corpora, before redistributing. No license is asserted here on the maintainers' behalf.