Fiorillo reader v1

Abbreviations: MIMIC is the Medical Information Mart for Intensive Care, a credentialed database of clinical records; NLM is the U.S. National Library of Medicine; PMC is PubMed Central; UCI is the University of California, Irvine, whose Machine Learning Repository holds the readmission data; AUROC is the area under the receiver operating characteristic curve; CPU is central processing unit; NRD is the Nationwide Readmissions Database; NIS is the National Inpatient Sample; NHAMCS is the National Hospital Ambulatory Medical Care Survey.

Fiorillo reader v1 is a biomedical masked-language model with 395,881,664 parameters and the ModernBERT-large architecture. I made it from Laya (convaiinnovations/laya): I took Laya's encoder, gave it ModernBERT-large's masked-language-model head, and trained it further on 2,000,158,720 tokens of PubMed titles and abstracts. I added no MIMIC or other credentialed clinical data: its further training read PubMed abstracts only, and neither Laya's card nor ModernBERT-large's lists clinical notes among their training data (their full training text is not published). It is the reader I trained for Fiorillo v0.2, a medical typed-decision model. I release it on its own because the decision layer built on it did not clear the bar I set before training (see What it did not achieve). In a comparison I computed after the results, the decision layer on reader v1 had lower log loss than the same decision layer on Laya's own reader on both questions and higher PubMedQA accuracy, with no difference detected in readmission AUROC; the comparison fixed before training, at 500 million tokens, showed the same pattern. It is for research and for fine-tuning. It is not a medical device and must not be used to make clinical decisions.

The openly licensed biomedical ModernBERT-large readers I found in Hugging Face checks of 27 and 28 September 2026 were each trained on tens of billions of tokens that include clinical notes from MIMIC: BioClinical ModernBERT-large (thomas-sounack/BioClinical-ModernBERT-large, MIT license; 53.5 billion tokens including MIMIC-III and MIMIC-IV notes), Bio ModernBERT-large (thomas-sounack/Bio-ModernBERT-large, MIT license; its first phase included the same clinical data) and ModernBERT-bio-large (almanach/ModernBERT-bio-large, Apache 2.0; 55 billion tokens, a fifth of them MIMIC clinical notes). The other biomedical models in the ModernBERT family that those checks found are base size, gated, without a license, in another language or without public weights; one of them, Clinical ModernBERT (Simonlee711/Clinical_ModernBERT, MIT license, base size), also includes MIMIC-IV notes.

It carries the name of Fiorillo v0.1, this project's decision model.

How to use

from transformers import pipeline

fill = pipeline("fill-mask", model="johanneli/fiorillo-reader-v1")
for text in ["Metformin is a first-line treatment for type 2 [MASK].",
             "The patient was treated with intravenous [MASK] for bacterial pneumonia.",
             "Aspirin irreversibly inhibits platelet [MASK]."]:
    print(text, [(p["token_str"].strip(), round(p["score"], 3)) for p in fill(text, top_k=3)])

On a CPU, with this repository's files, it printed (29 September 2026, 01:34 UTC; transformers 5.17.0, torch 2.14.0+cu130):

Metformin is a first-line treatment for type 2 [MASK]. [('diabetes', 0.991), ('DM', 0.003), ('Diabetes', 0.002)]
The patient was treated with intravenous [MASK] for bacterial pneumonia. [('antibiotics', 0.958), ('therapy', 0.01), ('immunoglobulin', 0.009)]
Aspirin irreversibly inhibits platelet [MASK]. [('aggregation', 0.794), ('activation', 0.089), ('function', 0.049)]

For classification, load it with AutoModelForSequenceClassification and fine-tune it; for sentence vectors, load it with AutoModel and pool the last hidden states. Its encoder has the same architecture and tokenizer as Laya's, so it can also take the place of Laya's encoder in a Laya decision model, which is how I used it. It was trained on blocks of 1,024 tokens. The architecture accepts up to 8,192 tokens, as ModernBERT-large does, but longer inputs were not tested.

Training data

• 7,000,000 PubMed titles and abstracts (2,127,280,106 tokens) from NLM's 2026 annual baseline, read from 360 of the baseline's 1,334 files in a random order fixed by a seed (20260927).
• I removed every article in PubMedQA (273,518 distinct PubMed identifiers across its three sets), which took out 74,437 abstracts, and 272 more abstracts that shared a run of 13 words with the contexts or long answer of one of PubMedQA's 1,000 expert-labeled items.
• I held out 3,331 abstracts (1,000 blocks of 1,024 tokens), used only to measure loss.
• Not used: full-text articles, clinical notes, and any MIMIC, NRD, NIS or NHAMCS data.

Training

• Start: Laya's encoder (convaiinnovations/laya, revision 55cf4c4e) with the masked-language-model head of ModernBERT-large (answerdotai/ModernBERT-large, revision 45bb4654); the output layer shares its weights with the input embeddings. The two models have the same vocabulary and merge rules, so Laya's tokenizer is used unchanged.
• Objective: masked-language modeling; 30 percent of tokens are selected, and of those 80 percent become [MASK], 10 percent a random token and 10 percent stay unchanged.
• Batches: 256 blocks of 1,024 tokens a step (262,144 tokens); 7,630 steps and 2,000,158,720 tokens in all; 1,953,280 of the 2,087,336 training blocks were drawn, each at most once.
• Optimizer: AdamW (betas 0.9 and 0.98, epsilon 1e-6, weight decay 0.01 except on biases and normalization weights), peak learning rate 0.0001, gradient clipping at 1.0, bfloat16 autocast; linear warmup over 57 steps, then constant, then linear decay to zero from step 6,867.
• How I chose the text: I compared three mixes at 500 million tokens each, by token share: A, abstracts only; B, abstracts 50 percent, PMC full text 40 percent, case reports 10 percent; C, abstracts 35 percent, PMC full text 30 percent, case reports 10 percent, medical web text 25 percent. A rule I fixed before training kept A unless another mix scored more than 0.01 better on the validation data of the two questions below. Neither did (selection scores A 0.80660, B 0.79845, C 0.80460; lower is better), so I continued A's run from the start of its decay (step 1,717) to 2,000,158,720 tokens. I also tried a round of training on the two questions' own training records and, by a rule I fixed before training, did not keep it (score 0.83395).
• Hardware: one NVIDIA H100 (80 GB); about 7.6 hours for this model's 7,630 steps (derived from the logs of the two runs), 73,172 tokens a second, peak memory 52.77 GiB.

Evaluation

Held-out masked-language loss

Lower is better. Only abstracts were in the further training, so the other rows are text this training did not see. The first number column is mix A's run at 500 million tokens, at the end of its own schedule.

With a decision layer

To see whether the reader helps a model that answers questions, I trained the same decision layer on each reader: a new decision head, trained with Fiorillo v0.1's method on two questions (the reader is fine-tuned with it after the first 250 steps) and calibrated with a temperature fitted on validation data. The questions are the answer to a biomedical research question from its abstract (PubMedQA: yes, no or maybe; 500 test items) and how soon a patient with diabetes is readmitted after a hospital discharge (UCI Diabetes 130-US hospitals, 1999 to 2008: none, after 30 days, or within 30 days; 6,998 test encounters). Every number is on the test split, with a 95 percent interval from 1,000 bootstrap resamples of the test rows (paired for differences). Fiorillo v0.1, which keeps Laya's reader and Laya's own decision head, is in the last row for reference.

Each reader got one decision-layer training run, and the intervals cover the sampling of test rows only, not the randomness of training. In this project, training the same reader again with nearly the same recipe moved PubMedQA log loss by up to 0.044 and readmission log loss by up to 0.024, which is more than every readmission difference below. Read the differences with that in mind.

Differences (the first reader minus the second; a negative log loss difference favors the first reader). The first five rows are every reader comparison I fixed before training:

What it did not achieve

The decision layer on reader v1 was the candidate for Fiorillo v0.2. Against Fiorillo v0.1, its PubMedQA log loss was 0.7248 against 0.7428, a difference of -0.0180 (-0.0527 to +0.0148), and its readmission log loss 0.8542 against 0.8524, a difference of +0.0018 (-0.0003 to +0.0040). Before training I had required the PubMedQA interval to lie wholly below zero. It did not, so v0.2 was not released, and v0.1 remains the current decision model.

Known contamination

The pretraining text of Laya and of ModernBERT-large is not published and may include PubMed pages. Reader v1's further training excluded every PubMedQA article, by PubMed identifier and by the 13-word check above. BioClinical ModernBERT-large was trained on PubMed and PMC text that includes PubMedQA's articles, which favors its reader on PubMedQA in the tables above. I did not exclude other benchmarks built from PubMed abstracts (for example BioASQ, PubMed 200k RCT, BC5CDR and NCBI-disease), so results on them may overstate what the reader learned.

Limitations

• It was evaluated only by held-out masked-language loss and by two questions under one decision-layer recipe, with one training run per reader; the differences between readers are within what retraining alone can move on readmission.
• Its further training read English titles and abstracts only: no full-text articles and no clinical notes. How it does on clinical notes was not tested.
• It was trained on blocks of 1,024 tokens; inputs longer than that were not tested.
• It is a reader, not a decision model: no classification head is included, so it needs fine-tuning before it classifies text or answers questions.
• The literature it read carries its own biases, such as the questions researchers chose to study and what got published.
• It is not a medical device and must not be used to make clinical decisions.

Lineage

ModernBERT-large (answerdotai/ModernBERT-large, Apache 2.0) was fine-tuned into Laya (convaiinnovations/laya, Apache 2.0). This model is Laya's encoder with ModernBERT-large's masked-language-model head, trained further on PubMed titles and abstracts. Fiorillo v0.1 (johanneli/fiorillo-v0.1) is built on Laya's own reader, not on this one.

License and citation

Apache 2.0, like Laya and ModernBERT-large; the LICENSE file holds the license text and the NOTICE file carries the attributions.

This model was trained in part on data from the U.S. National Library of Medicine (NLM): PubMed citations and abstracts. NLM is not responsible for this model and has not endorsed it, and the data it was trained on may not reflect the most current data available from NLM. Some abstracts may be protected by copyright; only the model's weights are released, and no abstract text is distributed with them.

Li JE. Fiorillo reader v1: a biomedical ModernBERT-large reader further trained on PubMed abstracts. Hugging Face, 2026. https://huggingface.co/johanneli/fiorillo-reader-v1