blue-scrub-150M
Phase 5 Hugging Face deployment from slm-train-scratch.
Model summary
- Model type: decoder-only causal language model (
LlamaForCausalLM). - Model size: 150M-class; measured checkpoint size is 154.42M parameters.
- Instruction tuning: Non-IT / non-instruction-tuned. This is a base language model, not a chat model and not instruction aligned.
- Tokenizer: byte-level BPE-style tokenizer with vocabulary size 32000.
- Precision: bfloat16 checkpoint.
Training data
The model was pretrained on a cleaned mixture of general educational text and medical/health-domain text.
- General/education source: HuggingFaceFW/FineWeb-Edu, cleaned and filtered subset: 2.00M rows.
- Medical/health source: TheBlueScrubs/the_blue_scrubs-v1, cleaned and filtered subset: 2.00M rows.
- Packed train tokens: 5.72B tokens.
- Packed validation tokens: 18.22M tokens.
Training run
- Epochs configured: 1.
- Approximate logged wall-clock training time: 73.3 hours (3.1 days). This is based on recorded metrics and may include evaluation, resume, and pause gaps.
Intended use
This checkpoint is intended for:
- research experiments with small/base language models;
- continued pretraining;
- domain adaptation experiments;
- evaluation of a compact medical-plus-education pretrained base model.
Limitations
- This is not an instruction-tuned or chat-aligned model.
- Outputs may be incomplete, incorrect, or unsafe without downstream alignment and evaluation.
- The model must not be used as a source of medical advice.
- The training mixture contains web and domain text; users should evaluate bias, factuality, memorization, and domain safety before downstream use.
Basic loading
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "mps/blue-scrub-150M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto")