qwen25-0.5b-finemath-4plus-lit

The LitGPT-format export of a baseline model for TDT comparison experiments, trained on FineWeb-Edu and then FineMath-4plus. It contains the same weights as the Transformers-format model.

Purpose

These models were created as baselines for comparison experiments with TDT. Their weights come from conventional gradient-based autoregressive language model training and provide a reference for comparing TDT training or adaptation methods and low-bit variants. The released checkpoints themselves were not trained with TDT.

The name qwen25-0.5b identifies the architecture. These models use the Qwen2.5-0.5B architecture and tokenizer, but their initial training did not start from official Qwen pretrained weights. The FineWeb-Edu stage starts from random initialization; the FineMath variants continue training from that FineWeb-Edu checkpoint.

Architecture

The configured maximum of 32,768 positions is an architectural setting. Training used sequences of 2,048 tokens; long-context performance has not been established.

Data preparation and training objective

The same prepare_fineweb_edu.py script is used for FineWeb-Edu and FineMath.

• Read the text column from Parquet files and skip empty text.
• Tokenize with the Qwen/Qwen2.5-0.5B tokenizer, without BOS and with EOS at the end of each document.
• Create streaming chunks with litdata.optimize and TokensLoader.
• Load blocks of 2,049 tokens: use the first 2,048 as inputs and the shifted 2,048 as targets, minimizing next-token cross-entropy.
• Shuffle training data, leave validation data unshuffled, and use seed 42.

Use in comparison experiments and limitations

These are base models for text completion. The recorded pipeline does not include conversational supervised fine-tuning, instruction tuning, or RLHF. A chat template in the tokenizer does not establish that the model was trained to converse.

The training configurations use split_names: [train, train], so validation reads from the same split as training. Reported validation losses are not independent held-out results. For TDT comparisons, control the initial weights, tokenizer, data splits, token budget, and evaluation conditions, and use a separate evaluation dataset. This card does not report TDT comparison results or mathematics benchmark scores.

Data sources are FineWeb-Edu and, for the FineMath variants, FineMath. Both source datasets list odc-by in their public metadata. Refer to their dataset cards for terms and attribution details.

Related models

Code: mssfj/lowbit-math-reasoning

Training data and configuration

Continued pretraining starts from the FineWeb-Edu pretrained weights and uses next-token prediction on mathematics documents.

• Initial checkpoint: mssfj/qwen25-0.5b-fineweb-edu-10bt-litgpt, the default in continued_pretrain.sh.
• Source data: HuggingFaceTB/finemath, finemath-4plus configuration, train split.
• Pretokenized data: mssfj/finemath-4plus-qwen25.
• The preprocessing script divides the data into 128 Parquet shards and defaults to 32 tokenization workers.
• FineMath-stage training token budget: 9,600,000,000, separate from the FineWeb-Edu-stage budget.

The recorded settings for W&B run 84pczgsb are:

The current YAML uses micro batch size 16 and a checkpoint interval of 2,000, whereas the recorded run used 8 and 1,000. Use the W&B records when reproducing the execution conditions of that run.

Loss measurements

The W&B summary val_loss belongs to step 9,000, not the final checkpoint. The final checkpoint was reevaluated using pretokenized dataset revision c0e59cf67bdca55c2b298a06a298f229dfecd447, sequence length 2,048, original batch size 8, 100 batches, 8 data workers, seed 42, and bf16-mixed precision. To fit GPU memory, each batch was evaluated one sequence at a time and the losses were averaged. Reevaluation used PyTorch 2.12.0a0+0291f960b6.nv26.04.48445190, a different environment from the original run. A strict comparison of this small loss difference requires reevaluating both checkpoints in the same environment.

This evaluation uses the training split and is not an independent measure of mathematics performance.

Reproduction code

continued_pretrain.sh contains steps for obtaining the FineWeb-Edu checkpoint, preprocessing FineMath, continued pretraining, and conversion.

cd mylitgpt
litgpt pretrain \
  --config config_hub/pretrain/qwen25-0.5b-finemath-4plus.yaml \
  --initial_checkpoint_dir out/pretrain/qwen25-0.5b-fineweb-edu-10bt-lit \
  --out_dir out/pretrain/qwen25-0.5b-finemath-4plus

Prepare the tokenizer, pretokenized data, and initial checkpoint before training. This command uses the current YAML configuration.

Checkpoint format and usage

lit_model.pth is a LitGPT model-only state dict. It contains no optimizer state, training step, or data-loader state, so it is not a full training checkpoint for exact resumption. The repository includes model_config.yaml and tokenizer files.

All 195 tensors in the published LitGPT checkpoint were compared with the HF checkpoint converted back to LitGPT: the keys and every tensor value matched. The actual weights are FP32. Prefer the tensor dtype over the dtype declaration in the auxiliary HF config.json.

This format cannot be loaded directly with AutoModelForCausalLM.from_pretrained(). Use LitGPT or the corresponding HF-format repository. The auxiliary HF config.json retains the original Qwen setting tie_word_embeddings: true, but these weights have separate input embeddings and output weights. Set this field to false when converting to HF format.

# Install the LitGPT version included in the code repository.
cd mylitgpt
pip install -e .
hf download mssfj/qwen25-0.5b-finemath-4plus-lit \
  --local-dir checkpoints/finemath-4plus-lit
litgpt generate \
  --checkpoint_dir checkpoints/finemath-4plus-lit \
  --prompt "The answer to 2 + 3 is" \
  --max_new_tokens 64 \
  --temperature 0

This export format extracts model weights from the end-of-training checkpoint with convert_pretrained_checkpoint. Keep the original full training checkpoint separately if exact training resumption is required.