Duloch Pico 125M Base

Duloch Pico 125M Base is a compact, decoder-only causal language model trained from scratch on 2.502 billion tokens from the FineWeb sample-10BT corpus. The model contains exactly 125,095,680 learned parameters and supports sequences of up to 512 tokens.

This is a raw base model. It is not a chat model, instruction-tuned model, aligned model, or safety-tuned model. It predicts text continuations and should not be expected to follow instructions reliably.

Model details

Duloch Pico was trained from scratch and contains no Meta Llama weights. It is packaged using the standard Transformers Llama implementation because its decoder architecture, RMSNorm, RoPE, SwiGLU, and grouped-query attention are directly compatible. No custom code or trust_remote_code setting is required.

Intended use

The model is intended for research, education, small-model experiments, base-model evaluation, and further training. It may also be useful for testing resource-constrained inference and Apple Silicon workflows.

The model is not suitable as a factual authority, a safety-critical system, or an unsupervised user-facing assistant. Additional evaluation and appropriate safety controls are required before any deployment.

Training

The model was trained with next-token prediction using a deterministic subset of HuggingFaceFW/fineweb, specifically the sample-10BT configuration.

The local preparation selected 3,600,000 documents, balanced across 15 source Parquet files, then applied exact duplicate removal and trigram SimHash near duplicate filtering. The accepted training split contained 3,560,666 documents. Training stopped within the first epoch after 2,501,935,104 tokens, which is approximately 20 tokens per learned parameter.

The final scheduled held-out validation loss was 3.1502 at 2,500,001,792 tokens, equivalent to approximately 23.34 perplexity. Each scheduled validation used a fixed maximum of 64 windows rather than the entire validation corpus.

Training data attribution and terms

FineWeb was created and published by Hugging Face from Common Crawl material. The source dataset is available at HuggingFaceFW/fineweb under the Open Data Commons Attribution License, ODC-By 1.0. Use of FineWeb is also subject to the Common Crawl Terms of Use.

FineWeb is derived from public web crawls. It can contain copyrighted text, personal information, factual errors, unsafe material, bias, malformed pages, and residual duplicates despite upstream and local filtering. FineWeb reports that email addresses and public IP addresses were anonymised, while also warning that other personally identifiable information is likely to remain. The Apache-2.0 licence for these model weights does not grant rights to third-party training content.

Evaluation

The following results use the EleutherAI Language Model Evaluation Harness at commit b954108c9baaaa934b4ad842033b31a97ee30816. Evaluation was zero-shot, float32, batch size 32, and limited to the model's 512-token context. No BOS token was added. The full benchmark splits were used.

These scores describe a small base-model evaluation suite. They do not establish broad capability, factual reliability, instruction following, or safety. The aggregate machine-readable results and evaluation provenance are included in evaluation/summary.json.

Usage with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pdjamez/Duloch-Pico-125M-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=False,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=False,
    dtype=torch.float32,
)

prompt = "The weather in Scotland is"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Keep the prompt and generated continuation within the supported 512-token context. This base model has no chat template.

The direct loading and generation path above has been tested with Transformers 4.57.1 and 5.14.1. Transformers 5.14.1 can omit the display space between the prompt and continuation when its pipeline() convenience API separately decodes and then joins SentencePiece text. The generated token IDs and direct full-sequence decoding remain correct. The example above avoids that rendering issue.

The canonical repository uses standard Transformers files. Conversion to MLX BF16 has been validated with mlx-lm 0.31.3, but the MLX package is planned as a separate derivative repository.

The canonical Transformers model was also compared directly with the native Swift/MLX checkpoint using all 20,000 final-position logits at deterministic input lengths 1, 2, 17, 511 and 512. Every probe passed the declared bfloat16 cross-framework tolerance, and the top-five token set matched in every case. Native MLX and Transformers produced identical 64-token greedy generations, including Transformers generation with its key-value cache. They also produced the same next token from a 511-token prompt at the supported 512-token boundary.

One artificial one-token probe reversed two near-tied top candidates within the declared numerical tolerance; the top-five set remained identical. This means the validation establishes tolerance-aware numerical and operational parity, not bit-exact equality of every bfloat16 calculation. A separate-process reload after saving the FP32 Transformers model passed the same gate and produced byte-identical Safetensors weights.

Limitations

• The 512-token context is short by current standards.
• The model can generate false, contradictory, biased, offensive, unsafe, or private-looking content.
• It has limited factual recall, arithmetic, reasoning, and long-form coherence.
• Training was English-focused, although the web corpus contains some other languages.
• The full selected corpus received automated screening, but it was not exhaustively inspected by people. Memorisation has not been ruled out.
• The model has not been instruction tuned, preference aligned, red teamed, or safety tuned.
• Benchmark results can be affected by prompt format, tokenizer behaviour, harness version, precision, and context truncation.

Please use the repository's Community tab to report reproducibility problems, suspected memorised personal information, or correction and removal requests. Requests concerning records in the original FineWeb dataset should also use the removal route linked from the FineWeb dataset card. Selective deletion from released model weights may not be technically possible. A confirmed issue may instead require withdrawing or replacing an affected release.

Licence

The model weights and accompanying release files are provided under the Apache License 2.0. See LICENSE for the full terms. Training data remains subject to its original source terms and any applicable rights.

Acknowledgements

This model was trained using FineWeb data published by Hugging Face and was evaluated using the EleutherAI Language Model Evaluation Harness.

References

• Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, NeurIPS 2024 Datasets and Benchmarks Track.
• FineWeb dataset card.
• Common Crawl Terms of Use.