FCP-135M
Fact-Conditional Pretraining (FCP) makes factual information explicitly available during pretraining by conditioning on triples extracted from each document and predicting the original text. This reduces the pressure to memorize factual associations in model parameters.
This repository contains the final pretraining checkpoint at step 97,560, before downstream task-specific fine-tuning. The model was initialized from scratch using the SmolLM2-135M architecture and tokenizer. Model weights are stored in BF16 safetensors format. The tokenizer includes the factual-triple markers and the training ChatML template.
The processed pretraining corpus is available at anonymous-review-0/FCP-pretrained-20B.
Evidence-conditioned generation
Provide a passage and a question. The included pipeline extracts triples with
Qwen/Qwen3-8B, then uses them to condition FCP's answer. The extractor is a
separate model and requires additional memory.
Download the inference helpers into a working directory:
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="anonymous-review-0/FCP-135M",
local_dir="fcp_helpers",
allow_patterns=["fcp_inference.py", "triple_extraction.py", "requirements.txt"],
)
Install the dependencies using Python 3.11:
python -m pip install -r fcp_helpers/requirements.txt
Run from the same working directory:
import sys
sys.path.insert(0, "fcp_helpers")
from fcp_inference import FCPPipeline
pipe = FCPPipeline.from_pretrained(
"anonymous-review-0/FCP-135M", device_map="auto", torch_dtype="auto"
)
answer = pipe(
[{"role": "user", "content": "Who wrote Nineteen Eighty-Four?"}],
context="Nineteen Eighty-Four was written by George Orwell and published in 1949.",
)
print(answer)
Standard Transformers loading
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("anonymous-review-0/FCP-135M")
model = AutoModelForCausalLM.from_pretrained(
"anonymous-review-0/FCP-135M", device_map="auto", torch_dtype="auto"
)
The helper Python files are explicit local utilities; loading the weights does
not require trust_remote_code=True. manifest.json records the released file
hashes. Optimizer states, training logs, and private provenance are excluded.
Third-party resources
Architecture and tokenizer: HuggingFaceTB/SmolLM2-135M.
Triple extractor: Qwen/Qwen3-8B.
The inference helper code is covered by the included LICENSE-code.
Third-party resources retain their original terms and attribution.