BLOOM-560m: French vocabulary pruning
Author: Daniil Koblov
Experiment: Homework 1, Tokenizer Pruning for French
Checkpoint: frequency threshold m=1, exported from the completed Kaggle Tesla T4 run on 8 October 2026.
This is a vocabulary-pruned derivative of bigscience/bloom-560m. The transformer layers retain their pretrained weights. The shared input/output embedding matrix retains 41,017 token IDs and is padded to 41,024 rows (a multiple of 64). No weights were fine-tuned. This is the main pruned checkpoint, not the B1 extended-vocabulary or B3 random-embedding experiment.
Changes from the original model
Token frequency is counted on 675 complete French Wikipedia articles (10,013,255 UTF-8 text bytes). The vocabulary retains observed tokens, added/special tokens, the base alphabet and recursive BPE merge dependencies. Merge order is preserved and IDs are remapped contiguously. Input embeddings and the output head remain tied. Stored weights use bfloat16.
| Measure | Original BLOOM | Pruned, m=1 |
|---|---|---|
| Vocabulary | 250,680 | 41,017 |
| Embedding/output rows | 250,880 | 41,024 |
| Unused rows | 200 | 7 |
| Parameters | 559,214,592 | 344,322,048 |
| Held-out bits per byte | 0.986503 | 0.994172 |
| Held-out tokens per byte | 0.222893 | 0.224160 |
Parameters decrease by 38.4%, with 0.78% higher held-out bits per byte. Evaluation scores all 68 complete held-out articles (1,016,502 UTF-8 bytes) using windows of up to 512 tokens, with fp16 inference and float32 cross-entropy. All held-out texts round-trip exactly. Training tokenization is unchanged; mean held-out sequence length increases by 0.61%.
B4 serving measurements use 256 generated tokens, three warm-ups and ten measured repetitions per configuration. Median decode throughput improves by 15.0% at batch 1 and 15.4% at batch 8 in this Kaggle Tesla T4 session. These are within-run measurements, not a guarantee across hardware or workloads.
Alignment and scoring
The seven unused rows are zero-filled and are not tokenizer IDs. The saved generation_config.json suppresses IDs 41017–41023, so model.generate() cannot emit them with the supplied configuration. Input and output weights remain tied after loading.
Raw model.forward() still returns 41,024 logits. For the BPB protocol used here, slice logits to [..., :len(tokenizer)] before applying softmax or cross-entropy; passing labels directly uses the full output width and does not implement that exclusion. The original baseline likewise excludes its 200 undefined output IDs. config.vocab_size is the physical matrix width; config.tokenizer_vocab_size is 41,017. Padding preserves the real vocabulary and retained weight values.
Load
Use the tokenizer from this repository: its token IDs differ from the original BLOOM tokenizer.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "dakoblov/bloom-560m-fr-pruned"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo,
dtype=torch.float16 if device == "cuda" else torch.float32,
attn_implementation="eager",
).to(device).eval()
inputs = tokenizer("La France est un pays", return_tensors="pt").to(device)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
The experiment used Transformers 4.57.6 and Tokenizers 0.22.2. Generation can be repetitive; this is a base language model, not an instruction-tuned assistant.
Reproducibility
- Base model revision:
ac2ae5fab2ce3f9f40dc79b5ca9f637430d24971. - Corpus: dakoblov/fr-wiki-popular-5-30kb-10mb-1mb, revision
c0035cee5b2ffe89072d0a473b04e1532192b22b. - Source:
wikimedia/wikipedia, configuration20231101.fr, revisionb04c8d1ceb2f5cd4588862100d08de323dccfbaa. - Popular articles are selected using recurrence in monthly top-1000 pageview lists for 2025, restricted to complete texts of 5,000–30,000 UTF-8 bytes. The corpus provides provenance and reconstruction details.
- Seed:
1337. Whole-article train/held-out splits are disjoint. - The accompanying submission contains the executed
hw1_Koblov.ipynb, PDF report, raw measurements and ID mapping.
Limitations and license
This educational experiment measures French Wikipedia tokenization and prediction. Other languages and downstream tasks were not evaluated and may degrade substantially. Popularity and length filtering bias the corpus. Threshold choice uses the held-out set, so deployment claims need a separate test set. Inherited model biases and unsafe outputs remain possible; see the original BLOOM model card.
This derivative remains subject to the original BigScience BLOOM RAIL 1.0 license, reproduced in LICENSE, including its use restrictions. Original BLOOM weights are credited to BigScience. Modifications are vocabulary/merge pruning, token-ID remapping, embedding slicing, zero padding to a multiple of 64, generation suppression and checkpoint export; there was no additional weight training. Wikipedia corpus licensing and article attribution are documented in the linked dataset.