TULLUS/codeparrot-small-multi-F32 🦜

A verified full-precision (F32) Safetensors conversion of the original CodeParrot Small Multi checkpoint. This release preserves the source model's learned tensor values; it is not a new training run or fine-tune.

Provided Quants

(sorted by size.)

CodeParrot Small Multi is a small GPT-2 causal language model for code completion and continuation, not an instruction-tuned chat model. It was pretrained on cleaned GitHub code and described upstream as covering Java, JavaScript, PHP, Python, C#, C++, Go, Ruby, and TypeScript. Treat it as an older, compact code-model baseline for experimentation not as standalone coding assistant.

Intended use and realistic expectations

Prompt it with a code prefix or partial file for the model to continue, rather than expecting it to follow natural-language instructions like a chat assistant. For example, start with def add(a, b):\n return and let it continue the function. Surrounding code and a few relevant examples can help establish the style and pattern to continue, but better prompting does not turn this base model into an instruction-tuned model or guarantee correct code.

The upstream card describes 300,000 training steps, a context length of 1,024 tokens, and roughly 58 billion training tokens. Its HumanEval table leaves pass@1, pass@10, and pass@100 as --%; it does not publish numerical HumanEval results for this exact multilingual checkpoint. This card therefore does not assign it a benchmark score. Results reported for the separate codeparrot/codeparrot-small checkpoint must not be presented as Small Multi results.

In informal local probes, this checkpoint produced plausible code continuations. Those probes were exploratory, not a controlled benchmark, and cannot establish a pass rate or a general accuracy estimate. Validate any generated code with tests and review before use.

Model details

Provenance and source verification

The conversion is based on the pinned upstream revision shown above. The original pytorch_model.bin was downloaded and its SHA256 was checked before deserialization. It was loaded for this audit with weights_only=True.

Original upstream checkpoint:

File size: 247,234,217 bytes
SHA256:    0207f6b427e3cbf1bcb9726abb6bbba6620e9912e1abf23087ccdef61818ffb2

Published TULLUS Safetensors file:

File:      model.safetensors
File size: 444,048,000 bytes
SHA256:    9cbfd95c12db0da2a0a475516d16e7f424a0b6d78883b4b663d281fe20ba0ed7

A local source-to-artifact audit completed with the following results:

• The pinned upstream checkpoint hash matched the expected source hash.
• All 149 logical state-dictionary tensors compared exactly between the upstream source and the published TULLUS artifact.
• The tokenizer vocabulary mapping matched exactly.
• Forward logits were exactly equal for three test prompts, with maximum absolute difference 0.0 for each comparison.
• The input embedding and output-head matrices in the original checkpoint had identical values, element for element.

The equality audit establishes preservation of the learned tensor values for this conversion. It is not a claim that the source and converted repository files are byte-identical, nor that their configuration metadata is identical.

Embedding storage and parameter counts

The original checkpoint contains separate input-embedding and language-model output-head entries:

transformer.wte.weight  shape (32768, 768)
lm_head.weight          shape (32768, 768)

In the source checkpoint these matrices occupied separate storage but had exactly equal values (maximum absolute difference 0.0). The TULLUS conversion sets tie_word_embeddings=True, so When loaded, the model exposes both components while sharing the same underlying parameter object and storage.

Safetensors stores the shared embedding matrix once rather than storing a duplicate output-head matrix. After loading, the model exposes lm_head.weight as a logical state-dictionary entry, with both components referencing the same underlying parameter storage.

This is a storage-representation change, not an additional training step. The audit also confirmed exact logical tensor equivalence and exact forward-logit equality on three test prompts.

weight tensor specific checks

python - <<'PY'
from transformers import AutoModelForCausalLM

model_path = "TULLUS/codeparrot-small-multi-F32"
model = AutoModelForCausalLM.from_pretrained(model_path)
model.eval()

wte = model.transformer.wte.weight
lm_head = model.lm_head.weight

print("Config ties word embeddings:", model.config.tie_word_embeddings)
print("Logical state_dict entries:", len(model.state_dict()))
print("Output-head entry present:", "lm_head.weight" in model.state_dict())
print("Same parameter object:", lm_head is wte)
print("Same underlying storage:", lm_head.data_ptr() == wte.data_ptr())
print("Maximum weight difference:", (lm_head - wte).abs().max().item())
PY

Config ties word embeddings: True
Logical state_dict entries: 149
Output-head entry present: True
Same parameter object: True
Same underlying storage: True
Maximum weight difference: 0.0

BOS/EOS metadata correction

The upstream repository has a difference between its model configuration and tokenizer:

• Upstream config.json declares vocab_size = 32768, bos_token_id = 50256, and eos_token_id = 50256.
• The tokenizer vocabulary has 32,768 entries and maps <|endoftext|> to token ID 0; its BOS and EOS IDs are both ID 0.
• Valid IDs for a 32,768-entry vocabulary are ID 0 through 32767, so 50256 is outside this model's output vocabulary.

The TULLUS model and generation configurations set BOS/EOS to ID 0 / ID 0, aligning them with the tokenizer. The tokenizer mapping itself matches the upstream tokenizer exactly. This metadata correction does not change the learned tensors. It can affect generation behavior where BOS/EOS settings control special-token handling or stopping; this card does not claim identical stopping behavior for every decoding setup.

The tokenizer has no configured padding token (pad_token_id=None).

Usage

This model is a pretrained code-completion model. It continues code from a supplied prompt and is not instruction-tuned.

Quick start — Transformers pipeline

from transformers import pipeline

pipe = pipeline("text-generation", model="TULLUS/codeparrot-small-multi-F32")
outputs = pipe("def hello_world():")
print(outputs)

Direct generation — print only the continuation

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TULLUS/codeparrot-small-multi-F32"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

prompt = "def hello_world():"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)

new_tokens = outputs[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The second example is also useful because it prints only newly generated tokens, rather than echoing the prompt.

This example uses greedy decoding for a simple reproducible starting point. Output depends on the prompt and generation settings; the example does not guarantee a correct implementation.

License and acknowledgement

The upstream repository identifies the model as Apache-2.0 licensed. Consult the upstream model repository and its dataset references for provenance and applicable terms. TULLUS does not claim authorship of the upstream architecture, training data, or learned capabilities.

Thanks to the original developers for this wonderful, highly capable mini model from its era. And thanks to OpenAI for the help along the way. We dug deep into this cool project, verified the conversion against the original checkpoint, and produced a Safetensors release backed by tensor-equivalence and forward-logit checks.

TULLUS BAKES 🍪
It's Old Gold and we safetensorized it :]