HobbyLM-1B Broad-Instruct SFT, step 3450 (HF format)

A Hugging Face–format export of HobbyLM-1B after broad-instruction supervised fine-tuning, at training step 3450. It is a mixture-of-experts model with 20 layers, 64 routed experts plus 1 shared expert, and 8 experts active per token. Loading it requires trust_remote_code=True.

Source checkpoint

Weights in this repository

model.safetensors: SHA-256 d0899e0ac88c90a2c01ca481463bc0f96c6d50337f1b3e0dc56e1d3dc2999e89, 4,354,568,656 bytes, 280 float32 tensors.

The file stores lm_head.weight as an explicit copy of the tied embedding matrix (50304 × 1024 float32, 206,045,184 bytes). That copy is why it is about 206 MB larger than an export that omits the tied duplicate. Because tie_word_embeddings defaults to true, loading re-ties lm_head to the input embeddings.

Verification status: partial

The tensors in this file were compared against the native checkpoint above. The comparison accounted for the conversion's layout changes: the fused QKV and gate/up weights are split into separate matrices, and the expert weights are transposed.

• All sampled sections matched bit-for-bit. That is 496 sections across 232 of the 280 tensors, covering every non-expert tensor, lm_head against the tied embedding, and the routed and shared expert tensors in 7 of the 19 MoE layers.
• Full tensor identity has not been verified. The comparison sampled sections of each tensor rather than every byte. The expert tensors in the other 12 MoE layers (48 tensors) were not sampled.

Context length

max_position_embeddings is 4096, matching the training sequence length of the source checkpoint. The positional encoding matches the source checkpoint: plain RoPE with theta 10000 and no scaling.

4K retrieval quality has not been established. Training on 4096-token sequences, or running inference without errors at that length, does not show that the model reliably uses or retrieves information from across a 4096-token context. No long-context retrieval evaluation of this checkpoint is reported here.

Revision history

• f4b8557098c01fd42195f32feee3a3275dfbfbe3 (2026-09-26): initial upload. Its config.json, configuration_hobbylm.py and modeling_hobbylm.py were copied unchanged from the pretrained base repo harims95/hobbylm-1b-hf. As a result config.json declared max_position_embeddings: 1024, the base model's pretraining length, instead of this checkpoint's 4096.
  • The model code computes rotary position embeddings directly from position IDs and never reads max_position_embeddings, so the value does not change the model's forward computation.
  • Some serving and evaluation tools derive a default context limit from this field. Under that revision they may have capped inputs at 1024 tokens, depending on the tool, its version and any explicit override. That behavior was not tested.
• 7f4ebaf0261bbfe7155adc8c7228bc17e3c60b1c (2026-10-01): metadata only. Set max_position_embeddings to 4096, recorded "rope_scaling": null explicitly, and added this model card. Weights, model code and generation defaults were unchanged.
• This revision changes generation defaults only. generation_config.json now decodes greedily by default: do_sample is false, and the sampling-only settings temperature: 0.7 and top_p: 0.9 are removed. bos_token_id, eos_token_id, pad_token_id (all 50256) and max_new_tokens: 256 are unchanged. Weights, model code and config.json are unchanged. Earlier revisions remain available in the repository history.

Usage

import tiktoken, torch
from transformers import AutoModelForCausalLM

repo = "harims95/hobbylm-1b-broad-sft-3450-hf"
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype=torch.float32).eval()
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode("Hello")])
out = model.generate(ids, attention_mask=torch.ones_like(ids), max_new_tokens=32, do_sample=False)
print(enc.decode(out[0].tolist()))

• Tokenizer: no tokenizer files are included. The model uses the GPT-2 BPE vocabulary (for example tiktoken's gpt2 encoding). <|endoftext|> (id 50256) serves as BOS, EOS and padding. The embedding matrix has 50304 rows; ids above 50256 are not produced by the GPT-2 tokenizer.
• Generation defaults: generation_config.json uses greedy decoding by default (do_sample: false) with max_new_tokens: 256, and <|endoftext|> (50256) as EOS and padding. To sample, pass sampling settings explicitly, for example do_sample=True, temperature=0.7, top_p=0.9 (the defaults in revision 7f4ebaf0 and earlier).
• Model code: the bundled modeling_hobbylm.py implements plain RoPE only, with no YaRN or other scaling. That matches this checkpoint, whose rope_scaling is null.