AppleMind-1.0-Mini

AppleMind-1.0-Mini is a compact decoder-only causal language model trained from scratch by AppleMind on a 300M-token curriculum.

The model has 1,020,480 parameters, a 256-token context window, and a 50,263-token digit-aware byte-level BPE tokenizer (GPT-2 + special tokens).

Model Details

Credits to BananaMind for inspiring me to make AppleMind.

Tokenizer

AppleMind 1.0 Mini uses a 50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens.

Training Data

The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.

Training Setup

Evaluation

AppleMind 1.0 Mini has not been formally evaluated with lm_eval yet. No benchmark scores are currently reported.

The model's current generation quality has been checked with basic text-generation prompts, including:

• Once upon a time
• The little boy
• In the forest

Prompt: Once upon a time

Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked its G during work used after several There and at there b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many

Prompt: The little boy

The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many

Prompt: In the forest

In the forest often form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but

Usage

AppleMind 1.0 Mini uses custom architecture code, so load it with trust_remote_code=True.

pip install -U transformers safetensors torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AppleMind-AI/AppleMind-1.0-Mini"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True,
    )
)

Note: AppleMind 1.0 Mini has a 256-token context window, so the prompt plus generated tokens should stay within that limit.

License

Apache 2.0