Quartz Micro Preview
The instruction-tuned release of Quartz Micro Preview Base — the same ~1B-parameter mixture-of-experts model, trained completely from scratch on a single consumer GPU, now given a full-parameter supervised fine-tune so it follows instructions instead of just completing text.
This is a preview, not a finished product. It is small on every axis by design, and its evaluation below is reported honestly, including where it still falls short — that's this project's standing practice, not a disclaimer added after the fact.
What changed vs. the base checkpoint
- Full-parameter SFT, not LoRA — the base model has zero instruction-following behavior, so a light adapter touch isn't the right tool; every parameter was updated.
- Dataset: databricks-dolly-15k,
human-written (not distilled from another company's model), formatted as
### Instruction:\n...\n\n### Response:\n(with an added### Context:\nblock for context-grounded examples), tokenized with the pretrain's own custom 32K byte-level BPE tokenizer. 14,051 of 15,011 examples kept after filtering to the model's 2,048-token context. - Only the response tokens are supervised (prompt tokens masked to
label=-100); optimizer is 8-bit AdamW (bitsandbytes) with full gradient checkpointing, matching the pretrain's own training stack. - This is the second SFT attempt (v2): the first pass underperformed, so this run used a higher learning rate and a fresh optimizer state rather than continuing from the first run's optimizer. It completed 1,317 steps on a single GTX 1660 Ti (6GB VRAM).
Architecture
Unchanged from the base checkpoint — DeepSeek-style fine-grained mixture-of-experts:
| Total parameters | 1,031.0M (~1.03B) |
| Active parameters / token | 394.0M |
| Hidden size | 1,024 |
| Layers | 20 (first 2 dense, rest MoE) |
| Attention | 16 query heads / 4 KV heads (GQA), head dim 64 |
| Context length | 2,048 tokens |
| Routed experts | 24 (6 active per token) |
| Shared experts | 2 (always active) |
| Expert FFN size | 640 (fine-grained segmentation) |
| Router | top-6 of 24, 0.01-weighted load-balancing loss |
| Vocabulary | 32,000 tokens |
| Tied embeddings | yes |
Held-out evaluation: base vs. this SFT checkpoint
Ten hand-authored prompts, not sampled from dolly-15k or any training data, spanning open QA, closed/context QA, summarization, brainstorming, classification, extraction, general QA, instruction-following, and creative writing. Full transcript below is unedited.
Honest summary: a modest, mixed improvement — not a clean win.
The clearest gain is grounded, context-based QA. The weakest spot is still strict instruction-following (format, length). Both checkpoints remain largely incoherent past the first sentence or two, which tracks with the model's size — that's not something this SFT pass was expected to fix.
Win — closed QA (uses given context correctly):
Prompt: Context: The Golden Gate Bridge is 2,737 meters long and was completed in 1937.\nQuestion: How long is the Golden Gate Bridge?
Base: ignores the context entirely, wanders into an unrelated story about "the Banana" process and the Coast Guard.
SFT: "The Golden Gate Bridge is 2,737 meters long and has an area of approximately 2.4 million square miles." — correctly extracts and restates the fact (the second clause is invented, but the requested answer is right).
Partial win — brainstorming (attempts the requested list format):
Prompt: List three ideas for reducing food waste in a household kitchen.
Base: no list structure, generic unrelated advice about water and diet.
SFT: "1. Make sure you have all food waste in your kitchen. \n2.f leaf food and water should be discharged from local resources. \n3. Focus on local food waste..." — correctly produces a numbered list (the content is still weak/garbled).
Partial win — topical relevance:
Prompt: Why do cats purr?
Base: drifts immediately into an unrelated riff on "the human body," never mentions cats again.
SFT: "Finding the right balance between breast and chest health is hard... So my recommendation is to keep your cat's skin healthy and strong..." — stays anchored to "cat" throughout, base does not.
No win — strict instruction-following:
Prompt: Write a short two-sentence greeting for a birthday card.
Base: an unrelated bulleted list of vague social advice.
SFT: "The free meeting is open for all people... The national party has the following 82 seats: The Democratic Party..." — completely unrelated to the prompt, and no better than base here.
No win — classification:
Prompt: Classify the following as either a fruit or a vegetable: broccoli
Base: never answers the classification, rambles about diet generally.
SFT: "broccoli, broccoli, cabbage,iant cucumber, cucumber, ormanus... This fruit is most closely associated with pears and cucumber." — also fails to answer, with garbled sub-word artifacts not seen in the base output.
No GGUF build
Every other release on the Vertex AGI Hugging Face org ships GGUF alongside the native weights. This one doesn't, and that's deliberate rather than an oversight.
llama.cpp's closest matching architecture (deepseek, the original
DeepSeekMoE design) shares this model's leading-dense-layers + fine-grained
routed-experts + always-on shared-experts shape almost exactly — except it
skips renormalizing the selected top-6 expert weights after routing, while
this model's trained behavior does renormalize them. Measured on real
prompts, the top-6 experts this model picks only carry ~61% of the total
softmax mass on average (26–99% depending on the token) before that
renormalization — so a GGUF built on that architecture would run the
routed-expert pathway at a token-varying fraction of its trained strength,
not the same computation this checkpoint actually does. Rather than ship a
file that loads and runs but isn't faithful to the model, we're leaving
GGUF out until llama.cpp has a matching architecture (or we write one).
Files
model.safetensors— model weights, fp32 (optimizer state dropped)configuration_quartz.py,modeling_quartz.py—transformers-compatiblePretrainedConfig/PreTrainedModelwrapper (QuartzMoEConfig,QuartzForCausalLM), wired up viaauto_mapinconfig.jsonmodel.py,config.py— the original plain-PyTorch model class and architecture config, used byload_model.pyconfig.json— architecture config in HF's expected formattokenizer.json,tokenizer_config.json,special_tokens_map.json— a standardtransformersfast tokenizertokenizer/vocab.json,tokenizer/merges.txt— the custom tokenizer's raw vocab/merges, used byload_model.py's plain-PyTorch path. Only compatible with this model's weights.load_model.py— minimal working example using the plain-PyTorch path
Usage
Via transformers (recommended):
pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"VertexAGI/quartz-micro-preview", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview")
prompt = "### Instruction:\nWhy do cats purr?\n\n### Response:\n"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))
trust_remote_code=True is required — this is a bespoke architecture
(DeepSeek-style fine-grained MoE), not one of transformers' built-in
model types. No KV-cache support yet, so generate() recomputes attention
over the full sequence each step.
Plain PyTorch (no transformers dependency):
pip install torch safetensors tokenizers
python load_model.py
Limitations
Same size-driven limitations as the base checkpoint (factual unreliability, weak general knowledge, no safety fine-tuning), plus SFT-specific ones documented honestly above: strict format/length instructions (e.g. "write exactly two sentences") are not reliably followed, and generation occasionally produces garbled sub-word artifacts the base checkpoint doesn't. Treat this as a proof-of-concept instruction tune on a genuinely tiny model, not a general-purpose assistant.
Built by Vertex AGI. Every model we ship — weights, not just claims.