Blaster
Blaster is a multimodal companion model: it can look at images, listen to microphone speech, and reply in text (with optional spoken replies via a codec). It is a fine-tune of AllenAI Molmo-7B-D with continuous audio understanding (SLAP), speaker markers, and discrete speech codes for generation.
This repository is the merged serving checkpoint — the weights you load for inference (vLLM / Transformers). Adapters and companion modules live in sibling repos under the same molmo-audio-* ids (marketing name: Blaster).
What it can do
| Capability | Notes |
|---|---|
| See | Answer questions about attached images |
| Hear | Respond to spoken turns without an ASR transcript bridge — audio is encoded and injected as continuous features |
| Talk (text) | Short, conversational replies for mic and typed chat |
| Optional voice-out | When the model emits audio codebook tokens, a Spectral FSQ (or legacy DiT) decoder can synthesize speech |
| Who spoke | Vocabulary includes <spk1>…<spk4>; a sibling diarization head supports multi-speaker cues |
It is designed for desk / appliance-style multimodal chat (image + mic + text), not as a pure ASR system or a general web agent by itself.
Quick start
Transformers (research / offline)
from transformers import AutoModelForCausalLM, AutoProcessor
repo = "0x8badbeef/molmo-audio-serving-diar-d"
model = AutoModelForCausalLM.from_pretrained(
repo, trust_remote_code=True, device_map="auto"
)
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
# Full multimodal mic+image paths use the project serving stack (below).
Recommended: project Docker stack
Blaster’s production path uses a small gateway + engine + optional DiT/FSQ workers:
export SERVING_MODEL_DIR=$PWD/molmo-audio-serving-diar-d
export MOLMO_LORA_DIR=$PWD/molmo-audio-lora-diar-d/checkpoint-112000 # or your tip
export DIAR_MODEL_DIR=$PWD/molmo-audio-slap-diar
export SPECTRAL_FSQ_DIR=$PWD/molmo-audio-spectral-fsq-v1
export MOLMO_AUDIO_CODEC=spectral_fsq
docker compose -f deploy/compose.yaml up --build -d
Sibling LoRA (unmerged, for continued training): molmo-audio-lora-diar-d.
Model family (Blaster)
| Hub id | Role |
|---|---|
molmo-audio-serving-diar-d (this repo) | Merged weights for serving |
molmo-audio-lora-diar-d | LoRA + audio_modules.pt checkpoints |
molmo-audio-slap-stage-a | SLAP speech encoder (understanding) |
molmo-audio-slap-diar | Diarization head |
molmo-audio-spectral-fsq-v1 | Speech codec (voice-out) |
molmo-audio-dit-dac-v3 | Legacy DiT audio generator |
molmo-audio-phase-h-restore | Bill of materials / restore bundle |
What’s inside this repo
| Files | Purpose |
|---|---|
model-*-of-*.safetensors | Merged language + vision weights |
config.json | Molmo config + audio special-token ids |
audio_modules.pt | SLAP encoder + projector used at serve time |
tokenizer* / special_tokens_map.json | <soa> <eoa> <audio_pad> <spkN> <audio_*> |
token_manifest.json | Gateway / router ids |
modeling_molmo.py (etc.) | Remote-code sources for Transformers |
Architecture sketch: frozen Molmo-7B-D backbone + folded LoRA; mic audio → SLAP → projector → continuous inject at <audio_pad>; optional discrete <audio_*> codes for synthesis.
Intended use
- Interactive multimodal chat with images and/or microphone audio
- Research and non-commercial demos of speech-aware VLMs
- Starting point for further LoRA continues (prefer the LoRA sibling tip)
Out of scope / limitations
- Not a dedicated ASR engine — comprehension is end-to-end into the LM, not a transcript-first pipeline
- Spoken and typed behavior still show class confusions on hard probes (e.g. status vs identity, instruction following); see training notes in the project
README_TRAINING.md - Voice-out quality depends on the paired codec (Spectral FSQ / DiT), not only this merge
- Tool-use / web search (when enabled in the UI) can still over-trigger on some turns
- Base Molmo vision/text strengths and limits still apply
Evaluation (high level)
- Attribution / diarization gates (oracle speaker markers): strong text; solid spoken
- Mic hearing and conversational probes are tracked per LoRA tip in the sibling LoRA card and local
hearing_eval_*.json/broad_baseline_matrix.mdartifacts
Numbers move with each Stage D continue; treat Hub “primary tip” docs as approximate until you pin a checkpoint id.
License
CC BY-NC-SA 4.0 (Attribution–NonCommercial–ShareAlike).
- Attribution required
- NonCommercial — no commercial use of these weights without a separate grant
- ShareAlike — adaptations must use the same license
Marketing name: Blaster. Technical Hub id remains molmo-audio-serving-diar-d.
The backbone allenai/Molmo-7B-D-0924 remains Apache-2.0. Some training mixes include ShareAlike sources (e.g. People’s Speech / Dolly); this weight license is intentionally more restrictive (adds NonCommercial).
Citation / credit
If you use Blaster, please credit:
- Blaster (
0x8badbeef/molmo-audio-serving-diar-d), CC BY-NC-SA 4.0 - Base model: AllenAI Molmo-7B-D
- Upstream data licenses as documented in the project training README