Sarvam-30B — Silent Marlin (W4A16) · Resilient AI Challenge, Text-to-Text
Compressed submission of sarvamai/sarvam-30b for the Resilient AI Challenge.
Goal: lowest inference energy on A100 while keeping ≥80% recovery per category.
1. Compression approach + benchmark scores
Method: stored-weight W4A16 (4-bit weights, 16-bit activations), compressed-tensors
pack-quantized, group size 128, symmetric int4. Produced with llmcompressor
(RTN; data-free). Serves natively in vLLM via the Marlin / WNA16-MoE kernels — no
runtime dequant overhead.
Quality-aware ignore-list (kept at BF16): every attention query_key_value and
dense, the layer-0 dense FFN, all MoE router gates (mlp.gate), all shared experts,
and lm_head. Only the 128 routed-expert projections (the bulk of the weights) are
quantized. This preserves routing fidelity and multilingual quality.
Size: ~64 GB (BF16) → ~20 GB (−69%).
Measured per-category quality (this model, official sampling temp 1.0 / top_p 0.95 / top_k 64, vLLM, 2 passes), vs the per-category 80% floors:
| Category | Recovery (acc) | 80% floor | Pass |
|---|---|---|---|
| Math (incl. multi-step) | 1.00 | 0.738 | ✅ |
| Medical | 1.00 | 0.523 | ✅ |
| Writing | 0.81 | 0.618 | ✅ |
| Questions (MCQ) | 1.00 | 0.755 | ✅ |
| Indic (5 languages) | 1.00 | 0.60 | ✅ |
The original Sarvam reasoning behaviour is retained: we tested an aggressive
"no-thinking" response-economy template (assistant <think></think> prefill +
brevity instruction) that cut medical/writing tokens 50–150×, but it dropped Math to
0.57 and MCQ to 0.50 — below floor. Sarvam-30B genuinely needs its reasoning
scratchpad for math and multiple-choice, so we ship the unmodified chat template
and take the energy win purely from 4-bit weights.
2. vllm_config.yaml — every argument explained
model: . # served from the repo root (positional model wins)
trust_remote_code: true # SarvamMoE ships a custom config/model class
max_model_len: 32768 # full useful context; not reduced (organizer warns against low values)
gpu_memory_utilization: 0.9
No served-model-name (a set value makes vLLM answer ONLY to that string and return
404 — silent zero — for URI-addressed requests). No tensor-parallel-size,
swap_space, or no-enable-log-outputs (infrastructure-specific; per FAQ).
3. Known issues at launch / inference
- vLLM version vs driver: vLLM 0.20.2 pins torch 2.11 whose default wheel targets CUDA 13 (driver ≥580). Validation was run on vLLM 0.19.1 + torch 2.10 (cu128) because all available GPUs had driver 570. The weights are engine-version-agnostic (standard compressed-tensors); on the evaluation A100 with a ≥580 driver, vLLM 0.20.2 serves them directly. SarvamMoE has been supported in vLLM since 0.18.0 (PR #33942).
- Energy was measured by integrating
nvidia-smipower at 1 Hz over the run; absolute joules carry environment error bars — token counts are the transferable quantity.
License
Apache 2.0 (inherited from sarvamai/sarvam-30b).