Nemotron-3.5-Lightning-Omni-30B-A3B-GGUF

NVIDIA released Nemotron 3.5 Lightning as a text-only agent model. This repository is Lightning with eyes and ears: the C-RADIO vision tower and the Parakeet audio tower from NVIDIA's own Nemotron-3-Nano-Omni, attached through their unified projector, quantized for llama.cpp, and tested modality by modality before upload. Image, audio, and video understanding are the released, gated capabilities. One-pass video WITH its soundtrack also works zero-shot; there is one prompting note (below) for getting both channels reliably in a single answer.

No training was involved, and that is the interesting part: Lightning shares the exact backbone geometry of the model those projectors were trained for, so perception transfers zero-shot. Details in How This Was Made below.

Model Summary

Quick Start

Download a quant plus the projector, then:

./llama-server -m Nemotron-3.5-Lightning-Omni-30B-A3B-Q4_K_M.gguf --mmproj mmproj-omni-vision-audio-v2-F16.gguf

Open http://localhost:8080 and drag in a photo, a wav, or an mp4. The same endpoint serves /v1/chat/completions with image_url, input_audio, and video content parts.

llama.cpp compatibility

Prebuilt fork binaries for Linux arm64, Linux x64, Win x64:

https://github.com/engram-ae/llama.cpp-omni/releases/tag/b10598

Files

Quantizations land one by one as each passes its gates; this table and TEST-MATRIX.md grow together until all nine tiers are present.

The unified projector mmproj-omni-vision-audio-v2-F16.gguf serves every tier; single-tower use is possible but the unified file is the intended path.

Every quantization in this table passed the text, image, audio, and silent-video gates before upload. The one-pass audio+video column is recorded per quant in TEST-MATRIX.md and is experimental in this revision.

Verification

Claims are cheap; here is what was measured, on a DGX Spark (GB10):

• Five gates per quantization at temperature 0, graded against known ground truths, published in TEST-MATRIX.md. Text, image, and audio must pass for a file to ship; the video columns are recorded honestly and do not block, because video is experimental here.
• One-pass A/V, the measured truth (2026-08-25, corrected twice as evidence came in): first hypothesis was temporal-pathway drift, killed by measurement. Second hypothesis was that audio irreversibly dominates, also killed by measurement. What actually happens: video-only works at every length (2 to 64 frames, matches the donor), and in one pass WITH a soundtrack both channels are fully present and attendable. With a NEUTRAL single-subject prompt ("what is in the video?") zero-shot Lightning tends to answer from the audio and under-report the picture. With a prompt that asks for BOTH, it returns both correctly, every time tested. Two verified examples, temp 0: "Describe what is shown on screen AND transcribe what the narrator says" and "Give me two things: (1) the on-screen visuals, (2) the spoken words" each produced the full visual description AND the full transcription in a single answer. So one-pass A/V is a real, released capability with a prompting note, not a broken feature. The donor Omni, co-trained on interleaved A/V, balances both even at a neutral prompt; a future Lightning projector tune could shift the neutral-prompt default the same way, but it is polish, not a fix for something broken.
• ASR transfer cost, measured: on the 38-word gate narration, the donor Omni model scores 2.6 percent word error rate (one proper noun); this zero-shot assembly scores 7.9 percent (three proper nouns, including the same one the donor misses). Single-clip gate, stated at exactly that scope, not a benchmark.
• Decode speed, measured on the same machine and binary:

The projector adds no decode cost (it runs once at encode). Numbers for this repository's own quantizations land here as they are measured.

• The underlying video and audio inference graphs were validated against NVIDIA's PyTorch reference during the fork's development (video 0.0019 percent relative L2 with exact EVS parity; audio transcribing the reference narration word for word). See the fork repository for that work: https://github.com/engram-ae/llama.cpp-omni

Benchmarks

The language model weights are byte-identical to NVIDIA's release, so NVIDIA's published text benchmarks describe this model's brain at BF16: SWE-bench Verified 51.56, GPQA Diamond 75.44, MMLU Pro 81.94 (their numbers, their card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16). Quantization effects on those scores are not measured here; the BF16 file in this repository is the reference conversion.

Multimodal quality is what this repository adds, and it is gated rather than benchmarked: see TEST-MATRIX.md and the measured ASR note above.

Model Architecture

The assembly, exactly:

How This Was Made (and why no training was needed)

NVIDIA trained the Nano-Omni projectors to translate C-RADIO and Parakeet features into the embedding space of a nemotron_h backbone with hidden size 2688, 52 layers, and a 131072 vocabulary. Nemotron 3.5 Lightning has exactly that geometry: it is a sibling checkpoint of the same architecture family. Attaching the projector to Lightning is therefore a compatible operation, and it works zero-shot: the perception pathway transfers, and the measured cost is two additional word errors on the 38-word ASR gate (see Verification), concentrated in rare proper nouns. Image understanding matches the donor exactly; video is where the drift concentrates, which is why it ships as experimental here.

A projector fine-tune against Lightning specifically is the known path to closing that gap and is planned as a follow-up revision.

Input

Text prompts; images (png/jpg/webp); audio (wav/mp3); video files (mp4 and other ffmpeg-readable containers), with or without a soundtrack. One media file per turn.

Output

Text.

Software Integration

• Runtime: llama.cpp. Full modality support in the llama.cpp-omni fork, release b10598 (prebuilt Linux arm64 CUDA 13 binaries included). Text, image, and audio also run on stock llama.cpp current master with these same files.
• Tested on: NVIDIA DGX Spark (GB10, CUDA 13, sm_121).

License / Terms of Use

• Language model weights: NVIDIA Nemotron 3.5 Lightning, released by NVIDIA under the Open Model, Data and Weights License 1.1 (OpenMDW-1.1): https://openmdw.ai/license/1-1/
• Encoder towers and projector weights: lifted from NVIDIA Nemotron-3-Nano-Omni, governed by the NVIDIA Open Model License terms of that release.
• The GGUF conversion, the quantizations, the unified projector packaging, and the llama.cpp video/one-pass graphs are my work; the underlying weights remain NVIDIA's under their respective licenses.

Model Version(s)

• v1 (2026-08): zero-shot assembly, nine quantizations, five-gate matrix.
• Planned v1.1: Lightning-specific projector fine-tune (closes the measured ASR proper-noun gap and the experimental-video seam).

Known Limitations

• One-pass audio+video works, with a prompting note: a neutral single-subject prompt may answer from the audio and under-report the picture. Ask for both channels explicitly (see Verification) and it returns both correctly. Video-only and audio-only are unconditionally reliable.
• ASR smudges rare proper nouns more than the donor Omni does (measured above). General transcription is faithful.
• Multimodal gates are functional checks, not benchmark suites. Treat TEST-MATRIX.md as exactly what it is: proof each file works, per modality, on real inputs.
• Video and one-pass A/V require the fork binaries until the planned upstream PR lands.

Release Date

2026-08 (Hugging Face).