Ling-3.0-flash-VL NVFP4 — GGUF
NVFP4 GGUF conversion of Ling-3.0-flash-VL, a next-generation native multimodal model from InclusionAI.
This repository provides the NVFP4 (NVIDIA 4-bit floating-point) GGUF files, making the model usable with llama.cpp and other GGUF-compatible inference engines.
Model Overview
Ling-3.0-flash-VL is a native multimodal model built upon Ling-3.0-flash. It integrates visual information into the complete process of understanding, reasoning, acting, and verification, advancing beyond simple perception to solving real-world tasks through vision.
The model features a sparse MoE architecture with 124B total parameters, but only 5.5B activated parameters per token, delivering powerful multimodal reasoning with exceptional efficiency.
Model Architecture
| Property | Value |
|---|---|
| Architecture | Sparse MoE (KDA & Gated MLA) |
| Total Parameters | 124B |
| Active Parameters | 5.5B |
| Context Length | 256K tokens |
| Visual Encoder | ViT with MLP Projector |
| Temporal Encoding | VideoRoPE (Spatial + Temporal) |
| Backbone | 42-layer hybrid (KDA/MLA 5:1 ratio) |
| Supported Modalities | Text, Image, Video |
GGUF Files
| File | Format | Description |
|---|---|---|
ling-3.0-flash-vl-nvfp4.gguf | NVFP4 | NVFP4 quantized model |
mmproj.gguf | — | Multimodal projector (vision) |
A chat_template.jinja file is also provided for use with chat-based inference.
About NVFP4: NVFP4 is NVIDIA's 4-bit floating-point format (E2M1) with microscaled per-block scaling factors. It offers significant memory savings while preserving quality close to bfloat16. NVFP4 inference is accelerated on NVIDIA Blackwell GPUs and supported via llama.cpp on compatible hardware.
Note on mmproj: The
mmproj.gguffile contains the visual projector needed for multimodal (image/video) inference.
Usage
llama.cpp (CLI)
# Run text-only inference
./llama-cli \
-m ling-3.0-flash-vl-nvfp4.gguf \
-p "Explain the significance of multimodal LLMs." \
--temp 0.6 --top-k 20 --top-p 0.95
llama-server (OpenAI-compatible API)
# Text-only
./llama-server \
-m ling-3.0-flash-vl-nvfp4.gguf \
--host [IP_ADDRESS] --port 8080
# Multimodal (image + video)
./llama-server \
-m ling-3.0-flash-vl-nvfp4.gguf \
--mmproj mmproj.gguf \
--host [IP_ADDRESS] --port 8080
Multimodal (Image / Video)
For image and video inputs, use llama-server or llama-cli with the --mmproj flag pointing to mmproj.gguf. Refer to your inference engine's documentation for passing visual data alongside text prompts.
Generation Parameters
Recommended parameters based on the official configuration:
| Parameter | Value | Note |
|---|---|---|
| Temperature | 0.6 | 1.0 recommended for some generative tasks |
| Top-K | 20 | |
| Top-P | 0.95 |
Thinking Mode
Ling-3.0-flash-VL has thinking mode enabled by default via the chat template.
- Output format: When thinking is enabled, the model outputs internal reasoning (chain-of-thought) followed by the final answer.
- Control: Thinking can typically be disabled per request via the
chat_template_kwargs(e.g.,"enable_thinking": false).
Key Features
- Native Multimodal: Deep integration of vision into reasoning and acting.
- Efficient Inference: MoE architecture activates only 5.5B parameters per token.
- Long Context: Support for up to 256K tokens.
- Video Understanding: VideoRoPE allows the model to understand temporal changes and event localization.
- Agentic Capabilities: Designed to interact with interfaces and translate visual info into action sequences.
Acknowledgements
- Original model: inclusionAI/Ling-3.0-flash-VL
License
This project is licensed under the MIT License.