Nemotron-3-Embed-8B — Community NVFP4

Unofficial community quantization — not an NVIDIA release.

NVFP4 build of nvidia/Nemotron-3-Embed-8B-BF16 (revision 8ca3ff38), the top-ranked open embedding model on the RTEB leaderboard at time of writing. All credit for the base model belongs to NVIDIA; this repo only changes the weight storage format. NVFP4 is Blackwell's native 4-bit format — ~3.5× smaller than BF16 with FP4 tensor-core execution on SM100/SM120 GPUs — and on the benchmarks where this model leads the leaderboard, the quantized scores land within 0.008 of NVIDIA's own published numbers.

Pick this variant for Blackwell GPUs where memory is the constraint. The companion FP8 build is closer to lossless and runs natively on Ada/Hopper too; the MLX 4-bit build serves Apple-Silicon Macs.

Benchmarks vs the unquantized model

Comparison column = NVIDIA's official per-task results from the mteb results repo — their numbers, not our reproduction. Our runs: mteb 2.18.12, vLLM 0.26.0, mean pooling, query: /passage: prefixes, max length 8192 (NVIDIA evaluated at 4096; few documents in these tasks exceed either limit). Tasks are the open (public) RTEB datasets in the domains where the base model ranks top-4 on the RTEB leaderboard: Finance #1, German #1, Code #2, Healthcare #4, Legal #4.

Mean delta −0.0015 across all 12 tasks; −0.0017 on the 10-task subset shared by all three community builds (the two AILA legal tasks were run only on the CUDA builds). The private RTEB datasets can only be run by the MTEB team, so this table covers the open subset.

Regression vs our own BF16 baseline (identical harness both sides)

BF16 baseline computed with the same code, adapter, prefixes, and pins on an A100. Gate: per-task nDCG@10 loss ≤ 0.01.

* TRECCOVID has 50 queries — the highest-variance task in the suite. The FP8 companion dips on the same task (−0.0076), so the dip is shared across quant formats and amplified by 4-bit; we report it as measured rather than excluding it. Embedding-level fidelity vs BF16 on token-ID-locked fixtures: cosine 0.9901–0.9930. Raw result JSON ships under results/.

Serving with vLLM

Requires a Blackwell GPU (SM100/SM120) for native FP4 execution.

from vllm import LLM
from vllm.config import PoolerConfig

llm = LLM(
    model="shadowrock-io/Nemotron-3-Embed-8B-Community-NVFP4",
    runner="pooling",
    pooler_config=PoolerConfig(seq_pooling_type="MEAN"),  # default LAST is silently wrong
    max_model_len=8192,
)
out = llm.embed(["query: what is NVFP4?", "passage: Blackwell's native 4-bit format."])

Required patch for vLLM ≤ 0.26.0: vLLM's pooling adapter replaces the checkpoint's absent lm_head with a placeholder layer, and ModelOptNvFp4LinearMethod.process_weights_after_loading crashes on the placeholder's meta tensors. Run scripts/patch_modelopt_guard.py once against your vLLM install before loading (idempotent; an upstream fix has been proposed).

Notes that matter for correct embeddings:

• Pooling must be MEAN and attention is bidirectional; both come from the checkpoint config, but the pooler override above guards against defaults.
• Prefixes are your job: query: / passage: . The server does not add them.
• Texts longer than max_model_len are rejected by vLLM's pooling runner — truncate at the tokenizer (truncation=True, max_length=8192) and pass token IDs.
• Embeddings are 4096-dim; L2-normalize before use. Matryoshka truncation (2048/1024): slice, then re-normalize.

Measured on: GeForce RTX 5070 Ti (SM120) and RTX PRO 6000 (Blackwell), vLLM 0.26.0, CUDA 12.8.

Quantization details

• Method: NVIDIA TensorRT Model Optimizer (ModelOpt) NVFP4 post-training quantization — FP4 weights with per-block FP8 scales, MLP-focused config per current ModelOpt accuracy guidance; embeddings, norms, and pooling untouched. Full module inventory: quantization/module_inventory.json.
• Derived in a fresh process from the pinned BF16 snapshot.
• Calibration: ~1k public samples from MS MARCO and MIRACL train splits, token-bucketed (32–16k tokens) with real prefix distribution. MS MARCO is research-licensed, so the manifest ships dataset IDs + a deterministic builder script, not text. Eval-set contamination audit (by ID and content hash) included; Swahili and Telugu were held out of calibration entirely, with Swahili evaluated as the regression suite's held-out language.
• Quantize/eval scripts ship under scripts/; raw eval JSON under results/.

Caveats

• "Blackwell-native format" is a statement about the artifact; verified FP4 kernel execution is claimed only for the GPUs and vLLM build listed above.
• TRECCOVID's marginal delta is disclosed above rather than averaged away. If your workload resembles high-variance biomedical retrieval, prefer the FP8 build.
• Multilingual evidence: one held-out language (Swahili, hard-negatives variant) plus German LegalQuAD. Telugu was also held out of calibration but not evaluated; full-corpus MIRACL was excluded for compute cost.

Intended use & limitations

Intended uses are the base model's: dense retrieval, semantic search, and RAG indexing over text corpora, with query: /passage: prefixed inputs. The base card's intended-use, safety, and language-coverage statements — nvidia/Nemotron-3-Embed-8B-BF16 — carry over unchanged; quantization alters none of the model's behavior boundaries, only its numeric precision. Our evaluation establishes parity on the benchmarks listed above and nothing beyond them: other languages, domains, sequence-length regimes, and hardware/runtime combinations inherit the base model's behavior with quantization noise that we have not measured there.

Attribution & citation

Quantization, validation harness, and card by Matt Busi (@mattbusi on Hugging Face) at ShadowRock. If you use this build, cite the NVIDIA base model — the embedding quality is theirs:

@misc{nvidia2026nemotron3embed,
  title  = {Nemotron-3-Embed-8B},
  author = {NVIDIA},
  year   = {2026},
  url    = {https://huggingface.co/nvidia/Nemotron-3-Embed-8B-BF16}
}

License

OpenMDW-1.1, inherited from the base model (see LICENSE). NOTICE carries the upstream Apache-2.0 attribution for the Ministral component plus our modification statement. Community build by ShadowRock; no NVIDIA affiliation or endorsement.

About ShadowRock

ShadowRock is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at shadowrock.io or on LinkedIn.