Giga Embeddings 0826 480M — GGUF for llama.cpp, Ollama, and LM Studio

Русская карточка · GGUF collection · Original model · Larger 3B GGUF · Original paper

Compact local Russian and English text embeddings for semantic search, RAG, text similarity, clustering, and classification. This repository contains a BF16 reference and three direct GGUF quantizations of the bidirectional Giga Embeddings 0826 480M encoder for stock llama.cpp, Ollama, and LM Studio.

Start with Q8_0. It is the recommended default and keeps quality closest to BF16 while cutting the download to 0.520 GB. Use BF16 for maximum fidelity. Q6_K is an alternative compact format for users who specifically need Q6, but it is not recommended over Q8_0. Q4_K_M is the minimum-size experimental option and has measurable representation and retrieval loss.

This is an independent ai-babai GGUF conversion, not an official ai-sage release.

LM Studio compatibility update (September 2026): the GGUF files have updated general.name and general.description metadata so LM Studio 0.4.25 recognizes them as text embeddings. Tensor bytes are unchanged. Download the current files instead of reusing a cached older revision.

Choose a quant in 10 seconds

Apple Silicon uses unified memory. Metal allocation and process RSS are different views of the same shared memory and must not be added. The measured Metal peak RSS values were 2.465 / 2.011 / 1.885 / 1.814 GB for BF16 / Q8_0 / Q6_K / Q4_K_M. All capacities use decimal GB (1 GB = 10^9 bytes).

SHA256SUMS · Machine-readable manifest

Quick start with llama.cpp

Download the recommended file:

hf download ai-babai/giga-embeddings-0826-480m-gguf \
  giga-embeddings-0826-480m-q8_0.gguf \
  --local-dir .

Run a recent stock llama.cpp server:

llama-server \
  -m giga-embeddings-0826-480m-q8_0.gguf \
  --embeddings \
  -c 2048 -b 2048 -ub 2048 -np 1 \
  --cache-type-k f32 --cache-type-v f32 \
  --flash-attn auto -ngl 99 \
  --host 127.0.0.1 --port 8080

Use -ngl 0 for CPU-only execution. Embed an instruction-prefixed query and plain documents through the OpenAI-compatible endpoint:

curl http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-480m-q8_0.gguf",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации.",
      "Париж — столица Франции."
    ]
  }'

The server returns normalized 1024-dimensional embeddings. Compare them with cosine similarity, or equivalently with a dot product after normalization.

Quick start with Ollama

Create Modelfile next to the downloaded GGUF:

FROM ./giga-embeddings-0826-480m-q8_0.gguf

Import it and call the embeddings API:

# Start this in a separate terminal if the Ollama app/service is not running:
ollama serve

ollama create giga-embeddings-0826-480m-q8 -f Modelfile

curl http://127.0.0.1:11434/api/embed \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-480m-q8",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации."
    ]
  }'

The four files were tested with Ollama 0.33.3 on Apple M4 Pro Metal. They expose the embedding capability only; they are not generative chat models.

Quick start with LM Studio

Use LM Studio 0.4.25 or later and import a current GGUF from this repository. In the tested 0.4.25 Build 1, the refreshed GGUF metadata lets the app identify the model as Text Embedding automatically. The model name begins with Qwen3 Embedding; the model remains a bidirectional embedding encoder, not a chat model. The lmstudio repository tag helps discovery on Hugging Face but does not control runtime classification.

Load the model and start LM Studio's local server. The OpenAI-compatible API is:

curl http://127.0.0.1:1234/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-480m-q8_0",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации."
    ]
  }'

Use the model identifier shown by LM Studio's local server if it differs from the example. The response contains 1024-dimensional embeddings. Query prompts, mean pooling, and normalization follow the same guidance below for every runtime.

If an older download still appears under LLMs, remove the old local copy and import the updated GGUF (check its SHA-256 below). LM Studio may retain its previous local classification; open Domain Control → Override Domain Type → Text Embedding if needed. LM Studio releases before 0.4.25 were not verified here.

Prompt, pooling, and normalization

For retrieval and other asymmetric tasks, add a task instruction to the query and embed documents without it:

Instruct: {one-sentence task description}
Query: {query text}

For symmetric tasks such as semantic similarity or deduplication, use the same instruction for both sides or no instruction. The GGUF metadata selects the required mean pooling and L2 normalization; do not substitute CLS or last-token pooling.

These .gguf files are intended for llama.cpp, Ollama, or LM Studio. Python SentenceTransformer does not load them directly.

Quality at a glance

The original authors report 70.98 Russian MTEB, 69.52 English MTEB, 72.87 code MTEB, and 56.97 multilingual MTEB for the source BF16 model. Those scores belong to the upstream model; we did not rerun the complete MTEB suites for these GGUF files.

Our full pinned evaluation used all 1,692 RuBQ and 300 SciFact test queries, instruction-prefixed queries, title + "\n" + text documents, a 512-token limit, and equal weighting of the two tasks.

¹Small positive differences are normal ranking variation and are not evidence that quantization improves the model. This evaluation is not a complete MTEB run, a leaderboard submission, or a cross-model comparison.

The stricter frozen RU/EN/code/multilingual holdout compared each quantized GGUF with this repository's BF16 GGUF across 512 aligned texts and a 256-query/2,048-document ranking set:

Q8_0 is the revision 2 recommended default after explicit review of both full retrieval and the frozen holdout. The original default-role rule recorded one English top-1 disagreement: 63/64 (98.4375%) versus its predeclared 99.0% family floor. Forensic review found that sole changed top result beneficial on the labelled holdout: Q8 selected a qrel-relevant SciFact document where BF16 did not. The measurement and threshold miss remain disclosed; the revision 2 role is a product decision, not a rewritten test result.

Q6_K has no clear quality/size advantage over Q8_0 and shows broader representation/ranking drift. Q4_K_M is published for minimum-size use with a clear experimental label: its full aggregate NDCG loss is 0.6682 points, its SciFact loss is 1.2942 points, and its holdout drift is substantially larger.

Measured speed

Median total throughput after two warmups and five measured repetitions, context 2048 and parallelism 1:

Quantization primarily saves storage and memory on the tested Metal backend; it was not a universal speedup. CUDA resource throughput and peak VRAM were not measured for the 480M release and are intentionally not estimated.

Tested runtimes

Validation used clean stock ggml-org/llama.cpp commit e750b887a82719c27200b71545f63ed78ec24719 (build 10763).

The llama.cpp/Ollama rows are the original release validation. The LM Studio row checks this metadata revision; the original full retrieval benchmarks were not repeated because every tensor remains byte-identical.

All tested lanes returned finite, unit-normalized 1024-dimensional embeddings and passed single/batch, repeat, permutation, and fresh-process checks in the llama.cpp/Ollama validation. Remote Linux copies matched the local SHA-256 hashes exactly. LM Studio checks cover model classification and its embeddings API; they do not repeat the full retrieval benchmark.

Limitations and numerical caveats

• This is a dense embedding model for retrieval, similarity, clustering, and classification. It is not a generative model or a cross-encoder reranker.
• BF16 is the frozen reference, not bit-identical to upstream PyTorch CPU/F32. The comparison recorded minimum aligned cosine 0.99782324 and maximum absolute score delta 0.00982617, concentrated in one Spanish 768-token sample; 511/512 aligned samples cleared the former 0.9985 floor.
• Cross-backend arithmetic is not bit-identical. Every artifact was stable within each backend, but the three quants exceeded a deliberately strict eight-prompt Mac-Metal-to-Linux diagnostic. This is a portability diagnostic, not corruption or a functional-runtime failure.
• In Ollama, Q8_0/Q6_K/Q4_K_M import as the exact GGUF blobs. Ollama 0.33.3 rewrote BF16 into a byte-different COPY layer of the same size with preserved model metadata, so it did not execute the frozen BF16 file byte-for-byte.
• Resource testing used context 2048. Full external retrieval used a 512-token contract; 394/56,826 RuBQ documents and 534/5,183 SciFact documents were truncated. Queries were not truncated.
• Windows, Jan, LM Studio before 0.4.25, and older llama.cpp/Ollama builds were not tested.

The LM Studio revision changes only GGUF general.name and adds general.description. Tensor bytes and all other model metadata are preserved. Previous repository revisions can remain cached locally and lack automatic Text Embedding recognition.

Artifact integrity and provenance

• Source: ai-sage/Giga-Embeddings-instruct-480M-0826
• Exact source revision: 2d0c1a92716eef0e5b6972df85b5883eb5b4f57a
• Source license: MIT
• Source model.safetensors SHA-256: 9ce03c6c5ae02baebb42ce3015b6f3e628c5fec7b7745bc2490f6ff961a654a5
• Source tokenizer.json SHA-256: 0870f9c0f06a677f7e939e7765ae3a5d9582f2a6d7e7258176ecd1a3eab96cef
• Source tokenizer_config.json SHA-256: 57981f331fd010c7be87f58e7570b2061dd14d98be53063ad8972e9c4db0b644
• Converter: bidirectional-model patch 409723a88b12071974ed5924a2dc1c8b2b2064f7 over upstream llama.cpp e750b887a82719c27200b71545f63ed78ec24719
• Validation workbench: 30272b4a0c2b52037a0134f9638d3a329d069e8b
• Q8_0, Q6_K, and Q4_K_M were quantized directly from the frozen BF16 GGUF; no cascade or requantization was used.
• Architecture preserved: bidirectional Qwen3, 310 tensors, mean pooling, 1024-dimensional L2-normalized output.

Machine-readable release metadata is available in manifest.json; package hashes are listed in SHA256SUMS.

Русские текстовые эмбеддинги

Локальная GGUF-версия Giga Embeddings 0826 480M для семантического поиска, RAG, сравнения текстов, кластеризации и классификации на русском и английском языках. Полное описание на русском.

Citation

Please cite both this GGUF release and the original Giga-Embeddings work:

@software{popkov2026gigaembeddings480mgguf,
  author  = {Maksim Popkov},
  title   = {Giga Embeddings 0826 480M GGUF},
  year    = {2026},
  url     = {https://huggingface.co/ai-babai/giga-embeddings-0826-480m-gguf}
}

@misc{kolodin2026gigaembeddings,
  title         = {Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings},
  author        = {Egor Kolodin and Egor Krasnoperov and Evgeniy Kosarev and Fyodor Minkin},
  year          = {2026},
  eprint        = {2608.23806},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.23806}
}