Giga Embeddings 0826 3B GGUF — Russian text embeddings for LM Studio, llama.cpp and Ollama

Русская карточка · GGUF collection · Original model · Smaller 480M GGUF · Original paper

Local Russian and English text embeddings for semantic search, RAG, text similarity, clustering, and classification with LM Studio, stock llama.cpp, and Ollama. This repository contains one BF16 reference and three direct quantizations of the bidirectional Giga Embeddings 3B 0826 model.

Embedding model only: use LM Studio or llama-server POST /v1/embeddings, or Ollama POST /api/embed. This is not a chat or generative model. The Python sentence-transformers package does not load these GGUF files directly.

Start with Q8_0. It is the recommended quality/size default. Choose Q4_K_M only when download size and memory matter most; it is explicitly an experimental lightweight option. BF16 is the high-precision reference. Q6_K is included for research, but is not a better default than Q8_0 or Q4_K_M.

This is an independent ai-babai GGUF conversion, not an official ai-sage release.

Choose a quant in 10 seconds

Apple Silicon uses unified memory. Metal allocation and process RSS are different views of the same shared memory and must not be added together. Peak process RSS in the same Mac runs was 8.21 / 5.20 / 4.45 / 3.84 GB for BF16 / Q8_0 / Q6_K / Q4_K_M. All capacities in this section use decimal GB (1 GB = 10^9 bytes); measured MiB values were converted to GB.

SHA256SUMS · Machine-readable manifest

Quick start

Download the recommended file:

hf download ai-babai/giga-embeddings-0826-3b-gguf \
  giga-embeddings-0826-3b-q8_0.gguf \
  --local-dir .

Run a recent stock llama.cpp server:

llama-server \
  -m giga-embeddings-0826-3b-q8_0.gguf \
  --embeddings \
  -c 2048 -b 2048 -ub 2048 -np 1 \
  --cache-type-k f32 --cache-type-v f32 \
  --flash-attn auto -ngl 99 \
  --host 127.0.0.1 --port 8080

Use -ngl 0 for CPU-only execution.

For retrieval, prepend an instruction to the query and embed documents as plain text:

curl http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-3b-q8_0.gguf",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации.",
      "Париж — столица Франции."
    ]
  }'

Compare the returned normalized 2048-dimensional embeddings with cosine similarity (equivalent to their dot product after normalization).

For symmetric tasks such as semantic similarity or deduplication, use the same instruction for both sides or no instruction. The GGUF metadata selects the required mean pooling; do not replace it with CLS or last-token pooling.

Quick start with LM Studio

In LM Studio 0.4.25+, download or import one of the updated GGUF files in this repository. The refreshed general.name metadata makes LM Studio list it as a Text Embedding model automatically. In the Developer tab, start the local server and load the model. For a downloaded local GGUF, the CLI equivalent is:

lms import -L --user-repo ai-babai/giga-embeddings-0826-3b-gguf -y giga-embeddings-0826-3b-q8_0.gguf
lms server start
lms ls
lms load 'paste-model-id-from-lms-ls-here' -y

Call the OpenAI-compatible embeddings API:

curl http://127.0.0.1:1234/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<model-id-from-LM-Studio>",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации."
    ]
  }'

Use the model ID shown by LM Studio; it depends on how the file was imported. The output has 2048 dimensions. On LM Studio 0.4.25 Build 1, fresh imports of all four updated files appeared as embedding models. For each variant, POST /v1/embeddings returned normalized 2048D vectors with a related passage closer to the query than an unrelated passage. These are functional smoke tests, not new quality benchmarks.

If an older file or cached import still appears as an LLM, replace/re-import it. For a local copy that cannot be replaced yet, open My Models → model → Domain Control → Override Domain Type → Text Embedding. That manual override changes LM Studio's classification of the local model; it is not needed after a fresh import of the updated files on the tested version. Older LM Studio versions may need the override or a runtime update.

Quick start with Ollama

After downloading giga-embeddings-0826-3b-q8_0.gguf, create a Modelfile next to it:

FROM ./giga-embeddings-0826-3b-q8_0.gguf

Import the embedding model and call the embeddings API:

ollama create giga-embeddings-0826-3b-q8 -f Modelfile

curl http://127.0.0.1:11434/api/embed \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-3b-q8",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации."
    ]
  }'

BF16, Q8_0, Q6_K, and Q4_K_M were compatibility-tested with Ollama 0.33.3 on Apple M4 Pro Metal. Each imports as a Qwen3 embedding-only model and returns finite, normalized 2048-dimensional vectors. This was a compatibility smoke test, not an Ollama quality or performance benchmark.

Quality at a glance

The original authors report 74.57 Russian MTEB, 71.93 English MTEB, 76.93 code MTEB, and 63.9 multilingual MTEB for the source BF16 model. Those numbers belong to the original model; we did not rerun the complete MTEB suites for these GGUF files.

Our giga-embeddings-external-retrieval-v1 evaluation used complete pinned RuBQ (e19b6ffa60b3bc248e0b41f4cc37c26a55c2a67b) and SciFact (d56462d0e63a25450459c4f213e49ffdb866f7f9) test splits, instruction-prefixed queries, title + "\n" + text documents, and a 512-token contract. Values below are equal-task macro averages. This is not a complete MTEB run, a leaderboard submission, or a cross-model comparison.

¹The small positive aggregate difference is not evidence that Q6_K improves the model. A separate frozen representation comparison found one reproducible long-code outlier for Q6_K.

The stricter frozen multilingual/code holdout compared each quantized GGUF with the BF16 GGUF:

These agreement numbers measure preservation versus our BF16 GGUF, not absolute retrieval accuracy. All four files passed functional runtime validation. The larger representation drift is why Q4_K_M is labeled experimental even though its full RuBQ+SciFact NDCG loss was small; Q6_K's lower minimum comes from one reproducible long-code sample.

Measured speed

Median total throughput, after two warmups and across five repetitions, context 2048 and parallelism 1:

The Mac had an Apple M4 Pro and 48 GB unified memory. On Metal, BF16 was faster than every quantized variant in the measured matrix; quantization primarily saves storage and memory. Q6_K was also unusually slow on the tested CPU and CUDA paths.

Tested runtimes

Validation used clean stock ggml-org/llama.cpp commit e750b887a82719c27200b71545f63ed78ec24719 (Linux build 10763).

All tested Linux lanes passed llama-embedding, llama-server --embeddings, single/batch/repeat/permutation checks, and finite unit-norm 2048-dimensional output checks. A separate fresh server process reproduced the same eight exact test embeddings on each tested Linux lane (cosine 1.0, maximum component delta 0).

Numerical and scope limitations

• Intended use is dense retrieval/RAG, semantic similarity, clustering, and classification in Russian and English. This is not a generative model or a cross-encoder reranker.
• Functional portability is not bit-identical cross-backend arithmetic. BF16 passed the frozen Mac Metal → CUDA numerical parity gate. Q8_0 and Q6_K retained 8/8 top-1 agreement but narrowly missed its mean-cosine threshold: about 0.99987 observed versus 0.99990 required. Q4_K_M showed larger drift, with about 0.99929–0.99937 mean cosine and 6/8 to 8/8 top-1 agreement across tested lanes.
• The portability sample contains eight exact prompts. Within-backend repeat, permutation, and fresh-process checks passed; the observed differences are backend arithmetic drift, not evidence of corrupt files.
• Runtime/resource testing used context 2048. A 4096-token workload and true cold-cache startup were not benchmarked.
• External retrieval used a 512-token contract; 10.3% of SciFact corpus documents were truncated.
• Ollama preserves Q8_0/Q6_K/Q4_K_M as byte-identical GGUF model layers. Its BF16 import performs an internal same-size COPY rewrite while preserving the model type and metadata; BF16 is functionally compatible but is not executed byte-for-byte after import.
• LM Studio 0.4.25 Build 1 was functionally smoke-tested with all four refreshed files on macOS. Other LM Studio versions and platforms, Windows, Linux Ollama, Jan, and older llama.cpp/Ollama builds were not tested.

Artifact integrity and provenance

• Source: ai-sage/Giga-Embeddings-instruct-3B-0826
• Exact source revision: ed7db5c91b900b39381b27b6e9c0a3d31137cd29
• Source license: MIT
• Source model.safetensors SHA-256: de8519bef7ee360043970b0081088c5294e2a9196ad2d0570a33c4f51bb2e134
• Source tokenizer.json SHA-256: 6fb1280bd7fd529f425929b5df823a5a44485cd7fa9679d1ec1acaac4962e8ca
• Source tokenizer_config.json SHA-256: 843eeba481465c1485a5b5f24bd24d6c12c4e502c16f093c3ab6a0f058c2c5f2
• Converter: local bidirectional-model patch 409723a88b12071974ed5924a2dc1c8b2b2064f7 on top of upstream llama.cpp e750b887a82719c27200b71545f63ed78ec24719
• Clean stock Linux validation runtime: ggml-org/llama.cpp@e750b887a82719c27200b71545f63ed78ec24719, build 10763
• Validation harness: 7ba4e00e76b27316b1b3709476d807958bcd1e9d
• Q8_0, Q6_K, and Q4_K_M were quantized directly from the accepted BF16 GGUF; no cascade or requantization was used
• Architecture preserved: bidirectional Qwen3, 398/398 tensors, mean pooling, 2048-dimensional output

Machine-readable provenance and hashes are also available in manifest.json and SHA256SUMS. The current GGUF revision changes descriptive metadata only; tensor payloads remain byte-identical to the prior release.

Русские текстовые эмбеддинги

Локальная GGUF-версия Giga Embeddings 0826 3B для семантического поиска, RAG, сравнения текстов, кластеризации и классификации на русском и английском языках. Полное описание на русском.

Citation

Please cite both this GGUF release and the original Giga-Embeddings work:

@software{popkov2026gigaembeddingsgguf,
  author  = {Maksim Popkov},
  title   = {Giga Embeddings 0826 GGUF},
  year    = {2026},
  url     = {https://huggingface.co/ai-babai/giga-embeddings-0826-3b-gguf}
}

@misc{kolodin2026gigaembeddings,
  title         = {Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings},
  author        = {Egor Kolodin and Egor Krasnoperov and Evgeniy Kosarev and Fyodor Minkin},
  year          = {2026},
  eprint        = {2608.23806},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.23806}
}