Octen-Embedding-8B W4A16
Overview
octen-embedding-8b-w4a16 is a weight-quantized checkpoint intended for efficient GPU inference, published by groxaxo.
It is intended for open-source evaluation, reproducible experimentation, and compatible local or
hosted inference workflows. The wording below is deliberately limited to what can be verified
from this repository's metadata and artifacts.
At a glance
| Field | Details |
|---|---|
| Format | AWQ / AutoRound |
| Source / base | Octen/Octen-Embedding-8B |
| Intended task | feature-extraction |
| License | apache-2.0 |
What is included
*.safetensors(2 files)config.jsongeneration_config.jsontokenizer.jsontokenizer_config.jsonchat_template.jinjaquantization_config.json- Additional configuration, tokenizer, processor, or shard files (10 visible artifacts total)
Quick start
vLLM (AWQ-compatible runtimes)
vllm serve groxaxo/octen-embedding-8b-w4a16 \
--quantization awq_marlin \
--dtype float16 \
--trust-remote-code
The exact kernel and flags depend on the quantizer and architecture. Check the files and source model card before selecting a production serving configuration.
Compatibility and responsible use
- Use a runtime that explicitly supports this format, architecture, and modality.
- Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
- Review the source model card and license before redistribution or deployment.
- Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
- Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.
Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.
Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.
This repo contains a W4A16 quantized version of Octen/Octen-Embedding-8B in the validated auto-round-auto-gptq format.
Quantization
| Item | Value |
|---|---|
| Base model | Octen/Octen-Embedding-8B |
| Quantization | W4A16, 4-bit weights / 16-bit activations |
| Tooling | AutoRound 0.12.2, transformers 5.6.2, torch 2.6.0+cu124 |
| Calibration | 8 samples, seqlen 512, 200 iterations, float32 tuning |
| Quantized size | 8.1 GB, 2 shards |
| Base size | 15.0 GB |
| Compression | ~1.9x |
| Embedding dim | 4096 |
| Layers quantized | 252/253; lm_head skipped |
Validation vs base model
Evaluation used a small retrieval set of 5 query-document pairs, last-token pooling, L2 normalization, and cosine similarity.
| Metric | Base | W4A16 | Delta |
|---|---|---|---|
| Recall@1 | 0.8 | 1.0 | +0.2 |
| Recall@5 | 1.0 | 1.0 | 0.0 |
| Mean query cosine, base vs quant | — | 0.9840 | — |
| Mean doc cosine, base vs quant | — | 0.9820 | — |
Assessment: this model passed all validation gates cleanly, with >0.98 mean cosine to the base model and no retrieval degradation on the validation set.
See validation-8b-auto-round-auto-gptq.json for the raw metrics.
RTX 3060 smoke test
This quantized model was loaded and run on an RTX 3060 12GB GPU.
| Result | Value |
|---|---|
| VRAM after load | 4.53 GB |
| Single short-query forward pass | 0.9s smoke test; later benchmark ~612ms |
| Output shape | [1, 4, 4096] |
| Embeddings | Valid normalized vectors; no NaNs observed |
Recommended usage
import torch
from transformers import AutoModel, AutoTokenizer
model_id = "groxaxo/octen-embedding-8b-w4a16"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.float16,
).cuda().eval()
texts = ["how to implement binary search"]
tokens = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
tokens = {k: v.cuda() for k, v in tokens.items()}
with torch.no_grad():
out = model(**tokens)
emb = torch.nn.functional.normalize(out.last_hidden_state[:, -1, :], p=2, dim=-1)
Note: the model card records local validation and smoke-test results. For production use, evaluate on your own retrieval distribution.