TinyE5-L6-384
TinyE5-L6-384 is a compact 384-dimensional text embedding model built from
sentence-transformers/all-MiniLM-L6-v2 and fine-tuned for semantic search and
information retrieval using E5-style query/passage prefixes.
The model uses mean pooling + L2 normalization and is designed for applications where model size, CPU latency, and deployment efficiency matter.
Highlights
- 6 Transformer layers
- 384-dimensional embeddings
- 22.7M parameters
- Only the last 2 Transformer layers were fine-tuned
- Mean pooling
- L2-normalized embeddings
- E5-style
query:andpassage:prefixes - FP32 and dynamically quantized INT8 ONNX deployment
- INT8 ONNX size: 21.8 MB
- Measured INT8 CPU throughput: 377.7 texts/s
Model Architecture
| Property | Value |
|---|---|
| Base model | sentence-transformers/all-MiniLM-L6-v2 |
| Transformer layers | 6 |
| Hidden / embedding size | 384 |
| Total parameters | 22,713,216 |
| Fine-tuned layers | Last 2 Transformer layers |
| Trainable parameters | 3,548,928 |
| Trainable percentage | 15.62% |
| Pooling | Mean pooling |
| Normalization | L2 normalization |
| Query prefix | query: |
| Passage prefix | passage: |
| Training max length | 128 tokens |
Benchmark Results
Quality
| Model | Size | STS Spearman ↑ | SciFact Recall@10 ↑ | SciFact nDCG@10 ↑ |
|---|---|---|---|---|
| TinyE5-L6-384 (Safetensors) | 86.7 MB | 0.8138 | 0.7342 | 0.5897 |
| TinyE5-L6-384 (FP32 ONNX) | 86.2 MB | 0.8138 | 0.7342 | 0.5897 |
| TinyE5-L6-384 (INT8 ONNX) | 21.8 MB | 0.8058 | 0.7599 | 0.6020 |
CPU Inference Performance
| Model | Precision | Size | Latency ↓ | Throughput ↑ | Session RAM ↓ |
|---|---|---|---|---|---|
| TinyE5-L6-384 ONNX | FP32 | 86.2 MB | 2.35 ms/text | 424.7 texts/s | 90.4 MB |
| TinyE5-L6-384 ONNX | INT8 | 21.8 MB | 0.77 ms/text | 1299.4 texts/s | 11.2 MB |
INT8 vs FP32 ONNX
| Metric | FP32 | INT8 | Change |
|---|---|---|---|
| Model size | 86.2 MB | 21.8 MB | ~74.7% smaller |
| CPU latency | 2.35 ms/text | 0.77 ms/text | ~67.2% lower |
| CPU throughput | 424.7 texts/s | 1299.4 texts/s | ~205.9% higher |
| Session RAM | 90.4 MB | 11.2 MB | ~87.6% lower |
| STS Spearman | 0.8138 | 0.8058 | -0.0080 |
| SciFact Recall@10 | 0.7342 | 0.7599 | +0.0257 |
| SciFact nDCG@10 | 0.5897 | 0.6020 | +0.0123 |
End-to-End Benchmark Time
| Model | STS Time ↓ | SciFact Time ↓ |
|---|---|---|
| TinyE5-L6-384 (Safetensors) | 0.72s | 3.71s |
| TinyE5-L6-384 (FP32 ONNX) | 8.82s | 265.58s |
| TinyE5-L6-384 (INT8 ONNX) | 5.46s | 179.11s |
Summary: INT8 quantization reduces TinyE5-L6-384 from 86.2 MB to 21.8 MB (~74.7% smaller), lowers CPU inference latency from 2.35 ms/text to 0.77 ms/text (~67.2% lower), and increases CPU throughput from 424.7 to 1299.4 texts/s (~3.06× throughput). Session RAM also drops from 90.4 MB to 11.2 MB (~87.6% lower). STS Spearman decreases slightly from 0.8138 to 0.8058, while SciFact Recall@10 improves from 0.7342 to 0.7599 and nDCG@10 improves from 0.5897 to 0.6020. The all-MiniLM-L6-v2 baseline still achieves the highest overall quality, with 0.8194 STS Spearman, 0.7923 Recall@10, and 0.6494 nDCG@10.
Docker Example
INT8 ONNX
services:
embedding-server:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.9
ports:
- "80:80"
volumes:
- ./data:/data
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- int8-onnx
- --pooling
- mean
Available Variants
All variants are published under the same Hugging Face repository:
GrowBitLabs/tinye5
| Variant | Revision | Model |
|---|---|---|
| Safetensors FP32 | main | model.safetensors |
| ONNX FP32 | fp32-onnx | onnx/model.onnx |
| ONNX INT8 | int8-onnx | onnx/model.onnx |
Safetensors
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- main
- --pooling
- mean
FP32 ONNX
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- fp32-onnx
- --pooling
- mean
INT8 ONNX
command:
- --model-id
- GrowBitLabs/tinye5
- --revision
- int8-onnx
- --pooling
- mean
For CPU deployment, the INT8 ONNX revision is recommended when model size and inference throughput are the priority.
Attribution
Developed by GrowBit Labs.