DeepX Embedding v1.0
Vietnamese Legal Document Retrieval — State-of-the-Art
DeepX Embedding v1.0 is a 772M parameter embedding model optimized for Vietnamese legal document retrieval. It combines Gated DeltaNet-2 linear attention (O(n)) with Hyperloop weight sharing to achieve strong retrieval quality while maintaining constant throughput regardless of sequence length.
nDCG@10 = 0.8162 on Zalo Legal Text Retrieval — surpassing previous SOTA (0.7813) by +4.5%.
Benchmark Results
| Model | Params | nDCG@10 |
|---|---|---|
| intfloat/multilingual-e5-large | 560M | 0.6660 |
| mainguyen9/vietlegal-e5 | 560M | 0.7310 |
| mainguyen9/vietlegal-harrier-0.6b (prev SOTA) | 600M | 0.7813 |
| DeepX Embedding v1.0 | 772M | 0.8162 |
Key Features
- Linear attention O(n) — Gated DeltaNet-2: processes 8K tokens with same VRAM as 512 tokens
- Hyperloop architecture — 35 compute passes from only 9 unique layer parameter sets
- Matryoshka embeddings — Quality at any dimension from 256d to 1536d
| Dimension | nDCG@10 | Quality vs Full |
|---|---|---|
| 256 | 0.78 | ~96% |
| 512 | 0.79 | ~97% |
| 768 | 0.80 | ~98% |
| 1024 | 0.81 | ~99% |
| 1536 (full) | 0.8162 | 100% |
- ColBERT dual output — Single vector (1536d) for ANN search + token vectors (128d) for MaxSim reranking
- Custom vocabulary — 186,046 tokens optimized for Vietnamese + English
- YaRN RoPE — 8K tokens validated, 128K supported
Architecture
Input text
→ Custom Tokenizer (186,046 vocab)
→ Frozen Token Embedding (186046 × 1536)
→ Begin Block: 4 unique NarrowA layers
→ Phase1 Loop ×2: [WideA + NarrowA×4] per iteration = 10 passes
→ Phase2 Loop ×4: [NarrowB×4 + WideB] per iteration = 20 passes
→ End Block: 1 unique WideB layer
→ RMSNorm → Attention Pooling → 1536-d vector
Total: 35 compute passes. Per-loop LoRA + RoDE (Rotary Depth Embedding) differentiate each iteration.
Model Size
| Component | Parameters |
|---|---|
| Token Embedding (frozen) | 286M |
| Backbone (trainable) | 486M |
| Total | 772M |
Gated DeltaNet-2 (GDN-2)
Pure linear attention with O(n) complexity. Each layer maintains a running state updated via learned decay, erase, and write gates:
state_t = decay_t * state_{t-1}
state_t -= erase_t * (erase_t @ state_t - write_t * v_t)
output_t = q_t @ state_t
No KV cache, no quadratic slowdown. Uses FLA (flash-linear-attention) Triton kernels for efficient chunk-parallel training.
Training
| Setting | Value |
|---|---|
| GPUs | 2× RTX 5070 Ti 16GB (pipeline parallel) |
| Precision | BF16 |
| Optimizer | AdamW 8-bit |
| Sequence length | 8192 max |
| Loss | InfoNCE (τ=0.07) + Matryoshka (256, 512, 768, 1024, 1536) |
| Total training | ~600 GPU-hours |
Training pipeline: conservative training → long-sequence expose (4K-8K) → hard negative mining → domain boost.
Usage
import torch
from transformers import AutoTokenizer
from modeling.pipeline import DeepXPipeline
from config import DeepXConfig
# Load
tokenizer = AutoTokenizer.from_pretrained("dxtech-asia/deepx-embedding-v1")
config = DeepXConfig()
pipeline = DeepXPipeline.from_pretrained(config, "deepx_v1.0.pt")
pipeline.eval().cuda()
# Encode
text = "Mức phạt khi vượt đèn đỏ là bao nhiêu?"
inputs = tokenizer(text, return_tensors="pt", max_length=8192, truncation=True)
id_remap = torch.load("id_remap.pt")
input_ids = id_remap[inputs["input_ids"]]
with torch.no_grad():
embedding = pipeline.encode(input_ids.cuda(), inputs["attention_mask"].cuda())
# Shape: (1, 1536), L2-normalized
Inference Speed
| Sequence Length | Latency (single doc) |
|---|---|
| 512 tokens | ~0.1s |
| 2048 tokens | ~0.2s |
| 8192 tokens | ~0.8s |
On RTX 5070 Ti, FP16 inference.
Citation
@misc{deepx2026embedding,
title={DeepX Embedding v1.0: Vietnamese Legal Retrieval with Gated DeltaNet-2 Linear Attention},
author={DX Tech Asia},
year={2026},
url={https://huggingface.co/dxtech-asia/deepx-embedding-v1}
}
License
Apache 2.0