Echo-DSRN Embedding Model (Echo-DSRN-v0.1.3-Embed-Exp)
[!WARNING] This repository contains experimental models designed strictly for academic evaluation and research purposes.
Critical Constraints:
- No Production Deployment: Experimental models must not be deployed in commercial, enterprise, or mission-critical environments under any circumstances.
- No Liability: Experimental models are provided "as-is" without warranties of any kind. The developers assume zero liability for downstream consequences, system integration failures, or regulatory non-compliance resulting from unauthorized deployment.
This is a high-performance experimental sentence embedding model based on the recurrent-hybrid Echo-DSRN architecture.
It scales linearly ($O(N)$) with sequence length, offering extreme efficiency and sub-millisecond latency on both CPU and GPU.
🚀 Model Details
- Developer: ethicalabs
- Base Model: ethicalabs/Echo-DSRN-114M-v0.1.2
- Architecture: Echo-DSRN Recurrent-Hybrid
- Parameters: 98.26M (98,264,064)
- Embedding Dimension: 2048
📊 Evaluation Results (MTEB STS)
Spearman Rank Correlation scores on Semantic Textual Similarity (STS) benchmark tasks:
| Benchmark Task | Echo-DSRN (Ours) |
|---|---|
| STS12 | 0.6667 |
| STS13 | 0.7692 |
| STS14 | 0.7683 |
| STS15 | 0.8227 |
| STS16 | 0.7460 |
| STSBenchmark | 0.7293 |
| SICK-R | 0.7876 |
| Average STS | 0.7557 |
⚡ Efficiency and Systems Scaling Profile
Inference performance (latency and peak VRAM allocation) on GPU and CPU configurations across different sequence lengths:
GPU Latency & VRAM Benchmark
| Sequence Length | Echo-DSRN Latency (GPU) | Echo-DSRN VRAM (GPU) |
|---|---|---|
| 128 | 15.93 ms | 516.69 MB |
| 256 | 17.56 ms | 548.44 MB |
| 512 | 32.14 ms | 604.95 MB |
| 1024 | 71.30 ms | 710.96 MB |
| 2048 | 155.26 ms | 932.99 MB |
| 4096 | N/A (OOR) | N/A (OOR) |
CPU Latency Benchmark
| Sequence Length | Echo-DSRN Latency (CPU) |
|---|---|
| 128 | 48.50 ms |
| 256 | 84.94 ms |
| 512 | 160.93 ms |
| 1024 | 328.93 ms |
| 2048 | 727.57 ms |
| 4096 | N/A (OOR) |
Note: 'N/A (OOR)' indicates sequence length exceeds model's maximum position embedding range.
🏗️ Architecture Details
| Property | Value |
|---|---|
| Layers | 8 |
| Hidden Dim | 512 |
| Vocab Size | 32017 |
| Attention Heads | 4 |
📊 Parameter Breakdown
| Component | Parameters | % of Total |
|---|---|---|
| Total | 98.26M (98,264,064) | 100% |
| Embeddings | 16.39M | 16.68% |
| DSRN Recurrent Blocks | 81.87M | 83.32% |
| Norms & Biases | 512 | 0.00% |
💻 Usage
You can load and use this model directly via sentence-transformers:
from sentence_transformers import SentenceTransformer
# Load model with auto-mapping enabled
model = SentenceTransformer("ethicalabs/Echo-DSRN-v0.1.3-Embed-Exp", trust_remote_code=True)
# Encode text to get 2048-dimensional embeddings
sentences = ["The recurrent slow state contains the aligned sequence representations.", "Echo-DSRN has linear complexity."]
embeddings = model.encode(sentences)
print(embeddings.shape) # (2, 2048)
🛠️ Training Procedure
The model was trained in three sequential phases:
- Contrastive Pre-training: Representation space alignment using natural language inference datasets.
- Fine-grained Similarity Tuning: Fine-tuning using semantic textual similarity benchmarks to calibrate similarity scores.
- Multi-Task Generalization Tuning: Training on NLI retrieval and STS semantic similarity.
This model card was automatically generated by scripts/generate_model_card.py.
Note on tokenizer padding
The benchmark results on this card (STS table) were measured with left
padding (padding_side: left), and this model version reproduces them under
that convention. A right-padded training version is planned: right padding
keeps padded-batch embeddings consistent with single-request embeddings
(leading pad tokens do not pollute the recurrent state), so future
checkpoints will be batch-composition independent.