Qwen3-0.6B-Engram
An implementation of DeepSeek conditional memory via an offloaded Look-Up Table (LUT) alongside Qwen3-0.6B-Base.
English: a 2gb engram embedding layer in qwen 3 0.6b
Benchmark Results (WikiText-2 Test Split)
| Model | Cross-Entropy Loss | Perplexity (PPL) | Next-Token Top-1 Acc |
|---|---|---|---|
| Vanilla Qwen3-0.6B | 2.8211 | 16.80 | 45.43% |
| Engram + Qwen3-0.6B | 2.6945 | 14.80 | 46.60% |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_PATH = "./qwen3-engram"
model = AutoModelForCausalLM.from_pretrained(
MODEL_PATH,
trust_remote_code=True,
torch_dtype=torch.float32,
device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
prompt = "The most common cause of acute renal failure is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=192, use_cache=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))