LuminoLex-1.7B
LuminoLex-1.7B is an experimental English sparse Mixture-of-Experts (MoE) causal language model developed by LuminoLex AI Lab, pretrained from scratch on 35 billion tokens. It features grouped-query attention (GQA), top-6 routed experts, an always-active shared expert, and tied embeddings.
Highlights
- Parameter Efficiency: 1.7B container capacity with 849.3M effective trained parameters and ~374M active parameters per token.
- Sparse Routing: Routes tokens dynamically across top-6 of 32 active expert banks alongside a dedicated shared expert.
- Consumer Hardware Friendly: Compact footprint designed for low-latency inference on consumer GPUs.
Quickstart
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "LuminoLex/LuminoLex-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.bfloat16,
device_map="cuda",
)
model.eval()
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "What is the capital of Japan?"}],
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=128,
do_sample=False,
repetition_penalty=1.08,
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip())
Prompt format:
User: {prompt}
Assistant:
Model Specifications
| Parameter | Value |
|---|---|
| Total container parameters | 1,699,863,296 |
| Effective trained parameters | 849,307,850 |
| Active parameters / token | 373,920,384 (container nominal: 409,869,056) |
| Architecture | Sparse MoE (18 trained layers in 20-layer container, hidden size 1,536) |
| Attention | Grouped-Query Attention (24 Q heads, 8 KV heads) |
| Routing configuration | Top-6 routing across 32 active expert banks + 1 shared expert |
| Training tokens | Approximately 35 billion |
| Context length | 1,024 tokens |
| Weight precision | BF16 |
| License | Apache 2.0 |
| Developer | LuminoLex AI Lab |
Architecture & Capacity Note
The checkpoint is packaged in a 1.7B parameter container (1,699,863,296 physical float elements, 20 layers x 64 expert slots) for downstream structural compatibility:
- Trained Layers: 18 active layers contain trained weights (layers 18 and 19 are unpopulated identity padding layers).
- Trained Experts: 32 active routed expert banks are utilized per active layer. The remaining 32 expert slots (32 to 63) are masked via a -9984 router gate bias, preventing them from being selected during inference.
- Active Parameters: In active computation, each token engages the embeddings, shared backbone, and top-6 active experts across the 18 trained layers, executing exactly 373,920,384 active parameters per token.
Local Gradio UI
Run the local chat interface:
pip install -r requirements.txt
python app.py
Limitations
LuminoLex-1.7B is a compact experimental model. Outputs may be inaccurate, repetitive, or inconsistent on complex multi-step reasoning, arithmetic, strict formatting, and code generation. Intended for research and evaluation purposes.