SNIF: StateGate Neural Inference Fabric (Qwen2.5-0.5B + DeepSeek-R1-1.5B)
SNIF (StateGate Neural Inference Fabric) is an open-source composite causal language model that dynamically connects Qwen2.5-0.5B (Edge Prefill and Fast Decode) and DeepSeek-R1-1.5B (Cognitive Reasoning) via a continuous 2048-dimensional latent interconnect.
Author: CS Hari Krishna (csharikrishna1806@gmail.com)
Repository: csharikrishna/stategate
License: MIT
Architecture Highlights
- Continuous Latent Deliberation: Zero intermediate text tokens transferred across models during routing.
- Asymmetric Compute Efficiency: Confident queries complete via lightweight edge prefill; complex queries escalate dynamically to deep reasoning.
- Low Hardware Footprint: Active memory footprint is ~1,980 MiB VRAM, enabling execution on consumer GPUs and free Google Colab environments.
- Concise Decode: Bounded conclusion decoding avoids repetitive multi-paragraph derivations, delivering up to 3.5x faster decode latency.
Quickstart
1. Installation
pip install torch transformers safetensors accelerate sentencepiece tiktoken
2. Inference
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "csharikrishna/snif-qwen-deepseek"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32
# Load tokenizer and composite model
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
torch_dtype=dtype,
device_map="auto" if torch.cuda.is_available() else None,
)
if device == "cpu":
model = model.to("cpu")
model.eval()
# 1. Factual Query (Fast exit locally via Qwen)
prompt1 = "What is the boiling point of water at sea level?"
inputs1 = tokenizer(prompt1, return_tensors="pt").to(model.device)
out1 = model.generate(**inputs1, max_new_tokens=40, do_sample=False)
print("Answer 1:", tokenizer.decode(out1[0][inputs1.input_ids.shape[-1]:], skip_special_tokens=True).strip())
print("Telemetry 1:", model.last_routing_info)
# 2. Logic Query (Escalates via 2048D NeuralBus to DeepSeek-R1 with Dynamic CHOP)
prompt2 = "If all bloops are razzies and some razzies are snarks, are all bloops snarks? State conclusion concisely."
inputs2 = tokenizer(prompt2, return_tensors="pt").to(model.device)
out2 = model.generate(**inputs2, max_new_tokens=40, do_sample=False)
print("Answer 2:", tokenizer.decode(out2[0][inputs2.input_ids.shape[-1]:], skip_special_tokens=True).strip())
print("Telemetry 2:", model.last_routing_info)
# Example Telemetry:
# {
# 'route_taken': 'ESCALATE_CHOP',
# 'p_uncertainty': 0.4888,
# 'reasoning_layers_executed': 13,
# 'reasoning_layers_saved': 15, # 53.6% compute reduction!
# 'chop_stability': 0.8769,
# 'intermediate_text_tokens': 0,
# 'elapsed_ms': 3058.4
# }
System Architecture
- Small Model (Edge):
Qwen/Qwen2.5-0.5B-Instruct(896D, 24 layers, 494M params) - Big Model (Reasoning):
deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B(1536D, 28 layers, 1.78B params) - Latent Interconnect: Universal 2048D Lingua Franca Hub
- Routing Policy: Pre-generation latent routing evaluating intermediate activations at Layer 9.
Citation
@misc{stategate2026,
author = {CS Hari Krishna},
title = {StateGate: Pre-Generation Latent Meta-Routing for Cost-Optimal Language Model Inference},
year = {2026},
publisher = {GitHub},
howpublished = {\url{https://github.com/csharikrishna/stategate}}
}
Note: Research paper and extended theoretical evaluation in preparation.
License and Trademark Notice
- Software and Model Weights License: MIT License
- Copyright: Copyright (c) 2026 CS Hari Krishna. All rights reserved.
- Trademark Notice: The names "StateGate", "SNIF", and "NeuralBus" are project and research identifiers belonging to CS Hari Krishna. This open-source license grants permission to use the software and weights, but does not grant rights to use these names to endorse or promote derivative products without attribution.