Healthcare Pathways-Inspired MoE SLM
A small decoder-only Transformer language model trained from scratch using a sparse Mixture-of-Experts architecture.
This is a Pathways-inspired architecture and is not Google's Pathways system.
Architecture
- Decoder-only Transformer
- Causal self-attention
- Sparse Mixture-of-Experts
- Top-k expert routing
- 4 experts
- Top-2 routing
- GELU activation
- LayerNorm
- AdamW optimizer
Model configuration
| Parameter | Value |
|---|---|
| Vocabulary size | 16,384 |
| Context length | 512 |
| Hidden size | 256 |
| Transformer layers | 6 |
| Attention heads | 8 |
| FFN size | 768 |
| Number of experts | 4 |
| Top-k experts | 2 |
| Activation | GELU |
Tokenizer
The tokenizer is hosted separately:
alokanand002/medical-bpe-16k
The model was trained using this tokenizer and its vocabulary must remain consistent with the model.
Intended use
This is an experimental Small Language Model intended for:
- Healthcare NLP research
- Medical terminology experiments
- MedTech NLP
- Transformer research
- Mixture-of-Experts experimentation
- Local inference
- Fine-tuning experiments
Limitations
This is an experimental research model.
It may generate incorrect, incomplete, or misleading medical information.
It has not been clinically validated and must not be used for diagnosis, treatment, or other clinical decision-making.
Loading
This repository contains the custom PyTorch model weights and architecture source code.
Because this is a custom architecture rather than a native
Transformers architecture, the model should be instantiated
using the accompanying model.py and configuration.
Tokenizer
Tokenizer repository:
https://huggingface.co/alokanand002/medical-bpe-16k
License
Add an appropriate license before public/commercial distribution.