Alo-70M (Instruct)
Model Summary
Alo-70M is the instruction-tuned version of the ultra-lightweight 69-million parameter Bengali language model, Alo-70M-Base. Built on a scaled-down LLaMA architecture, it is designed to act as a highly efficient, edge-deployable localized AI assistant.
Fine-tuned on a curated dataset of instruction-response pairs using the ChatML format, Alo-70M is aligned for tasks such as summarization, entity extraction, text editing, and question answering in native Bengali. Despite its compact footprint, it offers a viable path for edge AI deployment on standard CPUs and mobile hardware.
- Developer: Fahad Hossain
- Language: Bengali (Bangla)
- Model Type: Causal Language Model (Instruction-Tuned Autoregressive Transformer)
- Parameter Count: 69 Million
- License: Apache 2.0
Related Resources
- Base Model: spitfire4794/Alo-70M-Base
- Alignment Dataset: spitfire4794/Bangla-SFT-50k
- Tokenizer: spitfire4794/beng_bpe
Usage
Alo-70M was trained using the ChatML template. The chat template is built directly into the Jinja template of the tokenizer (spitfire4794/beng_bpe). You can leverage it using Hugging Face's apply_chat_template interface:
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "spitfire4794/Alo-70M"
# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
# Define the instruction in ChatML format
messages = [
{"role": "user", "content": "নিচের অনুচ্ছেদটি সংক্ষেপে সারসংক্ষেপ করুন: [এখানে আপনার টেক্সট লিখুন]"}
]
# Apply the pre-configured ChatML template
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
# Generate text
outputs = model.generate(
**inputs,
max_new_tokens=150,
repetition_penalty=1.1,
do_sample=True,
temperature=0.6,
top_p=0.9
)
# Decode response (omitting user prompt)
response = outputs[0][inputs.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
Supervised Fine-Tuning (SFT) Details
Alo-70M was aligned using a curated subset of the Bangla-SFT-50k dataset formatted using ChatML.
- Dataset Pruning: Initial SFT experiments revealed that forcing a sub-100M parameter model to learn complex markdown syntax/tables caused severe representation crowding. Thus, the 12,517-sample Structured Formatting category was excluded. The final active training mixture consisted of 37,536 aligned pairs.
- Engineering Properties: The training data strictly forbade conversational prefaces (e.g., "নিশ্চয়ই, আমি এটি করে দিচ্ছি") so that responses begin immediately with the target output, optimizing inference speeds.
- Hardware: NVIDIA T4 and L4 GPUs.
- Hyperparameters:
- Optimizer: Fused AdamW (
adamw_torch_fused) with beta_1 = 0.9, beta_2 = 0.999, epsilon = 10^-8 - Weight Decay: 0.0
- Learning Rate Schedule: Cosine decay, peaking at 3 times 10^-4 with a 10% linear warmup.
- Epochs: 3
- Effective Batch Size: 32 (per-device 8 with gradient accumulation of 4).
- Precision: Native Automatic Mixed Precision (AMP).
- Optimizer: Fused AdamW (
Model Architecture details
Like its base model, Alo-70M utilizes a parameter-efficient architecture:
- Layers: 12
- Hidden Dimension: 512 | Intermediate FFN: 1408
- Attention: Grouped-Query Attention (GQA) with 8 query heads / 4 KV heads.
- Positional Embeddings: RoPE (Base freq: 10,000)
- Word Embeddings: Untied.
- Context Window: 1024 tokens.
Evaluation Results
The model was evaluated zero-shot across Bengali reasoning and knowledge benchmarks (continuation-based log-probability evaluation):
| Benchmark | Alo-70M (SFT) | Alo-70M-Base | Gemma-3-270M-IT | TigerLLM-1B-IT |
|---|---|---|---|---|
| bangla_mmlu_bn | 26.29% | 26.31% | 26.81% | 27.66% |
| bangla_commonsenseqa_bn | 25.88% | 28.42% | 22.77% | 25.14% |
| indicbench_arc_bn_challenge | 24.15% | 22.70% | 25.34% | 27.13% |
| boolqa_bn | 48.70% | 48.42% | 51.30% | 52.40% |
| openbookqa_bn | 30.58% | 31.39% | 31.99% | 34.21% |
| piqa_bn | 50.05% | 50.49% | 49.51% | 49.51% |
| hellaswag_bn | 26.89% | 27.27% | 27.85% | 31.01% |
Note: The 69M instruction-tuned model outperforms the larger Gemma-3-270M-IT baseline on tasks like CommonsenseQA and PIQA.
Limitations and Biases
- Alignment Tax (Catastrophic Forgetting): While SFT successfully aligned the model for text generation stability and instruction following, it introduced a measurable degradation in pure zero-shot reasoning compared to the Base model (e.g., dropping from 28.42% to 25.88% on CommonsenseQA). This happens because applying instructions to a highly capacity-constrained 69M model over-indexes weights toward output formatting at the expense of some pre-trained logical representations.
- Knowledge Retrieval: With under 100M parameters, the model physically lacks the capacity to serve as a comprehensive encyclopedic knowledge base. It is better suited for text processing tasks (editing, summarizing) than fact-retrieval.
- Context Length: The model is optimized for a 1024-token context window. Prompts exceeding this length will be truncated or result in degraded quality.
Citation
Technical paper out soon.