Wiki-SmartBotLM-Instruct
A 270M parameter decoder-only Small Language Model (SLM) pretrained on English Wikipedia and instruction-tuned for general-purpose conversational AI.
Overview
Wiki-SmartBotLM-Instruct is a compact autoregressive language model developed entirely from scratch as an independent research project. The model was designed to explore the complete lifecycle of modern language model development—from pretraining through instruction tuning—using open datasets and modern transformer architectures.
Unlike models that rely on continued pretraining of existing foundation models, Wiki-SmartBotLM-Instruct was initialized with random weights and trained using a multi-stage pipeline.
The primary goals of this project were:
- Understand large language model training from first principles.
- Build a compact yet capable language model.
- Explore modern transformer architectures.
- Demonstrate the effectiveness of instruction tuning on a pretrained base model.
Model Details
| Property | Value |
|---|---|
| Model Name | Wiki-SmartBotLM-Instruct |
| Architecture | Decoder-only Transformer |
| Parameters | 270 Million |
| Context Length | 8192 Tokens |
| Training Framework | PyTorch |
| Precision | BF16 / FP16 |
| License | MIT |
Architecture
Wiki-SmartBotLM-Instruct incorporates several modern transformer improvements including:
- Rotary Positional Embeddings (RoPE)
- RMSNorm
- SwiGLU Feed Forward Networks
- Grouped Query Attention (GQA)
- Flash Attention
- Causal Self Attention
- Mixed Precision Training
Training Pipeline
The model was trained in two stages.
Stage 1 — Pretraining
The model was pretrained on the English Wikipedia corpus using next-token prediction.
Objectives:
- Learn English grammar
- Learn factual knowledge
- Learn encyclopedic writing style
- Learn semantic relationships
- Learn long-context language modeling
Stage 2 — Supervised Fine-Tuning (SFT)
Following pretraining, the model was instruction tuned using an open instruction-following dataset.
The objective of this stage was to improve:
- Instruction following
- Conversational ability
- Question answering
- Technical explanations
- General helpfulness
Training Data
Pretraining
- English Wikipedia (Hugging Face)
Supervised Fine-Tuning
- Alpaca Instruction Dataset
Capabilities
Wiki-SmartBotLM-Instruct can perform a variety of natural language tasks including:
- General Question Answering
- Technical Explanations
- Programming Assistance
- Educational Content Generation
- Text Summarization
- Brainstorming
- Long-form Text Generation
- Basic Reasoning
- Instruction Following
Benchmarks
The model was evaluated on the following text-based benchmarks:
- MMLU
- ARC Challenge
- HellaSwag
- Winogrande
- BoolQ
| Benchmark | WikiSmartBotLM | Gemma 3 270M IT | Gap |
|---|---|---|---|
| BoolQ | 37.8 | 61.4 (PT) ([Google AI for Developers][1]) | -23.6 |
| ARC Challenge | 24.2 | 28.2 ([Google AI for Developers][1]) | -4.0 |
| HellaSwag | 26.3 | 37.7 ([Google AI for Developers][1]) | -11.4 |
| PIQA | 53.5 | 66.2 ([Google AI for Developers][1]) | -12.7 |
| Winogrande | 50.2 | 52.3 ([Google AI for Developers][1]) | -2.1 |
It shows despite the training difference between Gemma3 270M and WikiSmartBotLM, It does come close to Gemma 3 in some benchmarks.
Example
Prompt
Explain what artificial intelligence is.
Response
Artificial intelligence (AI) is a branch of computer science focused on building systems capable of performing tasks that typically require human intelligence. These tasks include reasoning, learning, planning, perception, natural language understanding, and decision making. Modern AI encompasses techniques such as machine learning, deep learning, computer vision, and reinforcement learning, and is widely used in applications ranging from recommendation systems to autonomous vehicles.
Intended Uses
Wiki-SmartBotLM-Instruct is intended for:
- Educational use
- AI research
- NLP experimentation
- Local inference
- Learning transformer architectures
- Small-scale deployment
- Prototype conversational assistants
Limitations
As a compact language model, Wiki-SmartBotLM-Instruct has several limitations.
- May hallucinate factual information.
- Limited reasoning compared to larger frontier models.
- Knowledge is limited to the pretraining corpus.
- May struggle with highly specialized or domain-specific questions.
- Performance may degrade on very long conversations.
Ethical Considerations
Wiki-SmartBotLM-Instruct is intended for research and educational purposes.
Although instruction tuning improves response quality, users should independently verify information before relying on generated content in high-stakes domains such as healthcare, finance, or legal advice.
Future Work
Future versions of Wiki-SmartBotLM aim to include:
- Improved factual grounding
- Larger instruction datasets
- Enhanced reasoning performance
- Better multilingual support
- Retrieval-Augmented Generation (RAG)
- Tool calling support
- Longer context windows
Citation
If you use Wiki-SmartBotLM-Instruct in your research, please cite:
@misc{wikismartbotlm2026,
title={Wiki-SmartBotLM-Instruct: A 270M Parameter Decoder-Only Small Language Model},
author={Pranav Upadhyaya},
year={2026},
howpublished={Hugging Face Model Repository}
}
Acknowledgements
This project was made possible by the open-source AI community.
Special thanks to:
- Hugging Face
- PyTorch
- English Wikipedia contributors
- Alpaca dataset contributors
- The open-source transformer research community
About the Project
Wiki-SmartBotLM-Instruct was developed as an independent research initiative to better understand modern language model development. From random initialization to pretraining and instruction tuning, every stage of the model was built to explore the practical engineering challenges of creating compact, efficient, and accessible language models.
Feedback, issues, and contributions are always welcome.