🧠 MiniBot-0.9M-Instruct
Instruction-tuned GPT-2 style language model (~900K parameters) optimized for Portuguese conversational tasks.
📌 Overview
MiniBot-0.9M-Instruct is the instruction-tuned version of MiniBot-0.9M-Base, designed to follow prompts more accurately, respond to user inputs, and generate more coherent conversational outputs in Portuguese.
Built on a GPT-2 architecture (~0.9M parameters), this model was fine-tuned on conversational and instruction-style data to improve usability in real-world interactions.
🎯 Key Characteristics
| Attribute | Detail |
|---|---|
| 🇧🇷 Language | Portuguese (primary) |
| 🧠 Architecture | GPT-2 style (Transformer decoder-only) |
| 🔤 Embeddings | GPT-2 compatible |
| 📉 Parameters | ~900K |
| ⚙️ Base Model | MiniBot-0.9M-Base |
| 🎯 Fine-tuning | Instruction tuning (supervised) |
| ✅ Alignment | Basic prompt-following behavior |
🧠 What Changed from Base?
Instruction tuning introduced significant behavioral improvements with no architectural changes:
| Feature | Base | Instruct |
|---|---|---|
| Prompt understanding | ❌ | ✅ |
| Conversational flow | ⚠️ Partial | ✅ |
| Instruction following | ❌ | ✅ |
| Overall coherence | Low | Improved |
| Practical usability | Experimental | Functional |
💡 The model is now significantly more usable in chat scenarios.
🏗️ Architecture
The core architecture remains identical to the base model:
- Decoder-only Transformer (GPT-2 style)
- Token embeddings + positional embeddings
- Self-attention + MLP blocks
- Autoregressive generation
No structural changes were made — only behavioral improvement through fine-tuning.
📚 Fine-Tuning Dataset
The model was fine-tuned on a Portuguese instruction-style conversational dataset composed of:
- 💬 Questions and answers
- 📋 Simple instructions
- 🤖 Assistant-style chat
- 🎭 Basic roleplay
- 🗣️ Natural conversations
Expected format:
User: Me explique o que é gravidade
Bot: A gravidade é a força que atrai objetos com massa...
Training strategy:
- Supervised Fine-Tuning (SFT)
- Pattern learning for instruction-following
- No RLHF or preference optimization
💡 Capabilities
✅ Strengths
- Following simple instructions
- Answering basic questions
- Conversing more naturally
- Higher coherence in short responses
- More consistent dialogue structure
❌ Limitations
- Reasoning is still limited
- May generate incorrect facts
- Does not retain long context
- Sensitive to poorly structured prompts
⚠️ Even with instruction tuning, this remains an extremely small model. Adjust expectations accordingly.
🚀 Getting Started
Installation
pip install transformers torch
Usage with Hugging Face Transformers
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "AxionLab-official/MiniBot-0.9M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
prompt = "User: Me diga uma curiosidade sobre o espaço\nBot:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=80,
temperature=0.7,
top_p=0.9,
do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
⚙️ Recommended Settings
| Parameter | Recommended Value | Description |
|---|---|---|
temperature | 0.6 – 0.8 | Controls randomness |
top_p | 0.85 – 0.95 | Nucleus sampling |
do_sample | True | Enable sampling |
max_new_tokens | 40 – 100 | Response length |
💡 Instruct models tend to perform better at lower temperatures. Try values around
0.65for more accurate and focused responses.
🧪 Intended Use Cases
| Use Case | Suitability |
|---|---|
| 💬 Lightweight Portuguese chatbots | ✅ Ideal |
| 🎮 NPCs and games | ✅ Ideal |
| 🧠 Fine-tuning experiments | ✅ Ideal |
| 📚 NLP education | ✅ Ideal |
| ⚡ Local / CPU-only applications | ✅ Ideal |
| 🏭 Critical production environments | ❌ Not recommended |
⚠️ Disclaimer
- Extremely small model (~900K parameters)
- No robust alignment (no RLHF)
- May generate incorrect or nonsensical responses
- Not suitable for critical production environments
🔮 Future Work
- 🧠 Reasoning-tuned version (
MiniBot-Reason) - 📈 Scaling to 1M–10M parameters
- 📚 Larger and more diverse dataset
- 🤖 Improved response alignment
- 🧩 Tool-use experiments
📜 License
Distributed under the MIT License. See LICENSE for more details.
👤 Author
Developed by AxionLab 🔬