Param-1-5B
Param-1-5B is a bilingual (English–Hindi) large language model developed under the Param-1 family. With 5 billion parameters, this model extends the capabilities of Param-1-2.9B by incorporating enhanced mathematical reasoning and code understanding/generation.
The model is pretrained from scratch and designed to serve as a strong foundation for downstream tasks such as mathematical problem solving, and code-related understanding / generation.
Key Highlights
- 5B parameter dense Transformer model
- Bilingual: English and Hindi
- Enhanced domains: Math and Code (compared to Param-1-2.9B)
- Updated dataset mixture and percentage ratios
- Designed as a pretrained (PT) base model
Model Inference
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_name = "bharatgenai/Param-1-5B"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
model_name,
trust_remote_code=True,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
device_map="auto"
)
prompt = "Your prompt here"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=300,
do_sample=True,
top_k=50,
top_p=0.95,
temperature=0.6,
eos_token_id=tokenizer.eos_token_id,
use_cache=False
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Benchmarks
| Task | Param-1-5B (PT) |
|---|---|
| ARC Challenge | 55.1 (few) |
| ARC Easy | 75.6 (few) |
| HellaSwag | 76 (few) |
| HellaSwag Hi | 44.6 (few) |
| MMLU En | 50.8 (few) |
| MMLU Hi | 39.1 (few) |
| TriviaQA | 48.2 |
| TruthfulQA - Gen | 32.9 |
| TruthfulQA - MC1 | 27.1 |
| TruthfulQA - MC2 | 38.6 |
| PIQA | 79.6 |
| SuperGLUE - BoolQ | 72.7 |
| SuperGLUE - WiC | 50.5 |
| SuperGLUE - WSC | 48.1 |
| SuperGLUE - RTE | 65 |
Model Architecture
- Architecture: Transformer (Decoder-only)
- Number of parameters: ~5B
- Hidden size:
4096 - Intermediate size:
14336 - Number of layers:
16 - Number of attention heads:
32 - Number of key-value heads:
8 - Maximum position embedding:
4096 - Positional embeddings: Rotary (RoPE)
- Attention mechanism: Grouped-query attention
- Activation function:
silu - Precision:
bf16-mixed
Training Data
The model is pretrained on a large-scale bilingual corpus with a strong focus on English and Hindi, along with dedicated Math and Code data.
Dataset Composition (Total ≈ 6.61T tokens)
- English Natural Language:
3.6T - Hindi Natural Language:
2.77T - Math & Code:
238.4B- Math: 40%
- Code: 60%
Compared to Param-1-2.9B, Param-1-5B includes:
- A larger overall training corpus
- Explicit inclusion of Math and Code datasets
- Improved performance in mathematical and code understanding/generation
- Continued emphasis on English–Hindi bilingual capability
Training Details
- Training framework:
NVIDIA NeMo - Training infrastructure:
Yotta's Shakti Cloud - Hardware:
NVIDIA H100 - 1024 GPUs - Total training tokens:
6.61 Trillion
Limitations
- This is a pretrained base model and may require fine-tuning for instruction-following or chat use cases.
- The model may reflect biases present in large-scale web and code data.
- Performance on low-resource Hindi domains may vary depending on task complexity.
License
This model is released under the BharatGen non-commercial license.
Please refer to the LICENSE file for detailed terms and conditions.