BharatGPT-Instruct
BharatGPT-Instruct E2B is a multilingual Large Language model developed by CoRover.ai, trained to generate high-quality text across major Indic languages. The model is pretrained on a diverse, curated corpus spanning Hindi, Punjabi, Marathi, Malayalam, Odia, Kannada, Gujarati, Bengali, Urdu, Tamil, and Telugu, drawing on authentic Indian conversational data to capture the linguistic and cultural nuances of each language.
Usage
Load and use the model directly via the Hugging Face transformers library.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_ID = "CoRover/BharatGPT-Instruct"
tokenizer = AutoTokenizer.from_pretrained(
MODEL_ID,
token=HF_TOKEN,
fix_mistral_regex=True,
trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
token=HF_TOKEN,
dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
model.eval()
torch.backends.cudnn.benchmark = True
def generate_response(prompt, max_new_tokens=256):
messages = [
{
"role": "user",
"content": prompt
}
]
encoded = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
).to(model.device)
with torch.inference_mode():
output = model.generate(
input_ids=encoded["input_ids"],
attention_mask=encoded["attention_mask"],
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
)
generated_tokens = output[0][
encoded["input_ids"].shape[1]:
]
return tokenizer.decode(
generated_tokens,
skip_special_tokens=True
).strip()
print(generate_response("Who developed you?"))
Multilingual MMLU Benchmarks
BharatGPT-Instruct evaluated across Indian languages against Qwen-1.7B and Sarvam-30B.
| Language | BharatGPT-mini | BharatGPT-Instruct | Qwen-1.7B | Sarvam-30B |
|---|---|---|---|---|
| Hindi | 23.1 | 41.59 | 34.9 | 32.1 |
| Bengali | 23.9 | 39.22 | 32.3 | 27.4 |
| Marathi | 23.4 | 39.40 | 31.7 | 29.2 |
| Telugu | 23.6 | 38.44 | 30.7 | 25.9 |
| Gujarati | 23.0 | 38.38 | 32.7 | 32.1 |
| Malayalam | 23.2 | 37.96 | 30.9 | 26.3 |
| Punjabi | 23.7 | 37.58 | 31.5 | 25.8 |
| Tamil | 23.5 | 37.30 | 30.9 | 30.1 |
| Odia | 23.9 | 34.82 | 29.8 | 30.3 |
| Kannada | 23.8 | 38.30 | 31.3 | 29.4 |
RAG Evaluation
This is the most relevant comparison for our actual product surface-enterprise AI-Assistant grounded in a customer knowledge base. BharatGPT-Instruct scores 100.00 on Faithfulness, essentially tied with GPT-4o-mini and Qwen 3 1.7B, and matches both on Top-K Accuracy (%).
| RAG Metric | BharatGPT-mini | BharatGPT-Instruct | GPT-4o-mini | Qwen 3 1.7B |
|---|---|---|---|---|
| Faithfulness | 80.09 | 100.00 | 95.00 | 95.51 |
| Top-K Accuracy | 100 | 100 | 100 | 100 |
| Relevance | 70.95 | 91.07 | 92.12 | 95.76 |
| Recall | 94.1 | 94.12 | 93.07 | 92.51 |
Benchmark Results
BharatGPT models were evaluated on a diverse and comprehensive set of benchmark datasets and evaluation metrics to measure their performance across multiple dimensions of language understanding, reasoning, and text generation. The reported results cover a broad range of tasks, including knowledge-based question answering, reasoning, comprehension, and Safety and Truthfulness.
| Benchmark | BharatGPT-mini | BharatGPT-3B-Indic | BharatGPT-Instruct E2B | Qwen-1.7B | Sarvam-30B |
|---|---|---|---|---|---|
| General Knowledge | |||||
| MMLU | 23.60 | 52.90 | 57.56 | 55.50 | 45.60 |
| AGIEval | 28.32 | 30.40 | 32.57 | 40.20 | 30.70 |
| Commonsense Reasoning | |||||
| HellaSwag | 29.93 | 67.60 | 55.59 | 46.10 | 51.90 |
| PIQA | 62.08 | 75.70 | 78.51 | 72.40 | 62.80 |
| WinoGrande | 51.46 | 64.80 | 68.67 | 60.80 | 50.90 |
| Reading Comprehension | |||||
| BoolQ | 60.98 | 78.70 | 78.13 | 77.40 | 76.80 |
| Science Reasoning | |||||
| ARC-Easy | 52.78 | 73.40 | 83.21 | 72.60 | 44.80 |
| ARC-Challenge | 21.33 | 42.90 | 53.84 | 39.70 | 33.30 |
| Math Reasoning | |||||
| GSM8K | 1.00 | 31.40 | 23.50 | 68.00 | 70.60 |
| Safety & Truthfulness | |||||
| TruthfulQA | 18.20 | 44.70 | 38.00 | 50.30 | 64.80 |
| ToxiGen | 46.38 | 52.60 | 41.70 | 42.00 | 55.40 |
| Fairness & Bias | |||||
| WinoGender | 50.83 | 56.70 | 60.28 | 56.30 | 51.00 |
| CrowS-Pairs | 54.26 | 54.00 | 68.28 | 55.20 | 55.20 |
| BBQ | 38.40 | 79.80 | 55.07 | 38.30 | 59.40 |
Usage and Limitations
-
License: Non-Commercial. For academic and research purposes only. For commercial use, please visit Conversational Agentic AI Platform or Contact Us.
-
Terms of Use: Terms and Conditions
-
PIPEE -Responsible AI Framework:
- P - Purpose: Risk depends on context.
- I - Inclusivity: As the Hon'ble PM of India says, "AI means All Inclusive" - it must be human-centric and multi-modal - Text, Voice, and Video.
- P - Privacy: Where data is stored and inferred.
- E - Explainability: Can we audit the decision?
- E - Energy Efficiency: Just because someone has money for GPUs doesn't mean we should waste energy.