AyurParam

BharatGen introduces AyurParam, a domain-specialized large language model fine-tuned from Param-1-2.9B-Instruct on a high-quality Ayurveda dataset. It is designed to handle Ayurvedic queries, classical text interpretation, clinical guidance, and wellness knowledge. Ayurveda offers vast traditional medical wisdom, yet most language models lack domain-specific understanding. AyurParam bridges this gap by combining Param-1’s bilingual strengths with a curated Ayurvedic knowledge base, enabling contextually rich and culturally grounded responses.

🏗 Model Architecture

AyurParam inherits the architecture of Param-1-2.9B-Instruct:

• Hidden size: 204
• Intermediate size: 7168
• Attention heads: 16
• Hidden layers: 32
• Key-value heads: 8
• Max position embeddings: 2048
• Activation: SiLU
• Positional Embeddings: Rotary (RoPE, theta=10000)
• Attention Mechanism: Grouped-query attention
• Precision: bf16-mixed
• Base model: Param-1-2.9B-Instruct

📚 AyurParam Dataset Preparation

AyurParam’s dataset was meticulously curated to capture the depth of Ayurvedic wisdom, ensure bilingual accessibility (English + Hindi), and support diverse clinical and academic applications. The preparation process focused on authenticity, quality, and relevance.

🔎 Data Sources

Total Books Collected: ~1000

• ~0.15M Pages, ~54.5M words
• 600 from open-source archives (digitized classical texts)
• 400 from internet sources covering specialized Ayurvedic domains

Domains Covered (examples):

• Kaaychikitsa (कायचिकित्सा)
• Panchkarma (पंचकर्म)
• Shalya Tantra (शल्यतंत्र)
• Shalakya Tantra (शालाक्यतंत्र)
• Research Methodology
• Ashtang Hruday (अष्टांगहृदय)
• Kriya Shaarir (क्रिया शारीर)
• Padarth Vigyan (पदार्थ विज्ञान)
• Rachana Shaarir (रचना शारीर)
• Charak Samhita (चरक संहिता)
• Dravyaguna (द्रव्यगुण)
• Rasa Shastra & Bhaishajya Kalpana (रसशास्त्र एवम भैषज्यकल्पना)
• Rog Nidan (रोगनिदान)
• AgadTantra (अगदतंत्र)
• Balrog (बालरोग)
• Strirog & Prasuti Tantra (स्त्रीरोग एवम प्रसूति तंत्र)
• Swasthvrutta (स्वस्थवृत्त)
• Sanskrit grammar, commentaries, and supporting texts
• etc

🧩 Data Processing Pipeline

1. Source Gathering

• Collected and digitized 1000 Ayurvedic books across classical, clinical, and academic domains.
• Preserved Sanskrit terminology with transliteration and contextual explanation

2. Question–Answer Generation

• Method: By-page Q&A generation using an open-source LLM.
• Focus: Only Ayurveda-related, context-grounded questions.
• Review: Domain expert validation for accuracy and clarity.

3. Taxonomy

• Dosha, Dhatu, Mala, Srotas, Nidana, Chikitsa, etc.

4. Final Dataset Construction

• Q&A Types:
  • General Q&A – direct knowledge-based
  • Thinking Q&A – reasoning and application-oriented
  • Objective Q&A – fact-check, MCQ, structured answers
• Languages: English + Hindi
  • Training Samples: ~4.8 Million (all combined)
  • Includes single-turn and multi-turn conversations

🏋️ Training Setup

• Base model: Param-1-2.9B-Instruct
• Training framework: Hugging Face + TRL (SFT) + torchrun multi-node setup
• Prompt template: Custom-designed for Ayurvedic inference
• Scheduler: Linear with warmup
• Epochs: 3
• Total training samples: ~4.8M
• Test samples: ~800k
• Base learning rate: 5e-6
• Minimum learning rate: 0
• Additional tokens: <user>, <assistant>, <context>, <system_prompt>, <actual_response>, </actual_response>
• Vocab size: 256k + 4
• Global batch size: 1024
• Micro batch size: 4
• Gradient accumulation steps: 32

🚀 Inference Example

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "bharatgenai/AyurParam"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32,
    device_map="auto"
)

# Example Ayurvedic query
user_input = "What is the Samprapti (pathogenesis) of Amavata according to Ayurveda?"

# Prompt styles
# 1. Generic QA: <user> ... <assistant>
# 2. Context-based QA: <context> ... <user> ... <assistant>
# 3. Multi-turn conversation (supports up to 5 turns): <user> ... <assistant> ... <user> ... <assistant>

prompt = f"<user> {user_input} <assistant>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=300,
        do_sample=True,
        top_k=50,
        top_p=0.95,
        temperature=0.6,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

📊 Benchmark Results: Ayur Param vs Baselines

• BhashaBench-Ayur benchmark

1. Overall Performance

Similar Range Models

Larger Models

2. Question Difficulty

Similar Range Models

Larger Models

3. Question Type

Similar Range Models

Larger Models

From the above results, AyurParam not only outperforms all similar-sized models but also achieves competitive or better performance than larger models across multiple metrics.

Citation

Please cite our paper if used in your work:

@misc{nauman2025ayurparamstateoftheartbilinguallanguage,
      title={AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda}, 
      author={Mohd Nauman and Sravan Gvm and Vijay Devane and Shyam Pawar and Viraj Thakur and Kundeshwar Pundalik and Piyush Sawarkar and Rohit Saluja and Maunendra Desarkar and Ganesh Ramakrishnan},
      year={2025},
      eprint={2511.02374},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.02374}, 
}

Contact

For any questions or feedback, please contact:

• Sravan Kumar (sravan.kumar@tihiitb.org)
• Kundeshwar Pundalik (kundeshwar.pundalik@tihiitb.org)
• Mohd.Nauman (mohd.nauman@tihiitb.org)