AgriParam

BharatGen introduces AgriParam, a domain-specialized large language model fine-tuned from Param-1-2.9B-Instruct on a high-quality, India-centric agriculture dataset.
AgriParam is designed to understand and generate contextually rich responses for agricultural queries, farmer advisories, policy information, research insights, and rural knowledge dissemination.

🌱 Motivation

Agriculture is the backbone of India’s economy, yet existing language models lack deep domain knowledge tailored to Indian contexts, languages, and cultural nuances.
AgriParam bridges this gap by combining Param-1’s bilingual capabilities with a meticulously curated agricultural knowledge base.

🏗 Model Architecture

AgriParam inherits the architecture of Param-1-2.9B-Instruct:

• Hidden size: 2048
• Intermediate size: 7168
• Attention heads: 16
• Hidden layers: 32
• Key-value heads: 8
• Max position embeddings: 2048
• Activation: SiLU
• Positional Embeddings: Rotary (RoPE, theta=10000)
• Attention Mechanism: Grouped-query attention
• Precision: bf16-mixed
• Base model: Param-1-2.9B-Instruct

📚 Data Preparation

AgriParam’s training corpus was carefully crafted to ensure deep agricultural knowledge, cultural relevance, and bilingual (English-Hindi) accessibility.

Steps involved:

1. Source Gathering

   • 17k open-source, India-focused agricultural news & information passages.

2. Question Generation

   • Generated 5 curated Q&A pairs per passage using an open-source LLM.

3. Domain Taxonomy & Personas

   • Built an exhaustive, India-specific agricultural taxonomy.
   • Defined farmer, policy-maker, scientist, and agri-business personas.

4. Dataset Construction

   • 2M Q&A pairs grounded in taxonomy and personas.
   • Complete dataset translated into Hindi.
   • 6M multi-turn conversation samples created.

🏋️ Training Setup

• Base model: Param-1-2.9B-Instruct
• Training framework: Hugging Face + torchrun multi-node setup
• Prompt template: Custom-designed for agricultural inference
• Scheduler: Linear with warmup
• Epochs: 3
• Total training samples: 12M
• Test samples: 1.2M
• Base learning rate: 5e-6
• Minimum learning rate: 0
• Additional tokens: <user>, <assistant>, <context>, <system_prompt>
• Vocab size: 256k + 4
• Global batch size: 1024
• Micro batch size: 4
• Gradient accumulation steps: 32

🚀 Inference Example

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "bharatgenai/AgriParam"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32,
    device_map="auto"
)

# Example agricultural query
user_input = "What are the best practices for organic wheat farming in Uttar Pradesh?"

# 3 types of prompt
# 1. Generic QA: <user> ... <assistant>
# 2. Context based QA: <context> ... <user> ... <assistant>
# 3. Multi-turn conversation (supports upto 5 conversations): <user> ... <assistant> ... <user> ... <assistant>

# Based on your requirements use the type of prompt (refere the above examples)
prompt = f"<user> {user_input} <assistant>"
# prompt = f"<context> {user_context} <user> {user_input} <assistant>"
# prompt = f"<user> {user_input1} <assistant> {user_input2} <user> {user_input3} <assistant>..."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=300,
        do_sample=True,
        top_k=50,
        top_p=0.95,
        temperature=0.6,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

📊 Evaluation

• Crop-specific Q&A
• Policy & scheme awareness
• Rural advisory & extension services
• Bilingual (English/Hindi) capability

BhashaBench-Krishi (BBK)

Subject Domain Performance

Question Level Difficulty