LegalParam

BharatGen introduces LegalParam, a domain-specialized large language model fine-tuned from Param-1-2.9B-Instruct on an exhaustive India-centric legal dataset. Trained across a comprehensive taxonomy of acts, laws, policies, and regulations, LegalParam is built to deliver accurate, context-aware answers to legal queries while also supporting tasks such as summarizing lengthy legal documents and simplifying complex policy texts. Whether it’s aiding practitioners with quick references, assisting researchers in exploring legal frameworks, or helping citizens better understand their rights and obligations, LegalParam brings clarity and accessibility to the vast and intricate landscape of Indian law.

⚖️ Motivation

Law in India is vast, complex, and ever-evolving, yet most language models lack the depth and domain specialization needed to navigate acts, policies, and regulations in an India-centric context. Citizens, researchers, and practitioners often struggle with scattered information and dense legal language that is hard to interpret. LegalParam bridges this gap by combining Param-1’s strong instruction-following capabilities with a meticulously curated, exhaustive dataset of Indian laws, policies, and regulations, making legal knowledge more accessible, contextual, and actionable.

🏗 Model Architecture

LegalParam inherits the architecture of Param-1-2.9B-Instruct:

• Hidden size: 2048
• Intermediate size: 7168
• Attention heads: 16
• Hidden layers: 32
• Key-value heads: 8
• Max position embeddings: 2048
• Activation: SiLU
• Positional Embeddings: Rotary (RoPE, theta=10000)
• Attention Mechanism: Grouped-query attention
• Precision: bf16-mixed
• Base model: Param-1-2.9B-Instruct

📚 Data Preparation

LegalParam’s training corpus was designed to ensure comprehensive coverage of Indian legal knowledge and high-quality instruction tuning along with bilingual (Hindi + English) accessibility.

Steps involved:

1. Source Gathering

   • Open-source datasets on Indian laws, acts, and policies were collected and curated. Historical data (for acts and laws) for more grounded training

2. Question Generation

   • Q&A pairs were generated from acts and legal texts to create grounded supervision signals

3. Domain Taxonomy & Personas

   • An exhaustive taxonomy of the Indian legal framework was built.
   • Personas such as citizen, lawyer, policymaker, and researcher were defined to guide synthetic data generation.

4. Dataset Construction

   • ~2M Q&A pairs curated from open sources.
   • Additional synthetic data grounded in taxonomy and personas expanded the dataset.
   • In total, 5M Q&A pairs were used for fine-tuning.

🏋️ Training Setup

• Base model: Param-1-2.9B-Instruct
• Training framework: Hugging Face + torchrun multi-node setup
• Prompt template: Custom-designed for legal inference
• Scheduler: Linear with warmup
• Epochs: 3
• Total training samples: 12M
• Test samples: 1.2M
• Base learning rate: 5e-6
• Minimum learning rate: 0

• Additional tokens: <system>, <user>, <assistant>, <context>, <think>, </think>
• Vocab size: 256k + 6
• Global batch size: 1024
• Micro batch size: 4
• Gradient accumulation steps: 32

🚀 Inference Example

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "bharatgenai/LegalParam"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32,
    device_map="auto"
)

# Example legal query
user_input = "What steps should a farmer take to legally transfer agricultural land ownership?"

# 3 types of prompt
# 1. Generic QA
# 2. Context based QA (context as part of prompt)
# 3. Multi-turn conversation

# Based on your requirements use the type of prompt (refere the above examples)
prompt = f"<user>\n{user_input}<assistant>\n"
# prompt = f"<user>\n{user_or_rag_context}\n<assistant>\n"
# prompt = f"<user>\n{user_input1}\n<assistant>\n{user_input2}\n<user> {user_input3} <assistant>..."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=300,
        do_sample=True,
        top_k=50,
        top_p=0.95,
        temperature=0.6,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

📊 Evaluation

• Legal Skills & Communication Q&A
• Corporate, Commercial & Criminal Law
• Legal Theory & Jurisprudence
• Public International Law
• Humanitarian & Refugee Rights
• Bilingual (English/Hindi) capability

BhashaBench-Legal (BBL)

Subject Domain Performance

Question Level Difficulty