Param 1-2.9B-Instruct

BharatGen introduces the early checkpoint of SFT (Supervised Fine-Tuned) for Param 1, a bilingual language model trained from scratch in English and Hindi. With 2.9 billion parameters, this checkpoint builds upon the pretraining phase and serves as a foundation for more downstream tasks, safety testing, and customization.

Pre-Training Details

• Dataset: 7.5 Trillion tokens

• Data Quality: Highly curated with standard filtering and multiple processing steps.

• Scheduler: Cosine Annealing

• Learning_rate: 3e-4 to 3e-6

• Training Setup: Running on 512 H100 GPUs

• Framework: NVIDIA NeMo

• Precision: bf16-mixed

• For Pre-Trained Checkpoint (Param 1): https://aikosh.indiaai.gov.in/home/models/details/bharatgen_param_1_indic_scale_bilingual_foundation_model.html

SFT Training Details

• Dataset: 0.8 Million samples
• Epochs: 3
• Scheduler: Cosine Annealing
• Learning Rate: 5e-6 to 5e-8
• Training Hardware: 32 H200 GPUs
• Framework: NVIDIA NeMo
• Precision: bf16-mixed

Filtered high-quality bilingual data was used, combining public and in-house sources for safety-aware and culturally relevant behavior.

🚀 Model Inference

• After cloning provide the model path (model directory) in the inference script

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load tokenizer and model
model_name = "bharatgenai/Param-1-2.9B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    device_map="auto"
)

# Conversation Input
conversation = [
  {
    "content": "You are helpful assistant.",
    "role": "system"
  },
  {
    "content": "What is the BharatGen Mission?",
    "role": "user"
  }
]

# padding special token
inputs = tokenizer.apply_chat_template(
    conversation=conversation,
    return_tensors="pt",
    add_generation_prompt=True 
)
inputs = inputs.to(model.device)

# --- Generate output ---
with torch.no_grad():
    output = model.generate(
        inputs,
        max_new_tokens=300,
        do_sample=True,
        top_k=50,
        top_p=0.95,
        temperature=0.6,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=False
    )

# Get only the generated tokens (exclude the prompt length)
generated_tokens = output[0][inputs.shape[-1]:]
generated_text = tokenizer.decode(generated_tokens, skip_special_tokens=True)

print("Assistant Output:\n", generated_text)

📊 Benchmarks (zero-shot)

Notes:

• Benchmarks reflect zero-shot performance post-SFT.
• PT = Pretrained

🧠 Model Architecture

• Hidden size: 2048
• Intermediate size: 7168
• Attention heads: 16
• Hidden layers: 32
• Key-value heads: 8
• Max position embeddings: 2048
• Activation: SiLU
• Positional Embeddings: Rotary (RoPE, theta=10000)
• Attention Mechanism: Grouped-query attention
• Precision: bf16-mixed

Important Guidelines for Early Checkpoint Release of Param-1-2.9B-Instruct

1. Early Development Status

• This model is in the initial phase of Param-1 Instruct Model.
• It is yet to undergo full-supervised fine-tuning, safety alignment, or rigorous evaluation.
• The release is intended to showcase progress, gather feedback, and encourage research and experimentation.
• Outputs may at times be incoherent, irrelevant, or of suboptimal quality.

2. Data Sources and Potential Artifacts

• To preserve the Model's understanding on the global front, part of the training Data also includes data crawled from the Internet hence it may contain inherited artifacts;
• Due to the increased prevalence of AI-generated content online in the current times, the model may occasionally mimic such statements and incorrectly identify itself.
• These artifacts are natural consequences of using publicly available data found on the internet although critical but important since such data is important for the model to build a global know-how and we will be addressing issues like this in future iterations of the current Model.

3. Lack of Alignment and Guardrails

• A preliminary-level alignment or safety mechanisms have been implemented at this stage.
• The model is yet to under go full-scale instruction tuning, supervised fine-tuning, or reinforcement learning from human feedback (RLHF).
• As a result, it may occasionally:
  • Generate biased, offensive, or unsafe content
  • Be susceptible to misuse or prompt injection (jailbreaking)
  • Respond to harmful or unethical prompts without refusal
• This model must not be deployed in any production without reading Intent use section.

4. Intended Use

• This release is provided exclusively for research, experimentation and contribution to the open source community.
• Suggested use cases include:
  • Assessing early-stage LLM behavior
  • Debugging model training pipelines and configurations
  • Benchmarking or custom fine-tuning by the community
• Access to early-checkpoint should embibe a sense of motivation and enthusiasm among the open source community to take such early-stage check point and build India-Specific Innovative use cases on top of it. This should also help foster innovation among the Community.

5. Licensing and Responsibility

• Released under an open license with responsible usage guidelines.
• License: MIT
• Users are expected to:
  • Adhere to ethical usage practices and legal regulations
  • Avoid malicious or unsafe deployment
  • Credit the authors as per the licensing terms

6. Acknowledgement of Origin

• A home-grown effort initiated in India with limited resources.
• This work represents a bottom-up initiative to develop LLMs from scratch within India.
• It reflects our humble, resource-constrained journey to contribute meaningfully to the open-source AI ecosystem.
• We hope to foster collaboration and growth within the broader community.

7. Transparency & Community Collaboration

• We welcome contributions and open dialogue.
• We encourage the community to share feedback, report issues, and collaborate.
• Future versions will introduce better alignment, improved training scale, and more curated datasets.
• Together, we aim to evolve toward safer and more capable AI systems.