Akhorika E2B Base (ananddey/akhorika-e2b-base)
Akhorika E2B Base (আখৰিকা) is a foundational 2.69B parameter language model (1.91B active text decoder parameters) continued-pretrained (CPT) on Assamese language corpora, based on Google's Gemma 4 E2B architecture.
Model Details
- Model Name: Akhorika E2B Base
- Base Architecture:
Gemma4ForConditionalGeneration(Google Gemma 4 E2B) - Parameters: 2.69B total (1.91B text decoder parameters)
- Tokenizer: Custom 32,000 vocab Assamese-first SentencePiece Unigram tokenizer (replacing the native 256k tokenizer)
- Embedding Initialization: FOCUS (semantic subword surface-form projection from base Gemma embeddings)
- Languages: Assamese (
as), English (en) - Training Type: Continued Pretraining (CPT)
- Checkpoint: Step 821 / 821 (epoch 1 complete)
- Eval Loss: 4.09 (PPL ~60 on the held-out Assamese eval set)
Quickstart & Inference
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ananddey/akhorika-e2b-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
attn_implementation="sdpa"
)
prompt = "অসম ভাৰতৰ উত্তৰ-পূব অঞ্চলৰ"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1,
pad_token_id=0,
eos_token_id=2
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))