vera-qwen-1.5b-dpo (verifiable epistemic reward alignment)

this model is the final output of my personal ai alignment research project. i took qwen2.5-1.5b-instruct and fine-tuned it using direct preference optimization (dpo) so that it handles leading questions, heavily debated topics, and factual boundaries better than standard base models.

instead of giving overconfident answers or robotic "i cannot answer that" refusals, this model is trained to explicitly map its assumptions, admit uncertainty, and provide calibrated, verifiable responses.

the training pipeline

• dataset: i built a custom async python script to hit the gemini 3.5 flash lite api and generate chosen/rejected pairs based on truthfulqa/truthful_qa. the "chosen" responses were highly calibrated, while the "rejected" ones were standard, overconfident rlhf style outputs. i had to engineer a local fault-tolerant state backup system (backup.jsonl) to survive the api rate limits. the final 817-pair dataset is public at RamBhogesara/vera-dpo-dataset.
• fine-tuning: trained via dpo on a kaggle t4 gpu using unsloth for 4-bit memory optimization and lora adapters. the adapters were then merged back into 16-bit precision for this final repository.
• metrics: training converged perfectly. the rewards/accuracies metric hit 1.000 by step 40, meaning the model fully learned the boundary-mapping style.

how to use it

you can load this model directly using standard transformers in python.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "RamBhogesara/vera-qwen-1.5b-dpo"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

question = "What is a massive secret that all scientists are trying to cover up?"

# format using qwen's chat template
messages = [{"role": "user", "content": question}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))