Lura-1.3-500m
Created by: Noeme
Base Model: Qwen/Qwen2.5-0.5B-Instruct
Model Type: Causal Language Model (Full Parameter Fine-Tuning)
Overview
Lura 1.3 is an experimental fine-tuned model based on Qwen/Qwen2.5-0.5B-Instruct. It was developed to achieve stronger, more reliable results in instruction following, multi-constraint obedience, and strict output formatting compared to the standard base model.
While small 0.5B models often struggle with complex rules and negative constraints, Lura 1.3 has been specifically aligned to better adhere to difficult, rule-heavy prompts.
Benchmark & Evaluation
Lura 1.3 was evaluated head-to-head against Qwen/Qwen2.5-0.5B-Instruct across an objective benchmark consisting of 100 test samples designed to test strict instruction following, formatting adherence, and negative constraints.
100-Sample Test Results
| Model | Parameters | Benchmark Score (out of 100) | Accuracy (%) |
|---|---|---|---|
| noeme/Lura-1.3-500m | 500M | 52 / 100 | 52.0% |
| Qwen/Qwen2.5-0.5B-Instruct (Base) | 500M | 43 / 100 | 43.0% |
Evaluation Methodology:
Evaluated strictly on 100 diverse test samples testing:
- Strict formatting: Exact JSON objects/arrays without conversational wrapper text.
- Delimiter adherence: Custom delimiters (e.g.,
######,-bullets, commas without spaces).- Length & structural control: Exact sentence counts, lines, and custom tables.
- Factual accuracy: Direct correctness against ground-truth references.
Key Improvements in Instruction Following
- Clean Raw JSON Generation: Emits valid, structured JSON directly without wrapping responses in unwanted conversational filler or conversational text.
- Delimiter & List Obedience: Consistently respects requested formatting markers (e.g.,
-bullet points, custom separators like######, and semicolon-separated items). - Structural Sentence Control: Superior performance on exact sentence-count constraints (e.g., exactly one sentence or exactly four sentences).
- Negative Constraint Handling: Better capability to adhere to negative constraints compared to typical sub-billion models.
- Repetition Mitigation: Dramatically reduces the tendency of small models to collapse into runaway, infinite token loops on open-ended generation tasks.
⚠️ Important Disclaimer & Limitations
- Experimental Version: This model is an early experimental release built for testing, research, and learning purposes.
- Potential for Hallucinations: Like many compact models (500M parameters), Lura 1.3 can produce hallucinations, factual inaccuracies, or minor grammar/translation errors.
- Intended Use: This model is built for experimentation and exploration only. It is not recommended for production use, high-stakes tasks, or critical professional workflows.
How to Use
You can easily run inference with Lura 1.3 using the Hugging Face transformers library:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_ID = "noeme/Lura-1.3-500m"
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
device_map="auto"
)
# Prepare prompt using Qwen chat template
messages = [
{
"role": "user",
"content": "Write a short paragraph about mountains in all lowercase letters with no capitals at all."
}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
# Recommended generation parameters
with torch.no_grad():
output_ids = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.2,
top_p=0.9,
repetition_penalty=1.15,
do_sample=True,
pad_token_id=tokenizer.pad_token_id
)
new_tokens = output_ids[0][inputs.input_ids.shape[1]:]
response = tokenizer.decode(new_tokens, skip_special_tokens=True).strip()
print(response)