Qwen2.5-0.5B Financial Domain Adapted LLM
This model is a domain-adapted version of Qwen2.5-0.5B specifically tailored for financial NLP tasks. It features a custom Byte-Level BPE Tokenizer integrated directly into the base architecture via Tokenizer Surgery and aligned using TRL SFTTrainer with PEFT/LoRA on Apple Silicon (MPS).
Model Details
Model Description
Standard LLM tokenizers often fragment specialized financial terminology (such as ticker symbols like $NVDA, key metrics like EBITDA/CAGR, and currency pairs like EUR/USD) into excessive subwords, leading to increased prompt inflation and potential [UNK] errors.
This model addresses these issues by:
- Extending the base vocabulary with custom financial domain tokens.
- Resizing the underlying embedding matrix ($E \in \mathbb{R}^{V_{new} \times d}$) and language model head.
- Conducting continued pre-training using LoRA to train the new token embeddings while preserving base model capabilities.
- Developed by: Durga
- Model type: Causal Language Model (Transformer Decoder)
- Language(s) (NLP): English (Financial Domain)
- License: Apache 2.0
- Finetuned from model:
Qwen/Qwen2.5-0.5B
Uses
Direct Use
- Financial text generation and completion.
- Processing financial filings (SEC 10-K, 10-Q), earnings transcripts, and analyst reports.
- Reduced context fragmentation for domain-specific prompt engineering.
Out-of-Scope Use
- Generating binding financial advice, automated trading decisions, or official financial audits without human oversight.
Bias, Risks, and Limitations
- Domain Specificity: The model has been continued pre-trained on a focused financial corpus. It is designed for domain text compression and processing rather than general open-domain chat.
- Accuracy: Outputs should be independently verified when used for quantitative or regulatory compliance tasks.
How to Get Started with the Model
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
# Load model and custom adapted tokenizer
model_id = "durganani60/qwen2.5-0.5b-financial-adapted"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto"
)
# Test prompt containing financial domain tokens
prompt = "Q3 2024 financial report: $NVDA recorded EBITDA growth above 14.5% with expanding FCF margin."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
'''