BankAssist Qwen3

Developed by: Waseem Ghanem

BankAssist Qwen3 is a standalone fine-tuned version of Qwen/Qwen3-0.6B for banking intent classification.

I first fine-tuned Qwen3-0.6B on the Banking77 dataset using LoRA with PEFT and TRL. After training, I merged the learned LoRA weights into the base Qwen model to create this standalone version.

Unlike my separate LoRA adapter repository, this model can be loaded directly with Hugging Face Transformers without using PeftModel.

Example

Customer:
"My card still hasn't arrived."

Prediction:
card_arrival

What This Model Does

The model takes a customer banking message and generates one of the 77 intent labels defined by Banking77.

For example:

"My transfer is still pending."
        ↓
pending_transfer

The task is treated as text generation rather than using a traditional classification head.

Model Details

• Developed by: Waseem Ghanem
• Base model: Qwen/Qwen3-0.6B
• Dataset: Banking77
• Training examples: 9,993
• Test examples: 3,076
• Intent categories: 77
• Fine-tuning method: Supervised fine-tuning with LoRA
• Trainer: TRL SFTTrainer
• Final format: LoRA weights merged into the base model
• Language: English
• License: Apache 2.0

How I Built It

I started with the pretrained:

Qwen/Qwen3-0.6B

and fine-tuned it using Banking77.

A raw training example looks like:

Text:
"I am still waiting for my card."

Label:
card_arrival

Before training, I converted each example into Qwen's conversational format:

System:
Classify the customer's banking message into the correct banking intent.
Return only the intent name.

User:
I am still waiting for my card.

Assistant:
card_arrival

Qwen's chat template formats the conversation, and the tokenizer converts the text into token IDs that can be processed by the model.

The overall training pipeline was:

Banking77
    ↓
Qwen chat formatting
    ↓
Tokenization
    ↓
Qwen3-0.6B
    ↓
LoRA adapters
    ↓
Supervised fine-tuning
    ↓
Backpropagation
    ↓
LoRA weights updated
    ↓
Merge LoRA into Qwen
    ↓
Standalone BankAssist model

Fine-Tuning with LoRA

I used LoRA through Hugging Face PEFT instead of updating all of Qwen's original parameters.

During training, the original Qwen weights stayed frozen while smaller LoRA matrices were trained.

LoRA was applied to these attention projection modules:

q_proj
k_proj
v_proj
o_proj

The training loop was handled by TRL's SFTTrainer:

Forward pass
    ↓
Loss
    ↓
Backpropagation
    ↓
Gradients
    ↓
Optimizer update
    ↓
LoRA parameters updated

After training, I merged the LoRA adapter into the base model using PEFT's merge_and_unload() functionality.

That produced the standalone model published in this repository.

Training Configuration

Training on Limited Hardware

I trained the adapter locally on:

• GPU: NVIDIA GeForce RTX 3050
• VRAM: 4 GB

Because I only had 4 GB of VRAM available, I used several techniques to reduce memory usage:

• LoRA instead of full fine-tuning
• batch size 1
• gradient accumulation
• FP16
• gradient checkpointing
• maximum sequence length of 256

Using a batch size of 1 with 8 gradient accumulation steps gave an effective batch size of 8 without requiring all 8 examples to fit into GPU memory at the same time.

Evaluation

I evaluated both the original Qwen3-0.6B model and the fine-tuned BankAssist model on the same full Banking77 test split.

The evaluation used:

• 3,076 unseen examples
• all 77 Banking77 intents
• the same prompt
• the same tokenizer
• deterministic generation
• exact intent-label matching

I also counted predictions where the generated label was not one of the valid Banking77 intent names.

Results

Fine-tuning improved accuracy by 25.72 percentage points.

Accuracy increased from:

22.07% → 47.79%

Invalid predictions decreased from:

335 → 160

The fine-tuned model more than doubled the base model's accuracy in this evaluation.

This is still an experimental model and should not be treated as a production banking system.

How to Use

Install the required libraries:

pip install torch transformers accelerate

Then load the model directly from Hugging Face:

from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "WaseemGh98/bankassist-qwen3"

tokenizer = AutoTokenizer.from_pretrained(MODEL)

model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    torch_dtype="auto",
    device_map="auto"
)

model.eval()

Then prepare a banking message:

message = "My card still hasn't arrived."

messages = [
    {
        "role": "system",
        "content": (
            "Classify the customer's banking message into the correct "
            "banking intent. Return only the intent name."
        )
    },
    {
        "role": "user",
        "content": message
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

Run inference:

import torch

inputs = tokenizer(
    text,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=20,
        do_sample=False
    )

generated_tokens = outputs[0][inputs["input_ids"].shape[1]:]

prediction = tokenizer.decode(
    generated_tokens,
    skip_special_tokens=True
).strip()

print(prediction)

Example output:

card_arrival

Standalone Model vs LoRA Adapter

I also published the original LoRA adapter separately:

WaseemGh98/waseem-bankassist-qwen3-lora

The difference is:

LoRA repository
Qwen3-0.6B
    +
BankAssist adapter
    ↓
requires PEFT / PeftModel

while this repository contains:

Qwen3-0.6B
    +
merged LoRA changes
    ↓
standalone BankAssist model

So this version can be loaded directly using:

AutoModelForCausalLM.from_pretrained(
    "WaseemGh98/bankassist-qwen3"
)

Dataset Attribution

Training used:

mteb/banking77

Banking77 was originally introduced by PolyAI as a banking customer-support intent classification dataset containing 77 intent categories.

Original dataset:

PolyAI/banking77

Original dataset license:

CC BY 4.0

Licensing

The base model, Qwen/Qwen3-0.6B, is distributed under the Apache 2.0 license.

The original Banking77 dataset is distributed under CC BY 4.0.

This model is a fine-tuned derivative of Qwen3-0.6B. I do not claim ownership of the original Qwen model or the Banking77 dataset.

Developer

Waseem Ghanem

Hugging Face:

WaseemGh98