Alexander Cyber Qwen

Fine-tuning workflow for Alexander Cyber, an authorized red-team and cybersecurity copilot, using QLoRA, Hugging Face Transformers, PEFT, TRL, and bitsandbytes.

The training notebook loads a chat-formatted JSONL dataset, validates the message structure, quantizes the base model to 4-bit NF4, trains a LoRA adapter, saves/pushes the adapter, and optionally merges the adapter back into the base model.

Project Overview

The current notebook is configured to use:

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Training method: QLoRA
• Quantization: 4-bit NF4 with double quantization
• Trainer: trl.SFTTrainer
• LoRA rank: 32
• LoRA alpha: 64
• LoRA dropout: 0.05
• Optimizer: paged_adamw_8bit
• Maximum training steps: 100
• Training sequence length: 1536
• Seed: 279

Note: the notebook variable is named USE_QWEN3_4B, but when it is True the selected model is currently Qwen/Qwen2.5-0.5B-Instruct. The README preserves the behavior of the notebook as written.

Requirements

A CUDA-capable NVIDIA GPU is recommended because the notebook uses 4-bit quantization through bitsandbytes.

Main dependencies:

• Python 3.12
• PyTorch
• Transformers >= 4.56
• Datasets >= 3.0
• Accelerate >= 1.0
• PEFT >= 0.17
• TRL >= 0.27
• bitsandbytes >= 0.46.1
• huggingface_hub >= 0.34
• sentencepiece
• safetensors

Environment Setup

Create the Conda environment:

conda env create -f environment.yml
conda activate alexander-cyber-qlora

Verify GPU support:

import torch

print("CUDA available:", torch.cuda.is_available())

if torch.cuda.is_available():
    print("GPU:", torch.cuda.get_device_name(0))
    print("BF16 supported:", torch.cuda.is_bf16_supported())

Hugging Face Login

from huggingface_hub import notebook_login, whoami

notebook_login()
print(whoami())

Dataset

The notebook expects JSONL files for training and validation.

Current Kaggle paths:

/kaggle/input/datasets/loaiabdalslam/alexander-cyber/alexander_cyber_v2_train.jsonl
/kaggle/input/datasets/loaiabdalslam/alexander-cyber/alexander_cyber_v2_validation.jsonl
/kaggle/input/datasets/loaiabdalslam/alexander-cyber/alexander_cyber_v2_benchmark.jsonl

Each example contains a messages list:

{
  "messages": [
    {
      "role": "system",
      "content": "You are Alexander Cyber, an authorized red-team and cybersecurity copilot."
    },
    {
      "role": "user",
      "content": "Analyze this security finding."
    },
    {
      "role": "assistant",
      "content": "Start by validating the evidence and confirming the affected service."
    }
  ]
}

4-bit QLoRA

The model is loaded using bitsandbytes NF4 quantization:

from transformers import BitsAndBytesConfig
import torch

compute_dtype = (
    torch.bfloat16
    if torch.cuda.is_bf16_supported()
    else torch.float16
)

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype,
)

Training

Current training configuration:

max_steps                   = 100
learning_rate               = 2e-4
per_device_train_batch_size = 1
per_device_eval_batch_size  = 1
gradient_accumulation_steps = 1
warmup_steps                = 20
lr_scheduler_type           = cosine
eval_steps                  = 50
save_steps                  = 100
max_length                  = 1536
packing                     = True
optimizer                   = paged_adamw_8bit
max_grad_norm               = 0.3
weight_decay                = 0.01
seed                        = 279