Qwen3-1.7B Chat

This model is a Direct Preference Optimized (DPO) variant of Qwen3-1.7B-UltraChat-SFT, which itself is a supervised fine-tuned version of Qwen3-1.7B-Base.

Starting from the SFT model, preference alignment was performed using Direct Preference Optimization (DPO) on multiple human preference datasets to improve response quality, reasoning, truthfulness, and conversational naturalness. The resulting model produces responses that are generally rich, more helpful, coherent, and better aligned with user preferences while preserving the conversational capabilities learned during supervised fine-tuning. This model does not support thinking, vision or tool calling yet.

Preference optimization was performed using LoRA adapters, which were merged into the base weights after training.

View the LoRA Adapter. View the GitHub repository for the complete post-training pipeline.

Quantizations of this model are present in ayushshah/Qwen3-1.7B-Chat-GGUF. You can use them in LLaMA.cpp, Ollama or any other applications you like. This model and its quantizations are also available on Ollama. You can directly pull and use this model from Ollama. Visit the Ollama model page.

Model Overview

Qwen3-1.7B-Chat has the following features:

• Type: Causal Language Model
• Training Stage: Pretraining → Supervised Fine-Tuning → Direct Preference Optimization
• Number of Parameters: 1.7B
• Number of Parameters (Non-Embedding): 1.4B
• Number of Layers: 28
• Number of Attention Heads (GQA): 16 for Q and 8 for KV
• Context Length: 32,768

Training

Stage 1: Supervised Fine-Tuning

• Method: Supervised Fine-Tuning (SFT)
• Dataset: UltraChat-200k
• Framework: Unsloth
• Fine-Tuning: LoRA + embed_tokens + lm_head

Stage 2: Preference Alignment

• Method: Direct Preference Optimization (DPO)
• Frameworks: TRL + Unsloth
• Fine-Tuning: LoRA
• Precision: BF16
• Environment: Windows Subsystem for Linux (WSL)
• GPU: RTX 4090 24 GB using FlashAttention 2, xFormers and Triton

Dataset Preprocessing

1. The datasets were first filtered by removing rows with low chosen ratings. For argilla/distilabel-capybara-dpo-7k-binarized and argilla/ultrafeedback-binarized-preferences-cleaned, rows where the chosen rating was < 4 were removed. Similarly, for argilla/distilabel-intel-orca-dpo-pairs, rows where the preference was tie, the chosen rating was < 8, or in_gsm8k_train was true were removed.
2. For argilla/ultrafeedback-binarized-preferences-cleaned, only rows where the difference between the chosen and rejected ratings was >= 1 were retained.
3. Any row where either the chosen or rejected conversation exceeded 1024 tokens was removed.
4. The argilla/ultrafeedback-binarized-preferences-cleaned dataset was significantly larger than the other datasets; therefore, it was undersampled by randomly selecting 15,000 rows.
5. Finally, the datasets were concatenated and shuffled.

DPO Configuration

LoRA was applied to: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.

Evaluation

Evaluation was performed using EleutherAI's Language Model Evaluation Harness.

Blind A/B Evaluation tests with LLM-as-a-judge and Human-In-The-Loop calibration preferred the DPO model over the SFT model in majority of the runs.

Usage

Unsloth (Recommended)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="ayushshah/Qwen3-1.7B-Chat",
    max_seq_length=2048,
    dtype="bfloat16",
)

FastLanguageModel.for_inference(model)

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "ayushshah/Qwen3-1.7B-Chat"
)

model = AutoModelForCausalLM.from_pretrained(
    "ayushshah/Qwen3-1.7B-Chat",
    torch_dtype="auto",
    device_map="auto"
)

Limitations

Although preference optimization improves conversational quality, reasoning, and helpfulness, this model is still limited by the capabilities and knowledge of the underlying Qwen3-1.7B architecture. It may occasionally generate incorrect or hallucinated information, struggle with complex multi-step reasoning, or fail to follow ambiguous instructions accurately. Furthermore, this model has not undergone reinforcement learning from human feedback (RLHF) or additional safety-specific alignment beyond DPO and should not be relied upon for high-stakes applications without appropriate human oversight.

Compared to the SFT model, the primary observed trade-off was a slight reduction in strict instruction-following ability, accompanied by a tendency to generate more elaborate responses and a slightly higher likelihood of hallucinations on complex queries.

Citation

@misc{qwen3technicalreport,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2025},
  eprint={2505.09388},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2505.09388}
}