Model Card for uzairkhanux/Qwen3.8-27B-FP4

Model Details

Model Description

This model is a fine-tuned and 4-bit (FP4) quantized version of the Qwen/Qwen3.8-27B large language model. Following the fine-tuning process on the main model, it was quantized on-the-fly using the bitsandbytes library and pushed directly to the Hugging Face Hub. The quantization reduces the model's memory footprint from approximately 54GB (in native 16-bit) down to roughly 14GB, making it strictly accessible for deployment and inference on hardware with constrained VRAM limits, such as a single 16GB GPU or Kaggle 2x T4 setups.

• Developed by: uzairkhanux (Fine-tuning and Quantization) / Original base model by Qwen Team
• Shared by: uzairkhanux
• Model type: Causal Language Model (Transformer-based, Fine-tuned, FP4 Quantized)
• Language(s) (NLP): Multilingual (including English, Chinese, and others supported by the base model)
• License: Governed by the license of the original Qwen/Qwen3.8-27B base model.
• Finetuned from model: Qwen/Qwen3.8-27B

Model Sources

• Repository: https://huggingface.co/uzairkhanux/Qwen3.8-27B-FP4
• Base Model: https://huggingface.co/Qwen/Qwen3.8-27B

Uses

Direct Use

This model is intended for direct text generation, zero-shot/few-shot prompting, and conversational inference tailored to the specific domain it was fine-tuned on. It is designed to be loaded directly via the Hugging Face transformers ecosystem utilizing bitsandbytes for 4-bit loading.

Out-of-Scope Use

Because this specific version relies on CPU offloading to bypass memory spikes during initialization across 2x T4 GPUs, highly latency-sensitive production environments should avoid using it without dedicated high-VRAM hardware (such as A100s), as CPU-to-GPU memory swapping heavily degrades tokens-per-second throughput.

Bias, Risks, and Limitations

Like all foundational Large Language Models, this model inherits the biases present in its original training data and any data used during the fine-tuning phase. It may generate inaccurate, biased, or objectionable content. Furthermore, the 4-bit (FP4) quantization process is lossy; users should expect a slight degradation in complex reasoning compared to the native 16-bit/BF16 weights.

Recommendations

Users evaluating this model for specific downstream tasks should conduct rigorous testing against the unquantized baseline to measure the impact of FP4 precision loss. Always deploy appropriate safety guardrails if integrating this into user-facing applications.

How to Get Started with the Model

Use the ready-to-paste Python code below to load this model and run inference. This configuration specifically accounts for memory limitations by offloading excess weights to system RAM, preventing CUDA Out of Memory exceptions.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "uzairkhanux/Qwen3.8-27B-FP4"

# Configure FP4 Quantization WITH CPU Offloading capability
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="fp4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=False,
    llm_int8_enable_fp32_cpu_offload=True 
)

# Apply safety buffers to prevent CUDA Out Of Memory spikes (e.g., for Kaggle 2x T4)
memory_limits = {
    0: "12GB",      
    1: "12GB",      
    "cpu": "25GB"   
}

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    device_map="auto",
    max_memory=memory_limits,
    low_cpu_mem_usage=True
)

# Inference generation
prompt = "Explain the core mechanics of a transformer neural network."
messages = [
    {"role": "system", "content": "You are a highly capable AI assistant."},
    {"role": "user", "content": prompt}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

with torch.no_grad():
    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=512,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)