Qwen3.8 27B OpenVINO INT4

This repository contains an unofficial OpenVINO™ IR conversion of Qwen/Qwen3.8-27B with INT4 weight compression.

The model is intended for local vision-language generation on Intel hardware. It accepts text, images, and video in the upstream architecture. The included examples cover image-and-text inference through OpenVINO GenAI (recommended) and Optimum Intel.

Original model

• Original model: Qwen/Qwen3.8-27B
• Original creator: Qwen
• Architecture: dense 27B causal language model with a vision encoder
• License: Apache 2.0
• Converted format: OpenVINO IR
• Weight format: asymmetric INT4
• Task: image-text-to-text / vision-language generation

This is an unofficial converted model repository. It is not an official Qwen or OpenVINO release. Refer to the upstream model card for full model details, recommended generation settings, intended use, safety notes, and limitations.

Conversion

This model was exported with the development version of Optimum Intel. NNCF performed INT4 weight compression as part of the OpenVINO export.

optimum-cli export openvino \
  --model Qwen/Qwen3.8-27B \
  --task image-text-to-text \
  --trust-remote-code \
  --weight-format int4 \
  --group-size 128 \
  --ratio 1.0 \
  ov_Qwen3.8-27B_int4

Quantization

• Weight format: INT4 asymmetric
• Group size: 128
• Ratio: 1.0
• Export tool: Optimum Intel
• Compression backend: NNCF through Optimum Intel
• Runtime format: OpenVINO IR

INT4 weight compression reduces model storage and memory requirements compared with BF16/FP16. Quantization can change numerical behavior and output quality, so validate the converted model for your application.

Compatibility

Qwen3.8 support is new and requires development/nightly packages. The conversion used Transformers 5.2, the latest Optimum Intel development version, and nightly OpenVINO, OpenVINO Tokenizers, and OpenVINO GenAI packages available at export time. Exact installed versions are recorded in conversion_metadata.json.

Install the example dependencies:

pip install -r examples/requirements.txt

OpenVINO GenAI usage

Run from inside the model repository:

python examples/run_qwen38_vlm_openvino_genai.py \
  --model-dir . \
  --device GPU \
  --max-new-tokens 128 \
  --prompt "Describe this image in one paragraph."

Use --device CPU for CPU inference. The example prints the OpenVINO devices detected on the system before loading the model.

CPU, GPU, and NPU benchmark

The included benchmark runs every selected OpenVINO device in a separate process and produces Markdown, CSV, and JSON comparison reports:

python examples/benchmark_qwen38_vlm_xpu.py \
  --model-dir . \
  --devices CPU GPU NPU \
  --warmup-runs 1 \
  --runs 3 \
  --max-new-tokens 64 \
  --output-dir benchmark_results

The comparison table includes model load time, time to first token (TTFT), mean and percentile inter-token latency (ITL), token throughput, end-to-end latency, vision-encoding time, output-token count, and peak/delta process RAM. TTFT, ITL, throughput, and token counts use OpenVINO GenAI's native performance metrics. Unsupported devices and device-specific compilation failures are recorded in the reports without stopping the remaining benchmarks.

The script prints pipeline, warm-up, measured-run, and periodic heartbeat messages while each isolated worker is active. The default per-device timeout is 30 minutes; change it with --device-timeout SECONDS.

Use --devices GPU to benchmark one device, or change their order as needed. The default input is a public Hugging Face documentation image. For offline or repeatable testing, pass a local file with --image path/to/image.jpg. Peak process RAM is RSS and does not represent dedicated GPU VRAM. For useful comparisons, keep the prompt, image, token limit, package versions, and power settings identical across runs.

Reference GPU benchmark

The following result was measured by the repository author on one local AI PC on August 28, 2026. It is a reproducibility reference, not a vendor benchmark or a guarantee of performance on other systems.

Test system

Workload

Aggregate result

TTFT, ITL/TPOT, throughput, token counts, and component timings are reported by OpenVINO GenAI. End-to-end latency is measured around the complete generate() call. RAM was sampled every 50 ms. Because the Arc B390 is an integrated GPU, the process-RSS figures include shared host-memory effects and must not be read as dedicated VRAM usage.

Optimum Intel usage

python examples/run_qwen38_vlm_optimum.py \
  --model-dir . \
  --device CPU \
  --max-new-tokens 128 \
  --prompt "Describe this image in one paragraph."

Use as a local model in VS Code Copilot Chat (OpenVINO Model Server)

This model can be served with OpenVINO Model Server (OVMS) 2026.4.0 or newer through its OpenAI-compatible API and added to the VS Code Copilot Chat model picker as a Custom Endpoint model, running fully locally on an Intel GPU.

ovms --rest_port 8000 --rest_bind_address 127.0.0.1 \
  --model_name qwen3.8-27b-ov --model_path <path-to-this-repo> \
  --task text_generation --target_device GPU \
  --enable_prefix_caching true --cache_dir <cache-dir>

OVMS auto-detects the qwen3coder tool parser and qwen3 reasoning parser, so chat, tool calling (Copilot Agent mode), and reasoning output work without extra flags. Tested on Windows 11 with an Intel Arc B390 iGPU.

Step-by-step notebook (install, serve, VS Code setup, troubleshooting): github.com/morteza89/vscode-copilot-local-model-ovms

Notes and limitations

NPU status: not applicable with the tested OpenVINO release

NPU inference is not applicable to this repository with the tested OpenVINO 2026.4.0rc1 and OpenVINO GenAI 2026.4.0.0rc1 stack. This repository uses asymmetric INT4 weights, while the current OpenVINO GenAI NPU path requires a symmetric INT4 or NF4 export.

A separate symmetric INT4 channel-wise export was also tested on the same Intel AI Boost NPU. VLMPipeline failed during compilation, before inference, with OneShotBufferizeVPU2VPUIP failed: Only supports FP16. Inspection showed BF16/FP32 paths in the exported vision-language graphs, and an isolated vision merger compile also failed on an unbounded dynamic image-token dimension. As a result, no NPU latency or throughput result is reported.

Qwen3.8 27B is not currently listed as NPU-verified in the OpenVINO verified-model table. This limitation may change in a future OpenVINO or NPU driver release. CPU and GPU inference remain applicable to this model repository.

• This is a native vision-language model, not a text-only checkpoint.
• OpenVINO GenAI VLMPipeline is the recommended local generative runtime.
• The model is experimental in the current OpenVINO development stack.
• OpenVINO Model Server 2026.4.0 was tested on Windows 11 with an Intel Arc B390 iGPU (chat, tool calling, reasoning). Older OVMS releases are not supported.
• The repository inherits the upstream model's limitations. Additional differences can result from OpenVINO conversion, INT4 compression, package versions, device choice, and generation configuration.

License and attribution

The upstream model is licensed under Apache License 2.0. This converted model is distributed under the same license, included as LICENSE. All rights to the original model, training, and associated materials remain with their respective authors.