Quantum Assistant: Specialization of Multimodal Models for Quantum Computing

The first multimodal Vision-Language Model specialized for quantum computing with Qiskit

Model Description

This model is a fine-tuned version of Qwen3-VL-8B-Instruct specialized for quantum computing tasks using Qiskit 2.0. This model can interpret visual representations of quantum computing: circuit diagrams, Bloch spheres, and measurement histograms.

The model was trained using Rank-Stabilized Low-Rank Adaptation (rsLoRA) with rank 32 for 2 epochs on the Quantum Assistant Dataset, achieving significant improvements on multimodal quantum code generation tasks.

Key Capabilities

• Code Generation: Generate complete Qiskit code from natural language descriptions
• Function Completion: Complete function bodies from signatures and docstrings
• Visual Understanding: Interpret quantum circuit diagrams, Bloch spheres, and histograms
• Conceptual Explanations: Answer questions about quantum computing theory
• Qiskit 2.0 Compliant: Uses modern APIs (SamplerV2, EstimatorV2, generate_preset_pass_manager)

Evaluation Results

Evaluation was conducted on three complementary benchmarks: Qiskit HumanEval (151 function completion problems), Qiskit HumanEval Hard (151 code generation problems), and the synthetic test set (1,290 samples). Models were served via vLLM on A100 80GB PCIe with greedy decoding (temperature 0).

Consolidated Results

QHE: Qiskit HumanEval (function completion) · QHE Hard: code generation · †Qwen2.5-Coder-14B-Qiskit evaluated only on text samples (55% of synthetic dataset)

Key Improvements

Multimodal Advantage

The most significant differential is in multimodal samples: the fine-tuned model achieves 63.39% Pass@1 on image-based code generation vs 45.45% on text-only (+17.94 pp), validating that training on visual-textual samples develops domain-specific visual understanding capabilities.

Performance by Category

Training Strategy

The experimental strategy was organized in two phases: PEFT technique selection and hyperparameter optimization.

Phase 1: PEFT Variant Comparison

Five LoRA variants were compared with controlled configuration (r=16, α=32, 1 epoch):

Key findings:

• rsLoRA and DoRA achieved equivalent performance (Eval Loss 0.622)
• DoRA has 2.18× computational overhead (2,307s vs 1,060s) due to magnitude-direction decomposition
• rsLoRA selected for optimal performance-efficiency trade-off

Phase 2: Rank and Epoch Optimization

With rsLoRA selected, the impact of adapter rank and training duration was investigated:

Conclusions: rsLoRA with r=32 and 1-2 epochs maximizes generalization while avoiding memorization of the synthetic dataset.

Model Collection

This model is part of the Quantum Assistant collection. All models are merged versions ready for inference:

Usage

With vLLM

python -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 \
    --port 8000 \
    --model samuellimabraz/Qwen3-VL-8B-rslora-r32-2 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 12288 \
    --max-num-seqs 16 \
    --max-num-batched-tokens 49152 \
    --enable-chunked-prefill \
    --enable-prefix-caching

With Transformers

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "samuellimabraz/Qwen3-VL-8B-rslora-r32-2",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("samuellimabraz/Qwen3-VL-8B-rslora-r32-2")

messages = [
    {"role": "system", "content": "You are a quantum computing expert assistant specializing in Qiskit."},
    {"role": "user", "content": "Create a function that builds a 3-qubit GHZ state and returns the circuit."}
]

messages_with_image = [
    {"role": "system", "content": "You are a quantum computing expert assistant specializing in Qiskit."},
    {"role": "user", "content": [
        {"type": "image", "image": "path/to/circuit.png"},
        {"type": "text", "text": "Implement the quantum circuit shown in the image."}
    ]}
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt"
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=1024)
output = processor.batch_decode(
    generated_ids[:, inputs.input_ids.shape[1]:], 
    skip_special_tokens=True
)[0]
print(output)

Training Details

Dataset

• Training Data: Quantum Assistant Dataset
• Train Samples: 5,837 (45.1% multimodal)
• Validation Samples: 1,239 (45.2% multimodal)
• Task Distribution: 30% function completion, 32% code generation, 38% QA
• Categories: 7 quantum computing domains

Training Configuration

Freezing Strategy

Training Infrastructure

• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB VRAM)
• Training Time: ~35.8 minutes (2 epochs)
• Tracking: Weights & Biases | TensorBoard

System Prompt

You are a quantum computing expert assistant specializing in Qiskit.
Provide accurate, clear, and well-structured responses about quantum computing concepts,
algorithms, and code implementation. Use Qiskit 2.0 best practices.

Intended Uses & Limitations

Intended Uses

• Educational assistance: Learning quantum computing concepts with Qiskit
• Code generation: Creating Qiskit circuits from descriptions or diagrams
• Documentation: Understanding quantum circuit visualizations
• Research prototyping: Rapid development of quantum algorithms

Limitations

1. Domain specificity: Optimized for Qiskit 2.0; may generate deprecated APIs for older versions
2. Dataset size: Trained on 5,837 samples; may underperform on rare edge cases
3. Category imbalance: Better performance on circuits_and_gates than primitives_and_execution
4. Hardware specifics: Limited coverage of IBM Quantum hardware-specific optimizations
5. Execution: Generated code requires verification before running on real quantum hardware

Bias and Risks

• Model may perpetuate patterns from training data
• Visual understanding limited to common diagram styles in Qiskit documentation
• May generate syntactically correct but logically incorrect quantum algorithms
• Should not be used for production quantum computing without human review

Citation

If you use this model in your research, please cite:

@article{braz2026quantumassistant,
  title     = {Quantum Assistant: Specialization of Multimodal Vision-Language Models for Quantum Computing},
  author    = {Braz, Samuel Lima and Leite, Jo{\~a}o Paulo Reus Rodrigues},
  journal   = {Expert Systems with Applications},
  year      = {2026},
  issn      = {0957-4174},
  doi       = {10.1016/j.eswa.2026.133931},
  url       = {https://doi.org/10.1016/j.eswa.2026.133931},
  publisher = {Elsevier}
}

Related Resources

• Paper: doi:10.1016/j.eswa.2026.133931
• Dataset: samuellimabraz/quantum-assistant
• Model Collection: Quantum Assistant Models
• Demo: Quantum Assistant Space
• Code: GitHub Repository

Acknowledgments

• IBM Quantum and Qiskit team for open-source documentation
• Qwen Team for the base model
• UNIFEI (Universidade Federal de Itajubá) for academic support
• Advisor: Prof. João Paulo Reus Rodrigues Leite

License

This model is released under the Apache 2.0 License.