Paper 2604.08570
QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 6 stars
01 In brief
Summary
QuanBench+ is a unified benchmark for evaluating LLMs on quantum code generation across Qiskit, PennyLane, and Cirq, holding task intent constant while varying the framework.
It includes 42 tasks spanning quantum algorithms, gate decomposition, and state preparation, and uses executable functional tests with Pass@1, Pass@5, and KL-divergence-based acceptance for probabilistic outputs.
The strongest one-shot scores are 59.5% (Qiskit), 54.8% (Cirq), and 42.9% (PennyLane); with feedback-based repair, these rise to 83.3%, 76.2%, and 66.7%, respectively.
Results show Qiskit is consistently easiest and PennyLane hardest, indicating framework-specific familiarity drives performance.
Prefill mainly reduces boilerplate errors, while feedback repairs surface-level issues but leaves semantic mistakes dominant.
The benchmark excludes fidelity as a metric, favoring operational correctness.
Limitations include task imbalance and limited framework coverage, with future work planned for broader evaluation and tool-augmented workflows.
Overall, reliable multi-framework quantum code generation remains unsolved, requiring better quantum software exposure and reasoning support beyond model scale alone.
The paper answers three research questions on cross-framework accuracy, prefill impact, and feedback repair effectiveness, concluding that progress is real but incomplete.
The benchmark is available at https://github.com/JawadKotaichh/quanbench-plus.
The study evaluates 12 models, including Gemini-3-Pro and GPT-5.1, and uses Python 3.10 with specific library versions.
The KL threshold is calibrated at 0.05 from canonical executions.
Error analysis shows most failures are semantic, not syntactic, and feedback reduces total errors from 977 to 665, with remaining errors concentrated in reasoning.
The paper also discusses threats to validity, including canonical solution correctness and category imbalance.
Future work includes extending to more…
02 From the paper
Abstract
Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.