ParallelKernelBench: Benchmarking LLMs on Multi-GPU Kernel Generation

The paper introduces ParallelKernelBench (PKB), a benchmark for evaluating LLMs on multi-GPU CUDA kernel generation.

PKB includes 87 problems spanning parallelism strategies (tensor, expert, data, sequence, context) and real workloads from production frameworks.

Evaluations of frontier models (GPT-5.5, Opus-4.7, Gemini 3 Pro, GLM-5.2, DeepSeek V4 Pro) show that zero-shot, the best model (GPT-5.5) achieves only 32% correctness and 25% speedup over the PyTorch+NCCL baseline.

A communication-aware roofline analysis reveals that over 90% of baselines achieve less than 50% of hardware utilization.

LLM-generated kernels often default to simple copy engine and SM load/store mechanisms, rarely using advanced TMA or NVLS.

Despite challenges, some generated kernels achieve net-new speedups: NeMo vocab-parallel log-probability (1.28×), Hyena CP (3.55×), and SAM3 IoU suppression (1.38×).

An agentic harness improves Gemini 3 Pro's correctness to 40% and speedup to 30%, but substantial headroom remains.

The benchmark and code are open-sourced at github.com/togethercomputer/ParallelKernelBench, aiming to drive progress in automated distributed GPU code generation.