The Year in AI Papers
The year/Google DeepMind

Paper 2607.02770

Gemma 4 Technical Report

Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, et al.

Published
Jul 2026
Research lab
Google DeepMind
Citations
18
GitHub
Not linked

01 In brief

Summary

Gemma 4 is a new family of open-weight, natively multimodal language models from Google DeepMind, featuring dense (2.3B, 4.5B, 12B, 31B) and Mixture-of-Experts (26B total, 3.8B active) architectures.

Key innovations include a thinking mode for reasoning traces, an encoder-free architecture for the 12B model that processes raw audio and image patches, and long-context efficiency via local-to-global attention ratios, p-RoPE, and KV cache sharing, reducing the global KV cache footprint by up to 37.5%.

The models also use quantization-aware training (QAT) and a multi-token prediction (MTP) drafter for speculative decoding.

Gemma 4 31B is the leading dense open model on Arena, rivaling much larger models.

Benchmarks show significant improvements over Gemma 3 across STEM, vision, audio, and long-context tasks, with the E2B model matching Gemma 3 27B performance with 10x fewer parameters.

Safety evaluations show major improvements over previous Gemma models.

The models are released under Apache 2.0.

02 From the paper

Abstract

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.