The Year in AI Papers
The year/Google DeepMind

Paper 2512.20605

Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning

Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel, Angelika Steger, Maciej Wolczyk, Johannes von Oswald, Nino Scherrer, Kaitlin Maile, Guillaume Lajoie, Blake A. Richards, Rif A. Saurous, James Manyika, et al.

Published
Dec 2025
Research lab
Google DeepMind
Citations
5
GitHub
Not linked

01 In brief

Summary

This paper introduces a method for hierarchical reinforcement learning (RL) in autoregressive models by discovering and using temporally-abstract actions within the model's internal representations.

The authors show that autoregressive models pretrained on next-token prediction learn latent representations of subgoals in their residual stream activations.

They introduce a metacontroller, a higher-order non-causal sequence model, that learns to generate linear controllers steering these activations, effectively executing sequences of abstract actions with learned termination conditions.

This enables 'internal RL', where RL is performed in the abstract action space rather than at the token level.

Experiments on grid world and MuJoCo ant tasks with hierarchical structure show that internal RL achieves high success rates on sparse-reward tasks where standard RL finetuning (e.g., GRPO) and prior hierarchical RL methods (e.g., CompILE) fail.

The approach demonstrates compositional generalization to novel subgoal sequences and significantly improves exploration and credit assignment efficiency.

02 From the paper

Abstract

Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. During RL, these models explore by generating new outputs, one token at a time. However, sampling actions token-by-token can result in highly inefficient learning, particularly when rewards are sparse. Here, we show that it is possible to overcome this problem by acting and exploring within the internal representations of an autoregressive model. Specifically, to discover temporally-abstract actions, we introduce a higher-order, non-causal sequence model whose outputs control the residual stream activations of a base autoregressive model. On grid world and MuJoCo-based tasks with hierarchical structure, we find that the higher-order model learns to compress long activation sequence chunks onto internal controllers. Critically, each controller executes a sequence of behaviorally meaningful actions that unfold over long timescales and are accompanied with a learned termination condition, such that composing multiple controllers over time leads to efficient exploration on novel tasks. We show that direct internal controller reinforcement, a process we term "internal RL", enables learning from sparse rewards in cases where standard RL finetuning fails. Our results demonstrate the benefits of latent action generation and reinforcement in autoregressive models, suggesting internal RL as a promising avenue for realizing hierarchical RL within foundation models.