The Year in AI Papers
The year/Labs/Together AI

Research lab

Together AI

Open models, systems research, inference, training, and the infrastructure behind production AI.

Papers
19
Citations
281
Official code
8

19 papers from Together AI

01

ICML 2026

ParallelKernelBench: Benchmarking LLMs on Multi-GPU Kernel Generation

The paper introduces ParallelKernelBench (PKB), a benchmark for evaluating LLMs on multi-GPU CUDA kernel generation. PKB includes 87 problems spanning parallelism strategies (tensor, expert, data, sequence, context) and real workloads from production frameworks. Evaluations of frontier models (GPT-5.5, Opus-4.7, Gemini 3 Pro, GLM-5.2, DeepSeek V4 Pro) show…

Willy Chan, Nathan Paek, Simon Guo, Simran Arora, et al.
Published
Jun 2026
Citations
Not indexed
Code
47 stars
Read summary Original
02

arXiv.org

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

OSCAR is an INT2 KV-cache quantization method that uses attention-aware covariance structures, estimated offline, to derive fixed rotations and clipping thresholds, aligning quantization with downstream attention. It composes rotations as R = U · H_Had · P_br, where U is the eigenbasis of query-aware (Q^T Q) or score-aware (V^T S^T S V) covariance, H_Had…

Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, et al.
Published
May 2026
Citations
2
Code
555 stars
Read summary Original
03

arXiv.org

Mamba-3: Improved Sequence Modeling using State Space Principles

Mamba-3 introduces three methodological improvements to state space models (SSMs) for better quality, capability, and inference efficiency. First, exponential-trapezoidal discretization generalizes prior heuristics, enabling a more expressive recurrence that can replace the short causal convolution. Second, complex-valued state transitions, implemented via…

Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, et al.
Published
Mar 2026
Citations
68
Code
Not linked
Read summary Original
04

arXiv.org

FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

FlashAttention-4 addresses the asymmetric hardware scaling of NVIDIA Blackwell GPUs (B200/GB200), where tensor core throughput doubles compared to Hopper, but shared memory bandwidth and exponential unit throughput do not scale accordingly. This shifts bottlenecks to non-matmul operations. The paper introduces techniques to mitigate these bottlenecks: (1)…

Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timothy Liu, et al.
Published
Mar 2026
Citations
36
Code
Not linked
Read summary Original
05

arXiv.org

V1: Unifying Generation and Self-Verification for Parallel Reasoners

The paper introduces V1, a framework that unifies generation and self-verification for parallel reasoning in LLMs. It identifies that pointwise self-verification suffers from calibration collapse, while self-aggregation methods like RSA lead to diversity collapse. V1 comprises two components: V1-Infer, an uncertainty-guided pairwise ranking algorithm using…

Harman Singh, Xiuyu Li, Kusha Sareen, Monishwaran Maheswaran, et al.
Published
Mar 2026
Citations
7
Code
39 stars
Read summary Original
06

arXiv.org

Speculative Speculative Decoding

The paper introduces speculative speculative decoding (SSD), a framework that parallelizes drafting and verification in speculative decoding (SD) by having the draft model predict likely verification outcomes and pre-speculate for them while verification runs. This eliminates drafting overhead on cache hits, making SSD lossless and faster than SD. The…

Tanishq Kumar, Tri Dao, Avner May
Published
Mar 2026
Citations
11
Code
Not linked
Read summary Original
07

arXiv.org

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

The paper introduces UPipe, a context parallelism technique that reduces activation memory in Transformer training by chunking attention heads. It builds on DeepSpeed-Ulysses, processing attention in stages with a subset of heads (U) at a time, reusing buffers to cut intermediate memory from O(H) to O(U). For Qwen3-32B, this reduces attention intermediate…

Ravi Ghadia, Maksim Abraham, S. Vorobyov, Max Ryabinin
Published
Feb 2026
Citations
0
Code
25 stars
Read summary Original
08

arXiv.org

ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

ThunderAgent is a program-aware agentic inference system that improves throughput for multi-turn LLM agent workflows by abstracting them as agentic programs with metadata (ID, context length, tool environments, placement, phase, status). It introduces a program-aware scheduler with state-aware pausing and dynamic migration across GPU nodes to reduce KV…

Hao Kang, Ziyang Li, Xinyu Yang, Weili Xu, et al.
Published
Feb 2026
Citations
13
Code
Not linked
Read summary Original
09

arXiv.org

When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

Aurora is a unified training-serving system that addresses limitations of conventional speculative decoding, which separates offline speculator training from online serving, causing deployment lag, delayed utility feedback, and domain-drift degradation. Aurora closes the loop by continuously learning a speculator from live inference traces, framing it as…

Junxiong Wang, Fengxiang Bie, Jisen Li, Zhongzhu Zhou, et al.
Published
Feb 2026
Citations
1
Code
Not linked
Read summary Original
10

arXiv.org

DSGym: A Holistic Framework for Evaluating and Training Data Science Agents

DSGym is a standardized framework for evaluating and training data science agents in isolated, stateful execution environments. It addresses limitations in existing benchmarks, which often allow tasks to be solved without accessing data files (shortcut solvability), lack cross-benchmark standardization, and have narrow domain coverage. DSGym provides a…

Fan Nie, Junlin Wang, Harper Hua, Federico Bianchi, et al.
Published
Jan 2026
Citations
12
Code
58 stars
Read summary Original
11

arXiv.org

Learning to Discover at Test Time

The paper introduces TTT-Discover, a method that performs reinforcement learning at test time to solve scientific discovery problems. Unlike prior test-time scaling methods that use a frozen LLM for search, TTT-Discover continues to train the LLM on the specific test problem, using an entropic objective and PUCT-based state reuse to prioritize the most…

Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, et al.
Published
Jan 2026
Citations
71
Code
613 stars
Read summary Original
12

arXiv.org

Escaping the Verifier: Learning to Reason via Demonstrations

The paper introduces RARO (Relativistic Adversarial Reasoning Optimization), a method to train large language models (LLMs) to reason using only expert demonstrations, without task-specific verifiers or human preferences. RARO frames the problem as inverse reinforcement learning, setting up an adversarial game between a policy and a relativistic critic…

Locke Cai, Ivan Provilkov
Published
Nov 2025
Citations
5
Code
Not linked
Read summary Original
13

arXiv.org

CDLM: Consistency Diffusion Language Models for Faster Sampling

CDLM (Consistency Diffusion Language Models) is a training-based method to accelerate inference in Diffusion Language Models (DLMs) by addressing two bottlenecks: excessive refinement steps and incompatibility with KV caching. It integrates consistency modeling to reduce sampling steps via multi-token finalization and enforces a block-wise causal attention…

Minseo Kim, Chenfeng Xu, Coleman Hooper, Harman M. Singh, et al.
Published
Nov 2025
Citations
17
Code
41 stars
Read summary Original
14

arXiv.org

Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-Wise Precision Boost

The paper introduces Kitty, an algorithm-system co-design for 2-bit KV cache quantization that maintains accuracy close to FP16 while reducing memory by nearly 8x. The authors observe that 4-bit quantization preserves accuracy, but 2-bit degrades it significantly. They propose Dynamic Channel-wise Precision Boost, which identifies critical key-cache…

Haojun Xia, Xiaoxia Wu, Jisen Li, Robert Wu, et al.
Published
Nov 2025
Citations
7
Code
Not linked
Read summary Original
15

arXiv.org

Beat the Long Tail: Distribution-Aware Speculative Decoding for RL Training

Reinforcement learning (RL) post-training for large language models is bottlenecked by the rollout phase, which accounts for over 70% of training time. The authors identify a long-tail distribution of rollout lengths, where a few long generations dominate wall-clock time, and note that historical rollouts reveal stable prompt-level patterns across epochs.…

Zelei Shao, Vikranth Srivatsa, Sanjana Srivastava, Qingyang Wu, et al.
Published
Nov 2025
Citations
9
Code
Not linked
Read summary Original
16

arXiv.org

ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels

ParallelKittens (PK) is a minimal CUDA framework that simplifies the development of overlapped multi-GPU kernels by distilling three key principles: transfer mechanisms, scheduling strategies, and design overheads. It extends ThunderKittens with eight core primitives and a unified programming template, enabling efficient compute-communication overlap. PK…

Stuart H. Sul, Simran Arora, B. Spector, Christopher R'e
Published
Nov 2025
Citations
9
Code
Not linked
Read summary Original
17

arXiv.org

Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining

The paper introduces Opportunistic Expert Activation (OEA), a batch-aware routing framework that reduces Mixture-of-Experts (MoE) decode latency without retraining. During autoregressive generation, MoE models become memory-bound at moderate batch sizes because latency scales with the number of unique activated experts. OEA operates in two phases: first,…

Costin-Andrei Oncescu, Qingyang Wu, Wai Tong Chung, Robert Wu, et al.
Published
Nov 2025
Citations
4
Code
Not linked
Read summary Original
18

arXiv.org

When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework

This paper introduces a theoretical framework to analyze when divide-and-conquer (D&C) strategies, which split long inputs into chunks processed by multiple agents, are effective for long-context LLMs. The framework decomposes system fidelity loss into three components: task noise (cross-chunk dependencies), model noise (performance degradation with input…

Zhen Xu, Shang Zhu, Jue Wang, Junlin Wang, et al.
Published
Jun 2025
Citations
7
Code
Not linked
Read summary Original
19

Research paper

FFT-Based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models

The paper introduces a computationally efficient method for low-rank adaptive optimization of large language models (LLMs) by replacing SVD/QR-based gradient projections with a dynamic column selection from a fixed Discrete Cosine Transform (DCT) matrix. The approach computes alignments between gradients and DCT columns, selects the top-r columns per…

Ionut-Vlad Modoranu, Mher Safaryan, Erik Schultheis, Dan Alistarh
Published
May 2025
Citations
2
Code
14 stars
Read summary Original