The Year in AI Papers
The year/Topics/Efficiency and serving

Research collection

Efficiency and serving

Making models cheaper to train or run: distillation, quantization, sparsity, mixture-of-experts, inference engines, and serving systems.

Papers
35
Research labs
4
Official code
17

1–35 of 35 papers in this collection

01

Independent research

SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

This technical report presents SLAI T-Rex, a full-stack framework for post-training the DeepSeek-V4 model family on Ascend SuperPOD. System-level optimizations (parallelism, communication, memory, kernels) increased Model FLOPs Utilization (MFU) from 11.67% to 34.22%, a 2.93x improvement. For Operations Research (OR) specialization, a solver-grounded…

Dongfang Li, Xiaodong Luo, Ruoyu Sun, Xuhui Chen, et al.
Published
Jul 2026
Citations
0
Code
32 stars
Read summary Original
02

DeepSeek

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

DSpark is a speculative decoding framework that combines a semi-autoregressive draft model with confidence-scheduled verification to accelerate LLM inference. The draft model uses a parallel backbone (based on DFlash) to generate tokens in one pass, followed by a lightweight sequential head (Markov or RNN) to model inter-token dependencies, mitigating…

Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, et al.
Published
Jul 2026
Citations
12
Code
6.9K stars
Read summary Original
03

Independent research

Program-as-Weights: A Programming Paradigm for Fuzzy Functions

The paper introduces Program-as-Weights (PAW), a paradigm for implementing fuzzy functions (tasks like log filtering or intent classification) by compiling natural-language specifications into compact neural programs. A 4B compiler, trained on the new FuzzyBench dataset (10M examples), emits a hybrid program: a discrete pseudo-program (paraphrase plus…

Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, et al.
Published
Jul 2026
Citations
0
Code
Not linked
Read summary Original
04

Independent research

BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decoding

BlockPilot introduces a sample-adaptive policy for diffusion-based speculative decoding, addressing the inefficiency of fixed block sizes. The authors observe that the optimal block size varies across samples but concentrates near the training block size, enabling a lightweight classification approach. BlockPilot predicts the block size from the last…

Hao Zhang, Yiming Hu, Yong Wang, Mingqiao Mo, et al.
Published
Jun 2026
Citations
0
Code
Not linked
Read summary Original
05

Independent research

Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

Code2LoRA is a hypernetwork framework that generates repository-specific LoRA adapters for code language models, injecting repository knowledge with zero inference-time token overhead. It has two usage scenarios: Code2LoRA-Static maps a single repository snapshot to an adapter for stable codebases, and Code2LoRA-Evo maintains an adapter via a GRU hidden…

Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie
Published
Jun 2026
Citations
1
Code
Not linked
Read summary Original
06

Independent research

On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters

This paper argues that parameter-efficient fine-tuning (PEFT), especially LoRA, can scale from a single shared foundation model to millions of persistent personal model instances. The authors propose a three-axis framework: Scale Up (strengthening the shared base model), Scale Down (shrinking the local adaptive state), and Scale Out (sustaining a large…

Mind Lab, :, Vin Bo, Song Cao, et al.
Published
Jun 2026
Citations
2
Code
Not linked
Read summary Original
07

arXiv.org

Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

Domino is a speculative decoding framework that decouples causal dependency modeling from expensive autoregressive draft execution. It uses a parallel draft backbone (based on DFlash) to generate preliminary draft distributions for the entire block, then applies a lightweight Domino head—comprising a GRU causal encoder and a low-rank correction head—to…

Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, et al.
Published
May 2026
Citations
8
Code
128 stars
Read summary Original
08

arXiv.org

MinT: Managed Infrastructure for Training and Serving Millions of LLMs

MinT is a managed infrastructure system for Low-Rank Adaptation (LoRA) post-training and online serving, designed for settings where many trained policies are produced over a small number of expensive base-model deployments. Instead of materializing full checkpoints, MinT keeps the base model resident and moves exported LoRA adapter revisions through…

Mind Lab, :, Song Cao, Vic Cao, et al.
Published
May 2026
Citations
1
Code
Not linked
Read summary Original
09

arXiv.org

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

This paper systematically investigates on-policy distillation (OPD) for large language models, identifying conditions for success, the token-level mechanism, and practical remedies. The authors find that OPD succeeds only when the student and teacher share compatible thinking patterns and the teacher offers genuinely new knowledge beyond the student's…

Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, et al.
Published
Apr 2026
Citations
145
Code
892 stars
Read summary Original
10

arXiv.org

TAPS: Task Aware Proposal Distributions for Speculative Sampling

This paper investigates how the training distribution of draft models affects speculative decoding performance, using HASS and EAGLE-2 drafters trained on MathInstruct, ShareGPT, and mixed data, evaluated on MT-Bench, GSM8K, MATH-500, and SVAMP. Results show task-specific training yields clear specialization: MathInstruct-trained drafts excel on reasoning…

Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna, Hasan Abed Al Kader Hammoud, et al.
Published
Mar 2026
Citations
0
Code
8 stars
Read summary Original
11

arXiv.org

SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

SpecEyes is an agentic-level speculative acceleration framework for multimodal large language models (MLLMs) that addresses the sequential bottleneck of tool-use loops. It uses a lightweight, tool-free model to speculatively answer queries that do not require deep agentic reasoning, bypassing the expensive tool-calling pipeline. The framework consists of…

Haoyu Huang, Jinfa Huang, Zhongwei Wan, Xiawu Zheng, et al.
Published
Mar 2026
Citations
1
Code
63 stars
Read summary Original
12

arXiv.org

Flash-KMeans: Fast and Memory-Efficient Exact K-Means

Flash-KMeans is a GPU implementation of exact k-means that addresses performance bottlenecks in modern AI workloads. The paper identifies two main kernel-level issues: the assignment stage suffers from an IO bottleneck due to materializing the N×K distance matrix in HBM, and the centroid update stage suffers from atomic write contention due to…

Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, et al.
Published
Mar 2026
Citations
5
Code
704 stars
Read summary Original
13

Together AI

FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

FlashAttention-4 addresses the asymmetric hardware scaling of NVIDIA Blackwell GPUs (B200/GB200), where tensor core throughput doubles compared to Hopper, but shared memory bandwidth and exponential unit throughput do not scale accordingly. This shifts bottlenecks to non-matmul operations. The paper introduces techniques to mitigate these bottlenecks: (1)…

Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timothy Liu, et al.
Published
Mar 2026
Citations
36
Code
Not linked
Read summary Original
14

Together AI

Speculative Speculative Decoding

The paper introduces speculative speculative decoding (SSD), a framework that parallelizes drafting and verification in speculative decoding (SD) by having the draft model predict likely verification outcomes and pre-speculate for them while verification runs. This eliminates drafting overhead on cache hits, making SSD lossless and faster than SD. The…

Tanishq Kumar, Tri Dao, Avner May
Published
Mar 2026
Citations
11
Code
Not linked
Read summary Original
15

DeepSeek

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

DualPath is an inference system that addresses the storage I/O bottleneck in multi-turn, agentic LLM inference under prefill-decode (PD) disaggregation. In such workloads, KV-Cache hit rates are high (≥95%), making KV-Cache loading from external storage the dominant performance factor. Existing systems load KV-Cache only into prefill engines, saturating…

Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, et al.
Published
Feb 2026
Citations
10
Code
Not linked
Read summary Original
16

Together AI

Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

The paper introduces UPipe, a context parallelism technique that reduces activation memory in Transformer training by chunking attention heads. It builds on DeepSpeed-Ulysses, processing attention in stages with a subset of heads (U) at a time, reusing buffers to cut intermediate memory from O(H) to O(U). For Qwen3-32B, this reduces attention intermediate…

Ravi Ghadia, Maksim Abraham, S. Vorobyov, Max Ryabinin
Published
Feb 2026
Citations
0
Code
25 stars
Read summary Original
17

Together AI

ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

ThunderAgent is a program-aware agentic inference system that improves throughput for multi-turn LLM agent workflows by abstracting them as agentic programs with metadata (ID, context length, tool environments, placement, phase, status). It introduces a program-aware scheduler with state-aware pausing and dynamic migration across GPU nodes to reduce KV…

Hao Kang, Ziyang Li, Xinyu Yang, Weili Xu, et al.
Published
Feb 2026
Citations
13
Code
Not linked
Read summary Original
18

arXiv.org

SLA2: Sparse-Linear Attention with Learnable Routing and QAT

SLA2 is a sparse-linear attention method for diffusion models that addresses two limitations of the prior SLA approach: a heuristic split between sparse and linear branches, and a mismatch between SLA's output and the intended sparse-linear decomposition. SLA2 introduces a learnable router that dynamically selects which attention computations use sparse or…

Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, et al.
Published
Feb 2026
Citations
15
Code
328 stars
Read summary Original
19

Together AI

When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

Aurora is a unified training-serving system that addresses limitations of conventional speculative decoding, which separates offline speculator training from online serving, causing deployment lag, delayed utility feedback, and domain-drift degradation. Aurora closes the loop by continuously learning a speculator from live inference traces, framing it as…

Junxiong Wang, Fengxiang Bie, Jisen Li, Zhongzhu Zhou, et al.
Published
Feb 2026
Citations
1
Code
Not linked
Read summary Original
20

arXiv.org

DFlash: Block Diffusion for Flash Speculative Decoding

DFlash is a speculative decoding framework that uses a lightweight block diffusion model for parallel drafting, addressing the sequential bottleneck of autoregressive LLM decoding. It extracts hidden features from the target LLM and injects them into every draft layer's KV cache, conditioning the draft model to predict future token blocks in parallel. This…

Jian Chen, Yesheng Liang, Zhijian Liu
Published
Feb 2026
Citations
63
Code
5.6K stars
Read summary Original
21

arXiv.org

Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

The paper introduces Quartet II, a fully-NVFP4 quantization scheme for LLM pre-training on NVIDIA Blackwell GPUs. It proposes MS-EDEN, a new unbiased quantization routine for microscaling formats that reduces quantization error by more than 2x compared to stochastic rounding (SR), by applying randomized Hadamard rotations and merging EDEN correction…

Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan Alistarh
Published
Jan 2026
Citations
11
Code
80 stars
Read summary Original
22

Mistral AI

Ministral 3

The Ministral 3 series introduces three parameter-efficient dense language models (3B, 8B, and 14B) derived from the Mistral Small 3.1 24B parent model via Cascade Distillation, an iterative pruning and distillation technique. Each size has base, instruct, and reasoning variants, all with vision capabilities and Apache 2.0 licensing. The models support up…

Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, et al.
Published
Jan 2026
Citations
53
Code
Not linked
Read summary Original
23

Together AI

Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-Wise Precision Boost

The paper introduces Kitty, an algorithm-system co-design for 2-bit KV cache quantization that maintains accuracy close to FP16 while reducing memory by nearly 8x. The authors observe that 4-bit quantization preserves accuracy, but 2-bit degrades it significantly. They propose Dynamic Channel-wise Precision Boost, which identifies critical key-cache…

Haojun Xia, Xiaoxia Wu, Jisen Li, Robert Wu, et al.
Published
Nov 2025
Citations
7
Code
Not linked
Read summary Original
24

Meta AI

Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance

The paper introduces Soup Of Category Experts (SoCE), a model souping technique that improves LLM performance by leveraging benchmark composition and non-uniform weighted averaging. SoCE identifies weakly-correlated benchmark categories, selects expert models for each, and optimizes weights to combine them. Experiments on BFCL, MGSM, and ∞-Bench show SoCE…

Shalini Maiti, Amar Budhiraja, Bhavul Gauri, Gaurav Chaurasia, et al.
Published
Nov 2025
Citations
4
Code
73 stars
Read summary Original
25

Together AI

Beat the Long Tail: Distribution-Aware Speculative Decoding for RL Training

Reinforcement learning (RL) post-training for large language models is bottlenecked by the rollout phase, which accounts for over 70% of training time. The authors identify a long-tail distribution of rollout lengths, where a few long generations dominate wall-clock time, and note that historical rollouts reveal stable prompt-level patterns across epochs.…

Zelei Shao, Vikranth Srivatsa, Sanjana Srivastava, Qingyang Wu, et al.
Published
Nov 2025
Citations
9
Code
Not linked
Read summary Original
26

Together AI

ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels

ParallelKittens (PK) is a minimal CUDA framework that simplifies the development of overlapped multi-GPU kernels by distilling three key principles: transfer mechanisms, scheduling strategies, and design overheads. It extends ThunderKittens with eight core primitives and a unified programming template, enabling efficient compute-communication overlap. PK…

Stuart H. Sul, Simran Arora, B. Spector, Christopher R'e
Published
Nov 2025
Citations
9
Code
Not linked
Read summary Original
27

arXiv.org

Black-Box On-Policy Distillation of Large Language Models

The paper introduces Generative Adversarial Distillation (GAD), a method for black-box distillation of large language models (LLMs) where only the teacher's text outputs are accessible. GAD frames the student as a generator and trains a discriminator to distinguish student from teacher responses, forming a minimax game. The discriminator acts as an…

Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, et al.
Published
Nov 2025
Citations
43
Code
Not linked
Read summary Original
28

Together AI

Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining

The paper introduces Opportunistic Expert Activation (OEA), a batch-aware routing framework that reduces Mixture-of-Experts (MoE) decode latency without retraining. During autoregressive generation, MoE models become memory-bound at moderate batch sizes because latency scales with the number of unique activated experts. OEA operates in two phases: first,…

Costin-Andrei Oncescu, Qingyang Wu, Wai Tong Chung, Robert Wu, et al.
Published
Nov 2025
Citations
4
Code
Not linked
Read summary Original
29

arXiv.org

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

This paper systematically compares integer (INT) and floating-point (FP) low-bit quantization formats for LLMs across different granularities. The authors introduce a theoretical framework based on quantization signal-to-noise ratio (QSNR) and crest factor, revealing a performance crossover: FP excels at coarse granularities, but at fine-grained block-wise…

Mengzhao Chen, Meng Wu, Hui Jin, Zhihang Yuan, et al.
Published
Oct 2025
Citations
18
Code
81 stars
Read summary Original
30

Neural Information Processing Systems

AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders

AdaSPEC is a novel knowledge distillation method for speculative decoding (SD) that improves draft model alignment with the target model by selectively filtering tokens during training. Unlike conventional KD that minimizes KL divergence across all tokens, AdaSPEC identifies and excludes 'hard' tokens that are difficult for the small draft model to learn,…

Yuezhou Hu, Jiaxin Guo, Xinyu Feng, Tuo Zhao
Published
Oct 2025
Citations
7
Code
39 stars
Read summary Original
31

arXiv.org

BitNet Distillation

BitDistill is a lightweight pipeline for fine-tuning full-precision LLMs (e.g., Qwen3) into 1.58-bit ternary precision for specific downstream tasks, achieving performance comparable to full-precision models while reducing memory by 10× and speeding up CPU inference by 2.65×. The method comprises three stages: (1) modeling refinement with SubLN modules to…

Xun Wu, Shaohan Huang, Wenhui Wang, Ting Song, et al.
Published
Oct 2025
Citations
2
Code
Not linked
Read summary Original
32

arXiv.org

SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention

The paper introduces SLA (Sparse-Linear Attention), a trainable attention method for Diffusion Transformers (DiTs) that combines sparse and linear attention to reduce computational cost. The authors observe that attention weights can be decomposed into a small fraction of large weights with high rank and a large fraction of low-rank weights. SLA classifies…

Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, et al.
Published
Sep 2025
Citations
44
Code
328 stars
Read summary Original
33

arXiv.org

SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights

SINQ is a calibration-free post-training quantization method for large language models that introduces a dual-scale parameterization (row and column scale vectors) for weight matrices. It uses a Sinkhorn-Knopp-style algorithm to iteratively normalize row and column standard deviations, balancing activation-aware column scaling with row-wise kurtosis…

Lorenz K. Müller, Philippe Bich, Jiawei Zhuang, Ahmet Çelik, et al.
Published
Sep 2025
Citations
5
Code
627 stars
Read summary Original
34

arXiv.org

Set Block Decoding is a Language Model Inference Accelerator

Set Block Decoding (SBD) is a new inference acceleration paradigm for autoregressive language models that integrates next token prediction (NTP) with masked token prediction (MATP) in a single architecture. SBD allows sampling multiple, non-consecutive future tokens in parallel, using solvers from discrete diffusion literature like the Entropy Bounded (EB)…

Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haziza, et al.
Published
Sep 2025
Citations
18
Code
Not linked
Read summary Original
35

Together AI

FFT-Based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models

The paper introduces a computationally efficient method for low-rank adaptive optimization of large language models (LLMs) by replacing SVD/QR-based gradient projections with a dynamic column selection from a fixed Discrete Cosine Transform (DCT) matrix. The approach computes alignments between gradients and DCT columns, selects the top-r columns per…

Ionut-Vlad Modoranu, Mher Safaryan, Erik Schultheis, Dan Alistarh
Published
May 2025
Citations
2
Code
14 stars
Read summary Original