The Year in AI Papers
The year/Topics/Systems and efficiency

Topic area

Systems and efficiency

Every collection across systems and efficiency.

Papers
143
Research labs
10
Official code
80

101–143 of 143 papers in this topic area

101

Together AI

Beat the Long Tail: Distribution-Aware Speculative Decoding for RL Training

Reinforcement learning (RL) post-training for large language models is bottlenecked by the rollout phase, which accounts for over 70% of training time. The authors identify a long-tail distribution of rollout lengths, where a few long generations dominate wall-clock time, and note that historical rollouts reveal stable prompt-level patterns across epochs.…

Zelei Shao, Vikranth Srivatsa, Sanjana Srivastava, Qingyang Wu, et al.
Published
Nov 2025
Citations
9
Code
Not linked
Read summary Original
102

Together AI

ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels

ParallelKittens (PK) is a minimal CUDA framework that simplifies the development of overlapped multi-GPU kernels by distilling three key principles: transfer mechanisms, scheduling strategies, and design overheads. It extends ThunderKittens with eight core primitives and a unified programming template, enabling efficient compute-communication overlap. PK…

Stuart H. Sul, Simran Arora, B. Spector, Christopher R'e
Published
Nov 2025
Citations
9
Code
Not linked
Read summary Original
103

arXiv.org

Black-Box On-Policy Distillation of Large Language Models

The paper introduces Generative Adversarial Distillation (GAD), a method for black-box distillation of large language models (LLMs) where only the teacher's text outputs are accessible. GAD frames the student as a generator and trains a discriminator to distinguish student from teacher responses, forming a minimax game. The discriminator acts as an…

Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, et al.
Published
Nov 2025
Citations
43
Code
Not linked
Read summary Original
104

arXiv.org

MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generation

The paper introduces MMaDA-Parallel, a parallel multimodal diffusion framework for thinking-aware image editing and generation. The authors identify a failure mode in sequential autoregressive approaches where pre-generation reasoning can degrade performance due to error propagation. They propose ParaBench, a benchmark evaluating both text and image…

Ye Tian, Ling Yang, Jiongfan Yang, Anran Wang, et al.
Published
Nov 2025
Citations
8
Code
306 stars
Read summary Original
105

arXiv.org

DoPE: Denoising Rotary Position Embedding

The paper introduces DOPE (Denoising Rotary Position Embedding), a training-free method to improve length extrapolation in LLMs by mitigating instabilities caused by RoPE. The authors show via spectral analysis that RoPE's low-frequency components cause spectral amplification, leading to low-rank, over-aligned attention patterns and attention sinks. DOPE…

Jing Xiong, Liyang Fan, Hui Shen, Zunhai Su, et al.
Published
Nov 2025
Citations
2
Code
Not linked
Read summary Original
106

NVIDIA

TiDAR: Think in Diffusion, Talk in Autoregression

TiDAR is a sequence-level hybrid architecture that combines diffusion and autoregressive (AR) language modeling in a single forward pass using structured attention masks. It drafts tokens in parallel via diffusion (thinking) and samples final outputs autoregressively (talking), exploiting free GPU compute slots to achieve high throughput without quality…

Jingyu Liu, Xin Dong, Zhifan Ye, Rishabh Mehta, et al.
Published
Nov 2025
Citations
33
Code
Not linked
Read summary Original
107

arXiv.org

Diffusion Language Models are Super Data Learners

The paper demonstrates that under data-constrained pre-training, diffusion language models (DLMs) consistently surpass autoregressive (AR) models of equal size, a phenomenon termed the 'Intelligence Crossover.' This crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. The…

Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, et al.
Published
Nov 2025
Citations
50
Code
228 stars
Read summary Original
108

Together AI

Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining

The paper introduces Opportunistic Expert Activation (OEA), a batch-aware routing framework that reduces Mixture-of-Experts (MoE) decode latency without retraining. During autoregressive generation, MoE models become memory-bound at moderate batch sizes because latency scales with the number of unique activated experts. OEA operates in two phases: first,…

Costin-Andrei Oncescu, Qingyang Wu, Wai Tong Chung, Robert Wu, et al.
Published
Nov 2025
Citations
4
Code
Not linked
Read summary Original
109

arXiv.org

Continuous Autoregressive Language Models

This paper introduces Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress K tokens into a single continuous vector, reducing the number of autoregressive steps by a factor of K. The authors develop a likelihood-free…

Chenze Shao, Darren Li, Fandong Meng, Jie Zhou
Published
Oct 2025
Citations
10
Code
814 stars
Read summary Original
110

Moonshot AI

Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Linear is a hybrid linear attention architecture that outperforms full attention under fair comparisons across short-context, long-context, and reinforcement learning (RL) scaling regimes. Its core, Kimi Delta Attention (KDA), extends Gated DeltaNet with a finer-grained, channel-wise gating mechanism, improving memory control and enabling efficient…

Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, et al.
Published
Oct 2025
Citations
116
Code
1.6K stars
Read summary Original
111

arXiv.org

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

This paper systematically compares integer (INT) and floating-point (FP) low-bit quantization formats for LLMs across different granularities. The authors introduce a theoretical framework based on quantization signal-to-noise ratio (QSNR) and crest factor, revealing a performance crossover: FP excels at coarse granularities, but at fine-grained block-wise…

Mengzhao Chen, Meng Wu, Hui Jin, Zhihang Yuan, et al.
Published
Oct 2025
Citations
18
Code
81 stars
Read summary Original
112

Independent research

Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation

Ling 2.0 is a series of reasoning-oriented language foundation models from Inclusion AI, scaling from 16B to 1T parameters under a unified high-sparsity Mixture-of-Experts (MoE) paradigm. The series includes three non-thinking instruct models: Ling-mini-2.0 (16B total, 1.4B activated), Ling-flash-2.0 (103B total, 6.1B activated), and Ling-1T (1T total, 51B…

Ling Team, Ang Li, Ben Liu, Binbin Hu, et al.
Published
Oct 2025
Citations
30
Code
Not linked
Read summary Original
113

arXiv.org

The Principles of Diffusion Models

This book provides a comprehensive, mathematically grounded introduction to diffusion models, tracing their origins from three foundational perspectives: variational (VAEs to DDPMs), score-based (EBMs to NCSN and Score SDEs), and flow-based (Normalizing Flows to Flow Matching). It unifies these views through the Fokker-Planck equation and a common…

Chieh-Hsin Lai, Yang Song, Dongjun Kim, Yuki Mitsufuji, et al.
Published
Oct 2025
Citations
69
Code
Not linked
Read summary Original
114

Neural Information Processing Systems

AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders

AdaSPEC is a novel knowledge distillation method for speculative decoding (SD) that improves draft model alignment with the target model by selectively filtering tokens during training. Unlike conventional KD that minimizes KL divergence across all tokens, AdaSPEC identifies and excludes 'hard' tokens that are difficult for the small draft model to learn,…

Yuezhou Hu, Jiaxin Guo, Xinyu Feng, Tuo Zhao
Published
Oct 2025
Citations
7
Code
39 stars
Read summary Original
115

arXiv.org

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

This technical report introduces the Ring-linear model series, comprising Ring-mini-linear-2.0 (16B total, 1.6B active parameters) and Ring-flash-linear-2.0 (104B total, 7.4B active). Both use a hybrid architecture that combines linear and softmax attention to reduce I/O and computational costs in long-context scenarios, cutting inference cost to 1/10 of a…

Ling Team, Bin Han, Caizhi Tang, Chen Liang, et al.
Published
Oct 2025
Citations
9
Code
Not linked
Read summary Original
116

arXiv.org

LightMem: Lightweight and Efficient Memory-Augmented Generation

LightMem is a lightweight and efficient memory-augmented generation system for Large Language Models (LLMs), presented at ICLR 2026. It addresses inefficiencies in existing memory systems by drawing inspiration from the Atkinson-Shiffrin human memory model, organizing memory into three stages: sensory memory (pre-compression and topic segmentation),…

Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, et al.
Published
Oct 2025
Citations
111
Code
1.1K stars
Read summary Original
117

arXiv.org

Efficient Long-context Language Model Training by Core Attention Disaggregation

This paper introduces core attention disaggregation (CAD), a technique to improve long-context LLM training by separating the parameter-free softmax(QK^T)V computation (core attention, CA) from other model components and scheduling it on a dedicated pool of resources. CAD leverages two key properties: statelessness (CA has no trainable parameters) and…

Yonghao Zhuang, Junda Chen, Bo Pang, Yi Gu, et al.
Published
Oct 2025
Citations
3
Code
Not linked
Read summary Original
118

Volume 1

Glyph: Scaling Context Windows via Visual-Text Compression

Glyph is a framework that scales context windows by rendering long texts into images and processing them with vision-language models (VLMs), achieving 3-4x token compression while maintaining accuracy comparable to leading LLMs like Qwen3-8B. It consists of three stages: continual pre-training on rendered long-text data, an LLM-driven genetic search to…

Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, et al.
Published
Oct 2025
Citations
42
Code
595 stars
Read summary Original
119

arXiv.org

FineVision: Open Data Is All You Need

FineVision is a large-scale, open corpus of 24 million samples (17 million images, 89 million turns, 9.5 billion answer tokens) for training vision-language models. It unifies over 200 public sources into 185 subsets via a semi-automated, human-in-the-loop pipeline that includes schema mapping, cleaning, de-duplication, and decontamination against 66…

Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, et al.
Published
Oct 2025
Citations
38
Code
Not linked
Read summary Original
120

arXiv.org

BitNet Distillation

BitDistill is a lightweight pipeline for fine-tuning full-precision LLMs (e.g., Qwen3) into 1.58-bit ternary precision for specific downstream tasks, achieving performance comparable to full-precision models while reducing memory by 10× and speeding up CPU inference by 2.65×. The method comprises three stages: (1) modeling refinement with SubLN modules to…

Xun Wu, Shaohan Huang, Wenhui Wang, Ting Song, et al.
Published
Oct 2025
Citations
2
Code
Not linked
Read summary Original
121

arXiv.org

Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs

The paper introduces Honey-Data-15M, a 15-million-sample SFT dataset for fully open multimodal large language models (MLLMs), built to address data noise and a lack of complex reasoning data. It also presents HoneyPipe, a transparent data curation pipeline built on the DataStudio framework, which uses rule-based and model-based filtering, dual-level…

Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, et al.
Published
Oct 2025
Citations
25
Code
Not linked
Read summary Original
122

arXiv.org

UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE

UniMoE-Audio is a unified speech and music generation model that addresses task conflict and data imbalance through a Dynamic-Capacity Mixture-of-Experts (MoE) framework. The architecture introduces a Top-P routing strategy for dynamic expert allocation and a hybrid expert design with routed, shared, and null experts. To handle data imbalance, a…

Zhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng, et al.
Published
Oct 2025
Citations
6
Code
Not linked
Read summary Original
123

arXiv.org

Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States

The paper introduces Latent Refinement Decoding (LRD), a two-stage decoding framework for diffusion-based language models (dLLMs) that addresses information loss and premature commitment in existing hard-assignment methods. LRD operates in two phases: Phase 1 (Latent Refinement) performs distribution-preserving updates in embedding space by mixing [MASK]…

Qinglin Zhu, Yizhen Yao, Runcong Zhao, Yanzheng Xiang, et al.
Published
Oct 2025
Citations
9
Code
Not linked
Read summary Original
124

arXiv.org

KORMo: Korean Open Reasoning Model for Everyone

This paper introduces KORMo-10B, a 10.8B-parameter fully open bilingual Korean-English language model trained from scratch on a corpus where 68.74% of the Korean portion is synthetic. The authors systematically investigate the feasibility of using synthetic data for non-English fully open models (FOMs), addressing stability, tokenizer design, and bias.…

Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, et al.
Published
Oct 2025
Citations
4
Code
119 stars
Read summary Original
125

arXiv.org

Cache-to-Cache: Direct Semantic Communication Between Large Language Models

This paper introduces Cache-to-Cache (C2C), a new paradigm for direct semantic communication between large language models (LLMs) that bypasses text-based interaction. C2C projects and fuses the KV-cache of a source model (Sharer) into that of a target model (Receiver) using a neural cache fuser with a learnable gating mechanism. Oracle experiments showed…

Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, et al.
Published
Oct 2025
Citations
35
Code
425 stars
Read summary Original
126

International Conference on Automated Software Engineering

LongCodeZip: Compress Long Context for Code Language Models

LongCodeZip is a training-free, model-agnostic, plug-and-play framework for compressing long code contexts for code LLMs. It uses a dual-stage strategy: coarse-grained compression selects function-level chunks ranked by conditional perplexity relative to the instruction, and fine-grained compression segments retained functions into blocks via…

Yuling Shi, Yichun Qian, Hongyu Zhang, Beijun Shen, et al.
Published
Oct 2025
Citations
39
Code
163 stars
Read summary Original
127

arXiv.org

The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain

The paper introduces Dragon Hatchling (BDH), a new Large Language Model architecture based on a scale-free, biologically inspired network of n locally-interacting neuron particles. BDH couples strong theoretical foundations and inherent interpretability without sacrificing Transformer-like performance. It is a practical, performant attention-based state…

Adrian Kosowski, Przemysław Uznański, Jan Chorowski, Zuzanna Stamirowska, et al.
Published
Sep 2025
Citations
5
Code
3.4K stars
Read summary Original
128

NVIDIA

Fast-dLLM v2: Efficient Block-Diffusion LLM

Fast-dLLM v2 is a block diffusion language model that adapts pretrained autoregressive LLMs (Qwen2.5-Instruct 1.5B and 7B) for parallel text generation. It requires only ~1B tokens of fine-tuning, a 500x reduction compared to full-attention diffusion models like Dream (580B tokens). The method uses block-wise diffusion with complementary masking and a…

Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, et al.
Published
Sep 2025
Citations
114
Code
1.1K stars
Read summary Original
129

arXiv.org

SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention

The paper introduces SLA (Sparse-Linear Attention), a trainable attention method for Diffusion Transformers (DiTs) that combines sparse and linear attention to reduce computational cost. The authors observe that attention weights can be decomposed into a small fraction of large weights with high rank and a large fraction of low-rank weights. SLA classifies…

Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, et al.
Published
Sep 2025
Citations
44
Code
328 stars
Read summary Original
130

arXiv.org

MemMamba: Rethinking Memory Patterns in State Space Model

MemMamba is a novel architecture for long-sequence modeling that addresses the memory decay problem in state space models (SSMs) like Mamba. The paper first analyzes Mamba's memory mechanism, showing that its state update leads to exponential decay of early information both within layers (horizontal) and across layers (vertical), quantified by new metrics:…

Youjin Wang, Yangjingyi Chen, Jiahao Yan, Jiaxuan Lu, et al.
Published
Sep 2025
Citations
2
Code
Not linked
Read summary Original
131

arXiv.org

Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning

The paper introduces Q-Tuning, a unified framework for joint sample- and token-level pruning during supervised fine-tuning (SFT) of large language models. It addresses the inefficiency of existing methods that treat sample and token pruning separately. The authors propose the Error–Uncertainty (EU) Plane, which categorizes training samples into four…

Shaobo Wang, Jiaming Wang, Jiajun Zhang, Cong Wang, et al.
Published
Sep 2025
Citations
9
Code
3 stars
Read summary Original
132

arXiv.org

SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights

SINQ is a calibration-free post-training quantization method for large language models that introduces a dual-scale parameterization (row and column scale vectors) for weight matrices. It uses a Sinkhorn-Knopp-style algorithm to iteratively normalize row and column standard deviations, balancing activation-aware column scaling with row-wise kurtosis…

Lorenz K. Müller, Philippe Bich, Jiawei Zhuang, Ahmet Çelik, et al.
Published
Sep 2025
Citations
5
Code
627 stars
Read summary Original
133

arXiv.org

Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale

The HALA technical report introduces a family of Arabic-centric instruction and translation models built with a translate-and-tune pipeline. The pipeline compresses a strong AR↔EN teacher model (CohereLabs/command-a-translate-08-2025) to FP8, achieving ~2x higher throughput with no quality loss. This teacher translates 405K Open-Orca pairs into Arabic, and…

Hasan Abed Al Kader Hammoud, Mohammad Zbeeb, Bernard Ghanem
Published
Sep 2025
Citations
5
Code
5 stars
Read summary Original
134

arXiv.org

MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining

The paper introduces MACHINELEARNINGLM, a continued-pretraining framework that enables large language models (LLMs) to perform many-shot in-context learning (ICL) on tabular machine-learning tasks without gradient updates. The method synthesizes millions of tasks from structural causal models (SCMs), uses a random-forest teacher for a warm-start…

Haoyu Dong, Pengkun Zhang, Mingzhe Lu, Yanzhen Shen, et al.
Published
Sep 2025
Citations
4
Code
59 stars
Read summary Original
135

arXiv.org

Set Block Decoding is a Language Model Inference Accelerator

Set Block Decoding (SBD) is a new inference acceleration paradigm for autoregressive language models that integrates next token prediction (NTP) with masked token prediction (MATP) in a single architecture. SBD allows sampling multiple, non-consecutive future tokens in parallel, using solvers from discrete diffusion literature like the Entropy Bounded (EB)…

Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haziza, et al.
Published
Sep 2025
Citations
18
Code
Not linked
Read summary Original
136

arXiv.org

Hermes 4 Technical Report

Hermes 4 is a family of hybrid reasoning models (14B, 70B, 405B) that combine structured, multi-turn reasoning with broad instruction-following. The models were trained on a dataset of ~5 million samples (19B tokens), including 3.5M reasoning and 1.6M non-reasoning samples, synthesized via DataForge, a graph-based generator, and rejection sampling with…

Ryan Teknium, Roger Jin, Jai Suphavadeeprasit, Dakota Mahan, et al.
Published
Aug 2025
Citations
3
Code
Not linked
Read summary Original
137

arXiv.org

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

BeyondWeb is a synthetic data generation framework for LLM pretraining that outperforms existing synthetic datasets like Cosmopedia and Nemotron-Synth by up to 5.1 and 2.6 percentage points, respectively, across 14 benchmarks. It achieves up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. A 3B model trained on BeyondWeb for…

DatologyAI, :, Pratyush Maini, Vineeth Dorna, et al.
Published
Aug 2025
Citations
16
Code
Not linked
Read summary Original
138

arXiv.org

PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts

PRELUDE is a benchmark for evaluating long-context understanding in LLMs by asking whether a character's prequel story is consistent with the canonical narrative of a book. The task requires global comprehension and deep reasoning, as 88% of instances need evidence from multiple parts of the narrative. The dataset includes 795 instances across 13 books,…

Mo Yu, Tsz Ting Chung, Chulun Zhou, Tong Li, et al.
Published
Aug 2025
Citations
6
Code
Not linked
Read summary Original
139

arXiv.org

Speed Always Wins: A Survey on Efficient Architectures for Large Language Models

This survey systematically reviews efficient architectures for Large Language Models (LLMs), addressing the computational bottlenecks of the standard Transformer, particularly the quadratic complexity of self-attention and the high costs of large Feed-Forward Networks. It categorizes recent innovations into seven main areas: linear sequence modeling (e.g.,…

Weigao Sun, Jiaxi Hu, Yucheng Zhou, Jusen Du, et al.
Published
Aug 2025
Citations
17
Code
406 stars
Read summary Original
140

Z.ai / GLM

GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

GLM-4.5 is an open-source Mixture-of-Experts (MoE) large language model with 355B total and 32B activated parameters, developed by Zhipu AI and Tsinghua University. It features a hybrid reasoning method supporting both thinking and direct response modes. Trained on 23T tokens with multi-stage pre-training and mid-training (including repo-level code,…

GLM-4. 5 Team, :, Aohan Zeng, Xin Lv, et al.
Published
Aug 2025
Citations
409
Code
4.4K stars
Read summary Original
141

arXiv.org

Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

Seed Diffusion Preview is a large-scale discrete-state diffusion language model developed by ByteDance Seed and Tsinghua AIR, focused on code generation. It achieves an inference speed of 2,146 tokens per second on H20 GPUs, significantly faster than contemporary models like Mercury Coder and Gemini Diffusion, while maintaining competitive performance on…

Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, et al.
Published
Aug 2025
Citations
163
Code
Not linked
Read summary Original
142

Together AI

When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework

This paper introduces a theoretical framework to analyze when divide-and-conquer (D&C) strategies, which split long inputs into chunks processed by multiple agents, are effective for long-context LLMs. The framework decomposes system fidelity loss into three components: task noise (cross-chunk dependencies), model noise (performance degradation with input…

Zhen Xu, Shang Zhu, Jue Wang, Junlin Wang, et al.
Published
Jun 2025
Citations
7
Code
Not linked
Read summary Original
143

Together AI

FFT-Based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models

The paper introduces a computationally efficient method for low-rank adaptive optimization of large language models (LLMs) by replacing SVD/QR-based gradient projections with a dynamic column selection from a fixed Discrete Cosine Transform (DCT) matrix. The approach computes alignments between gradients and DCT columns, selects the top-r columns per…

Ionut-Vlad Modoranu, Mher Safaryan, Erik Schultheis, Dan Alistarh
Published
May 2025
Citations
2
Code
14 stars
Read summary Original