Topic area
Every collection across systems and efficiency.
Moonshot AI
The paper proposes Attention Residuals (AttnRes), replacing fixed unit-weight residual connections in LLMs with learned, input-dependent softmax attention over preceding layer outputs. This addresses PreNorm dilution, where hidden-state magnitudes grow with depth, diluting layer contributions. A scalable variant, Block AttnRes, partitions layers into…
Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, et al.Together AI
Mamba-3 introduces three methodological improvements to state space models (SSMs) for better quality, capability, and inference efficiency. First, exponential-trapezoidal discretization generalizes prior heuristics, enabling a more expressive recurrence that can replace the short causal convolution. Second, complex-valued state transitions, implemented via…
Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, et al.Z.ai / GLM
IndexCache is a method to accelerate sparse attention in large language models by exploiting cross-layer redundancy in token selection. DeepSeek Sparse Attention (DSA) uses a lightning indexer at each layer to select top-k tokens, but this indexer still has O(L^2) complexity. IndexCache partitions layers into a small set of Full (F) layers that run their…
Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, et al.arXiv.org
Flash-KMeans is a GPU implementation of exact k-means that addresses performance bottlenecks in modern AI workloads. The paper identifies two main kernel-level issues: the assignment stage suffers from an IO bottleneck due to materializing the N×K distance matrix in HBM, and the centroid update stage suffers from atomic write contention due to…
Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, et al.Together AI
FlashAttention-4 addresses the asymmetric hardware scaling of NVIDIA Blackwell GPUs (B200/GB200), where tensor core throughput doubles compared to Hopper, but shared memory bandwidth and exponential unit throughput do not scale accordingly. This shifts bottlenecks to non-matmul operations. The paper introduces techniques to mitigate these bottlenecks: (1)…
Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timothy Liu, et al.Together AI
The paper introduces speculative speculative decoding (SSD), a framework that parallelizes drafting and verification in speculative decoding (SD) by having the draft model predict likely verification outcomes and pre-speculate for them while verification runs. This eliminates drafting overhead on cache hits, making SSD lossless and faster than SD. The…
Tanishq Kumar, Tri Dao, Avner MayarXiv.org
The paper introduces CHIMERA, a compact synthetic dataset of 9,225 samples designed to address data-centric challenges in LLM reasoning post-training: cold-start data scarcity, limited domain coverage, and the annotation bottleneck. The dataset is built via a three-stage pipeline: subject expansion using GPT-5 to create a hierarchical taxonomy of 1,179…
Xinyu Zhu, Yihao Feng, Yanchao Sun, Xianzhi Du, et al.arXiv.org
The paper introduces Diagnostic-driven Progressive Evolution (DPE), a closed-loop training framework for Large Multimodal Models (LMMs) that addresses limitations of static data and heuristic self-evolution methods. DPE iteratively performs diagnosis, targeted data generation, and reinforcement learning. A diagnostic agent identifies capability blind spots…
Hongrui Jia, Chaoya Jiang, Yongrui Heng, Shikun Zhang, et al.Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)
The paper introduces dLLM, an open-source framework that standardizes the training, inference, and evaluation of diffusion language models (DLMs). It provides modular trainers for Masked Diffusion (MDLM) and Block Diffusion (BD3LM), a plug-and-play sampler abstraction supporting efficient decoding like Fast-dLLM, and a unified evaluation pipeline that…
Zhanhui Zhou, Lingjie Chen, Hanghang Tong, Dawn SongDeepSeek
DualPath is an inference system that addresses the storage I/O bottleneck in multi-turn, agentic LLM inference under prefill-decode (PD) disaggregation. In such workloads, KV-Cache hit rates are high (≥95%), making KV-Cache loading from external storage the dominant performance factor. Existing systems load KV-Cache only into prefill engines, saturating…
Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, et al.NVIDIA
This paper addresses the lack of transparency in training data for terminal-capable LLMs by introducing Terminal-Task-Gen, a synthetic task generation pipeline, and Terminal-Corpus, a large-scale open-source dataset. The authors fine-tune Qwen3 models (8B, 14B, 32B) to create Nemotron-Terminal, achieving substantial gains on Terminal-Bench 2.0: 8B improves…
Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, et al.Together AI
The paper introduces UPipe, a context parallelism technique that reduces activation memory in Transformer training by chunking attention heads. It builds on DeepSpeed-Ulysses, processing attention in stages with a subset of heads (U) at a time, reusing buffers to cut intermediate memory from O(H) to O(U). For Qwen3-32B, this reduces attention intermediate…
Ravi Ghadia, Maksim Abraham, S. Vorobyov, Max RyabininarXiv.org
HyTRec is a generative recommendation framework designed to efficiently model ultra-long user behavior sequences. It addresses the trade-off between linear attention's efficiency and softmax attention's precision by using a hybrid architecture: a linear attention branch (Temporal-Aware Delta Network, TADN) processes long-term history, while a softmax…
Lei Xin, Yuhao Zheng, Ke Cheng, Changjiang Jiang, et al.arXiv.org
MolHIT is a molecular graph generation framework based on a Hierarchical Discrete Diffusion Model (HDDM) that generalizes discrete diffusion by adding mid-level states encoding chemical priors, and a Decoupled Atom Encoding (DAE) that splits atom types by chemical roles (aromaticity, formal charge). This addresses limitations in prior graph diffusion…
Hojung Jung, Rodrigo Hormazabal, Jaehyeong Jo, Youngrok Park, et al.Together AI
ThunderAgent is a program-aware agentic inference system that improves throughput for multi-turn LLM agent workflows by abstracting them as agentic programs with metadata (ID, context length, tool environments, placement, phase, status). It introduces a program-aware scheduler with state-aware pausing and dynamic migration across GPU nodes to reduce KV…
Hao Kang, Ziyang Li, Xinyu Yang, Weili Xu, et al.arXiv.org
SLA2 is a sparse-linear attention method for diffusion models that addresses two limitations of the prior SLA approach: a heuristic split between sparse and linear branches, and a mismatch between SLA's output and the intended sparse-linear decomposition. SLA2 introduces a learnable router that dynamically selects which attention computations use sparse or…
Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, et al.arXiv.org
The paper introduces QRRanker, a listwise reranking framework that trains LLMs to score passage-query relevance using attention scores from selected Query-focused Retrieval (QR) heads, avoiding generation-based scoring and Likert-scale supervision. It uses a contrastive loss on continuous attention-derived scores, enabling training on arbitrary retrieval…
Yuqing Li, Jiangnan Li, Mo Yu, Guoxuan Ding, et al.Independent research
Step 3.5 Flash is a sparse Mixture-of-Experts (MoE) language model with 196B total parameters and 11B active parameters, designed to deliver frontier-level agentic intelligence with high computational efficiency. Its architecture uses a 3:1 interleaved Sliding Window/Full Attention layout, head-wise gated attention, and Multi-Token Prediction (MTP-3) to…
Ailin Huang, Ang Li, Aobo Kong, Bin Wang, et al.Independent research
The paper introduces Feature Activation Coverage (FAC), a metric that measures data diversity in an interpretable feature space derived from Sparse Autoencoders (SAEs) trained on LLM internal activations. The authors propose FAC Synthesis, a framework that identifies task-relevant features missing from a seed dataset and generates synthetic samples to…
Zhongzhi Li, Xuansheng Wu, Yijiang Li, Lijie Hu, et al.arXiv.org
LLaDA2.1 introduces a novel decoding scheme for discrete diffusion language models (dLLMs) that combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, controlled by dual probability thresholds. This allows two operating modes: Speedy Mode (S Mode) lowers the M2T threshold for faster generation, relying on T2T to correct errors, while Quality Mode…
Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, et al.Together AI
Aurora is a unified training-serving system that addresses limitations of conventional speculative decoding, which separates offline speculator training from online serving, causing deployment lag, delayed utility feedback, and domain-drift degradation. Aurora closes the loop by continuously learning a speculator from live inference traces, framing it as…
Junxiong Wang, Fengxiang Bie, Jisen Li, Zhongzhu Zhou, et al.arXiv.org
DFlash is a speculative decoding framework that uses a lightweight block diffusion model for parallel drafting, addressing the sequential bottleneck of autoregressive LLM decoding. It extracts hidden features from the target LLM and injects them into every draft layer's KV cache, conditioning the draft model to predict future token blocks in parallel. This…
Jian Chen, Yesheng Liang, Zhijian LiuQwen
OPUS is a dynamic data selection framework for LLM pre-training that scores candidate samples by projecting their optimizer-induced effective updates onto a target direction from a stable, in-distribution proxy. It addresses the limitations of static filters and raw-gradient-based dynamic methods by aligning selection with the actual update geometry of…
Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, et al.arXiv.org
The paper studies data efficiency for training Multimodal Process Reward Models (MPRMs), which provide step-level supervision for visual reasoning in Multimodal Large Language Models. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, which is computationally expensive. The authors find that random subsampling of training…
Jinyuan Li, Chengsong Huang, Langlin Huang, Shaoyang Xu, et al.arXiv.org
FASA is a training-free framework for query-aware token eviction in LLMs, addressing KV cache memory bottlenecks. It exploits functional sparsity in RoPE frequency chunks (FCs), where a small subset of 'dominant' FCs shows high contextual agreement with full attention heads. FASA uses a two-stage process: Token Importance Prediction (TIP) selects critical…
Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, et al.arXiv.org
The paper introduces Quartet II, a fully-NVFP4 quantization scheme for LLM pre-training on NVIDIA Blackwell GPUs. It proposes MS-EDEN, a new unbiased quantization routine for microscaling formats that reduces quantization error by more than 2x compared to stochastic rounding (SR), by applying randomized Hadamard rotations and merging EDEN correction…
Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan AlistarharXiv.org
MMFineReason is a large-scale multimodal reasoning dataset with 1.8M samples and 5.1B solution tokens, built via a three-stage pipeline: data aggregation and standardization, CoT distillation from Qwen3-VL-235B-A22B-Thinking, and quality/difficulty-based selection. The dataset covers mathematics (79.4%), science (13.8%), puzzle/game (4.6%), and general/OCR…
Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, et al.arXiv.org
This technical report from Meituan's LongCat team investigates scaling embeddings as an alternative to scaling experts in Mixture-of-Experts (MoE) language models. The authors find that N-gram Embedding, which augments token embeddings with hashed n-gram sub-tables, achieves a superior Pareto frontier compared to increasing expert count in specific…
Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, et al.arXiv.org
This paper surveys the use of large language models (LLMs) for data preparation, covering data cleaning, integration, and enrichment. It contrasts traditional rule-based and model-specific methods with LLM-enhanced approaches that leverage prompting, retrieval-augmented generation (RAG), fine-tuning, and agentic workflows. The survey identifies three core…
Wei Zhou, Jun Zhou, Haoyu Wang, Zhenghao Li, et al.Independent research
This paper challenges the assumption that arbitrary-order generation in Diffusion Large Language Models (dLLMs) enhances reasoning potential. The authors find that for general reasoning tasks like mathematics and coding, arbitrary-order generation can actually limit reasoning potential compared to standard autoregressive (AR) order. They introduce the…
Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, et al.Mistral AI
The Ministral 3 series introduces three parameter-efficient dense language models (3B, 8B, and 14B) derived from the Mistral Small 3.1 24B parent model via Cascade Distillation, an iterative pruning and distillation technique. Each size has base, instruct, and reasoning variants, all with vision capabilities and Apache 2.0 licensing. The models support up…
Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, et al.Annual Meeting of the Association for Computational Linguistics
The paper introduces RubricHub, a large-scale (~110k) multi-domain rubric dataset, generated via an automated Coarse-to-Fine framework to address limitations in existing rubric-based evaluation for open-ended tasks. The framework comprises three stages: principle-guided and response-grounded generation, multi-model aggregation, and difficulty evolution,…
Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, et al.arXiv.org
The paper introduces Multi-Head Linear Attention (MHLA), a linear attention mechanism that restores expressivity by partitioning tokens into blocks (heads) along the token dimension, computing local key-value summaries, and mixing them with query-specific learned coefficients. This addresses 'global context collapse' in standard linear attention, which…
Kewei Zhang, Ye Huang, Yufan Deng, Jincheng Yu, et al.DeepSeek
The paper introduces Engram, a conditional memory module that augments Mixture-of-Experts (MoE) large language models with static N-gram lookup tables, providing a complementary sparsity axis to conditional computation. Engram uses hashed N-gram embeddings with tokenizer compression, multi-head hashing, and context-aware gating to retrieve static knowledge…
Xin Cheng, Rui Tian, Wangding Zeng, Damai Dai, et al.Independent research
Solar Open is a 102B-parameter bilingual Mixture-of-Experts language model developed by Upstage to address data scarcity for underserved languages, focusing on Korean. The model uses a 196,608-token BPE tokenizer, a sparse MoE architecture with 12B active parameters, and was trained on 20 trillion tokens. Key innovations include synthesizing 4.5T tokens of…
Sungrae Park, Sanghoon Kim, Jungho Cho, Gyoungjin Gim, et al.Independent research
K-EXAONE is a 236B-parameter Mixture-of-Experts (MoE) language model developed by LG AI Research, activating 23B parameters per token. It supports a 256K-token context window and six languages: Korean, English, Spanish, German, Japanese, and Vietnamese. The model uses a hybrid attention mechanism (global and sliding window) and a fine-grained sparse MoE…
Eunbi Choi, Kibong Choi, Seokhee Hong, Junwon Hwang, et al.DeepSeek
The paper introduces Manifold-Constrained Hyper-Connections (mHC), a framework that addresses training instability and scalability issues in Hyper-Connections (HC) by projecting residual connection matrices onto the Birkhoff polytope (doubly stochastic matrices) using the Sinkhorn-Knopp algorithm. This restores the identity mapping property, ensuring norm…
Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, et al.arXiv.org
The paper introduces Recursive Language Models (RLMs), an inference-time framework that treats arbitrarily long prompts as part of an external environment, allowing an LLM to programmatically examine, decompose, and recursively call itself over prompt snippets. RLMs use a REPL environment where the prompt is stored as a variable, and the model writes code…
Alex L. Zhang, Tim Kraska, Omar KhattabarXiv.org
The paper introduces the expert-router coupling (ERC) loss, a lightweight auxiliary loss for Mixture-of-Experts (MoE) models that aligns router decisions with expert capabilities. The method treats each router embedding as a proxy token for the tokens routed to that expert, perturbs it with bounded noise, and feeds it through all experts to compute…
Ang Lv, Jin Ma, Yiyuan Ma, Siyuan QiaoarXiv.org
The paper introduces Next-Embedding Predictive Autoregression (NEPA), a self-supervised visual pretraining method that trains a Vision Transformer to predict future patch embeddings from past ones, using causal masking and stop-gradient, without pixel reconstruction, discrete tokens, or contrastive losses. NEPA achieves 83.8% and 85.3% top-1 accuracy on…
Sihan Xu, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, et al.arXiv.org
DataFlow is a unified, LLM-driven framework for data preparation in the era of data-centric AI, addressing the fragmentation of ad-hoc scripts and workflows. It provides a PyTorch-style API with nearly 200 reusable operators and six domain-general pipelines (text, math, code, Text-to-SQL, agentic RAG, knowledge extraction). The framework includes a global…
Hao Liang, Xiaochen Ma, Zhou Liu, Zhen Hao Wong, et al.arXiv.org
ReFusion is a masked diffusion language model that integrates sequence reorganization into a causal attention framework to overcome the limitations of standard masked diffusion models (MDMs), which suffer from high computational overhead due to the lack of Key-Value (KV) caching and incoherent generation from intractable token combination learning.…
Jia-Nan Li, Jian Guan, Wei Wu, Chongxuan LiarXiv.org
QwenLong-L1.5 is a long-context reasoning model built on Qwen3-30B-A3B-Thinking, introduced by Alibaba's Tongyi Lab. It achieves performance comparable to GPT-5 and Gemini-2.5-Pro on long-context benchmarks, surpassing its baseline by 9.90 points on average. The model's post-training recipe includes three key innovations: (1) a data synthesis pipeline that…
Weizhou Shen, Ziyi Yang, Chenliang Li, Zhiyuan Lu, et al.arXiv.org
LLaDA2.0 introduces a family of discrete diffusion language models (dLLMs) scaled up to 100B parameters by converting pre-trained auto-regressive (AR) models, avoiding costly training from scratch. The conversion uses a three-phase Warmup-Stable-Decay (WSD) continual pre-training strategy: progressively increasing block size in block diffusion (warmup),…
Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, et al.arXiv.org
The paper introduces RoPE++, an extension of Rotary Position Embeddings (RoPE) for large language models. Standard RoPE discards the imaginary component of the complex-valued attention score, losing phase information. RoPE++ re-incorporates this imaginary part as a new group of attention heads, computed in parallel with real attention. Two configurations…
Xiaoran Liu, Yuerong Song, Zhigeng Liu, Zengfeng Huang, et al.DeepSeek
DeepSeek-V3.2 introduces three key innovations to close the gap with closed-source models: DeepSeek Sparse Attention (DSA) reduces computational complexity from O(L^2) to O(Lk) while preserving long-context performance; a scalable reinforcement learning (RL) framework with a post-training compute budget exceeding 10% of pre-training cost enables…
DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, et al.arXiv.org
LFM2 is a family of Liquid Foundation Models designed for efficient on-device deployment, using a hardware-in-the-loop architecture search to create a hybrid backbone of gated short convolutions and a few grouped-query attention blocks. The family includes dense models (350M–2.6B) and an 8.3B MoE variant (1.5B active), all with 32K context. Pre-trained on…
Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, et al.Together AI
CDLM (Consistency Diffusion Language Models) is a training-based method to accelerate inference in Diffusion Language Models (DLMs) by addressing two bottlenecks: excessive refinement steps and incompatibility with KV caching. It integrates consistency modeling to reduce sampling steps via multi-token finalization and enforces a block-wise causal attention…
Minseo Kim, Chenfeng Xu, Coleman Hooper, Harman M. Singh, et al.Together AI
The paper introduces Kitty, an algorithm-system co-design for 2-bit KV cache quantization that maintains accuracy close to FP16 while reducing memory by nearly 8x. The authors observe that 4-bit quantization preserves accuracy, but 2-bit degrades it significantly. They propose Dynamic Channel-wise Precision Boost, which identifies critical key-cache…
Haojun Xia, Xiaoxia Wu, Jisen Li, Robert Wu, et al.Meta AI
The paper introduces Soup Of Category Experts (SoCE), a model souping technique that improves LLM performance by leveraging benchmark composition and non-uniform weighted averaging. SoCE identifies weakly-correlated benchmark categories, selects expert models for each, and optimizes weights to combine them. Experiments on BFCL, MGSM, and ∞-Bench show SoCE…
Shalini Maiti, Amar Budhiraja, Bhavul Gauri, Gaurav Chaurasia, et al.
