The year/Topics/Systems and efficiency

Topic area

Systems and efficiency

Every collection across systems and efficiency.

Papers
143
Research labs
10
Official code
80

1–50 of 143 papers in this topic area

01

DeepSeek

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

DeepSeek-V3.2 introduces three key innovations to close the gap with closed-source models: DeepSeek Sparse Attention (DSA) reduces computational complexity from O(L^2) to O(Lk) while preserving long-context performance; a scalable reinforcement learning (RL) framework with a post-training compute budget exceeding 10% of pre-training cost enables…

DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, et al.
Published
Dec 2025
Citations
671
Code
Not linked
02

Z.ai / GLM

GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

GLM-4.5 is an open-source Mixture-of-Experts (MoE) large language model with 355B total and 32B activated parameters, developed by Zhipu AI and Tsinghua University. It features a hybrid reasoning method supporting both thinking and direct response modes. Trained on 23T tokens with multi-stage pre-training and mid-training (including repo-level code,…

GLM-4. 5 Team, :, Aohan Zeng, Xin Lv, et al.
Published
Aug 2025
Citations
409
Code
4.4K stars
03

arXiv.org

Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

Seed Diffusion Preview is a large-scale discrete-state diffusion language model developed by ByteDance Seed and Tsinghua AIR, focused on code generation. It achieves an inference speed of 2,146 tokens per second on H20 GPUs, significantly faster than contemporary models like Mercury Coder and Gemini Diffusion, while maintaining competitive performance on…

Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, et al.
Published
Aug 2025
Citations
163
Code
Not linked
04

arXiv.org

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

This paper systematically investigates on-policy distillation (OPD) for large language models, identifying conditions for success, the token-level mechanism, and practical remedies. The authors find that OPD succeeds only when the student and teacher share compatible thinking patterns and the teacher offers genuinely new knowledge beyond the student's…

Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, et al.
Published
Apr 2026
Citations
145
Code
892 stars
05

arXiv.org

LLaDA2.0: Scaling Up Diffusion Language Models to 100B

LLaDA2.0 introduces a family of discrete diffusion language models (dLLMs) scaled up to 100B parameters by converting pre-trained auto-regressive (AR) models, avoiding costly training from scratch. The conversion uses a three-phase Warmup-Stable-Decay (WSD) continual pre-training strategy: progressively increasing block size in block diffusion (warmup),…

Tiwei Bie, Maosong Cao, Kun Chen, Lun Du, et al.
Published
Dec 2025
Citations
123
Code
501 stars
06

Moonshot AI

Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Linear is a hybrid linear attention architecture that outperforms full attention under fair comparisons across short-context, long-context, and reinforcement learning (RL) scaling regimes. Its core, Kimi Delta Attention (KDA), extends Gated DeltaNet with a finer-grained, channel-wise gating mechanism, improving memory control and enabling efficient…

Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, et al.
Published
Oct 2025
Citations
116
Code
1.6K stars
07

NVIDIA

Fast-dLLM v2: Efficient Block-Diffusion LLM

Fast-dLLM v2 is a block diffusion language model that adapts pretrained autoregressive LLMs (Qwen2.5-Instruct 1.5B and 7B) for parallel text generation. It requires only ~1B tokens of fine-tuning, a 500x reduction compared to full-attention diffusion models like Dream (580B tokens). The method uses block-wise diffusion with complementary masking and a…

Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, et al.
Published
Sep 2025
Citations
114
Code
1.1K stars
08

arXiv.org

LightMem: Lightweight and Efficient Memory-Augmented Generation

LightMem is a lightweight and efficient memory-augmented generation system for Large Language Models (LLMs), presented at ICLR 2026. It addresses inefficiencies in existing memory systems by drawing inspiration from the Atkinson-Shiffrin human memory model, organizing memory into three stages: sensory memory (pre-compression and topic segmentation),…

Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, et al.
Published
Oct 2025
Citations
111
Code
1.1K stars
09

arXiv.org

The Principles of Diffusion Models

This book provides a comprehensive, mathematically grounded introduction to diffusion models, tracing their origins from three foundational perspectives: variational (VAEs to DDPMs), score-based (EBMs to NCSN and Score SDEs), and flow-based (Normalizing Flows to Flow Matching). It unifies these views through the Fokker-Planck equation and a common…

Chieh-Hsin Lai, Yang Song, Dongjun Kim, Yuki Mitsufuji, et al.
Published
Oct 2025
Citations
69
Code
Not linked
10

Together AI

Mamba-3: Improved Sequence Modeling using State Space Principles

Mamba-3 introduces three methodological improvements to state space models (SSMs) for better quality, capability, and inference efficiency. First, exponential-trapezoidal discretization generalizes prior heuristics, enabling a more expressive recurrence that can replace the short causal convolution. Second, complex-valued state transitions, implemented via…

Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, et al.
Published
Mar 2026
Citations
68
Code
Not linked
11

DeepSeek

mHC: Manifold-Constrained Hyper-Connections

The paper introduces Manifold-Constrained Hyper-Connections (mHC), a framework that addresses training instability and scalability issues in Hyper-Connections (HC) by projecting residual connection matrices onto the Birkhoff polytope (doubly stochastic matrices) using the Sinkhorn-Knopp algorithm. This restores the identity mapping property, ensuring norm…

Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, et al.
Published
Dec 2025
Citations
65
Code
Not linked
12

arXiv.org

Recursive Language Models

The paper introduces Recursive Language Models (RLMs), an inference-time framework that treats arbitrarily long prompts as part of an external environment, allowing an LLM to programmatically examine, decompose, and recursively call itself over prompt snippets. RLMs use a REPL environment where the prompt is stored as a variable, and the model writes code…

Alex L. Zhang, Tim Kraska, Omar Khattab
Published
Dec 2025
Citations
64
Code
5.4K stars
13

arXiv.org

DFlash: Block Diffusion for Flash Speculative Decoding

DFlash is a speculative decoding framework that uses a lightweight block diffusion model for parallel drafting, addressing the sequential bottleneck of autoregressive LLM decoding. It extracts hidden features from the target LLM and injects them into every draft layer's KV cache, conditioning the draft model to predict future token blocks in parallel. This…

Jian Chen, Yesheng Liang, Zhijian Liu
Published
Feb 2026
Citations
63
Code
5.6K stars
14

DeepSeek

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

The paper introduces Engram, a conditional memory module that augments Mixture-of-Experts (MoE) large language models with static N-gram lookup tables, providing a complementary sparsity axis to conditional computation. Engram uses hashed N-gram embeddings with tokenizer compression, multi-head hashing, and context-aware gating to retrieve static knowledge…

Xin Cheng, Rui Tian, Wangding Zeng, Damai Dai, et al.
Published
Jan 2026
Citations
63
Code
4.6K stars
15

Mistral AI

Ministral 3

The Ministral 3 series introduces three parameter-efficient dense language models (3B, 8B, and 14B) derived from the Mistral Small 3.1 24B parent model via Cascade Distillation, an iterative pruning and distillation technique. Each size has base, instruct, and reasoning variants, all with vision capabilities and Apache 2.0 licensing. The models support up…

Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, et al.
Published
Jan 2026
Citations
53
Code
Not linked
16

arXiv.org

Diffusion Language Models are Super Data Learners

The paper demonstrates that under data-constrained pre-training, diffusion language models (DLMs) consistently surpass autoregressive (AR) models of equal size, a phenomenon termed the 'Intelligence Crossover.' This crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. The…

Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, et al.
Published
Nov 2025
Citations
50
Code
228 stars
17

arXiv.org

SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention

The paper introduces SLA (Sparse-Linear Attention), a trainable attention method for Diffusion Transformers (DiTs) that combines sparse and linear attention to reduce computational cost. The authors observe that attention weights can be decomposed into a small fraction of large weights with high rank and a large fraction of low-rank weights. SLA classifies…

Jintao Zhang, Haoxu Wang, Kai Jiang, Shuo Yang, et al.
Published
Sep 2025
Citations
44
Code
328 stars
18

arXiv.org

Black-Box On-Policy Distillation of Large Language Models

The paper introduces Generative Adversarial Distillation (GAD), a method for black-box distillation of large language models (LLMs) where only the teacher's text outputs are accessible. GAD frames the student as a generator and trains a discriminator to distinguish student from teacher responses, forming a minimax game. The discriminator acts as an…

Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, et al.
Published
Nov 2025
Citations
43
Code
Not linked
19

Moonshot AI

Attention Residuals

The paper proposes Attention Residuals (AttnRes), replacing fixed unit-weight residual connections in LLMs with learned, input-dependent softmax attention over preceding layer outputs. This addresses PreNorm dilution, where hidden-state magnitudes grow with depth, diluting layer contributions. A scalable variant, Block AttnRes, partitions layers into…

Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, et al.
Published
Mar 2026
Citations
42
Code
3.5K stars
20

Volume 1

Glyph: Scaling Context Windows via Visual-Text Compression

Glyph is a framework that scales context windows by rendering long texts into images and processing them with vision-language models (VLMs), achieving 3-4x token compression while maintaining accuracy comparable to leading LLMs like Qwen3-8B. It consists of three stages: continual pre-training on rendered long-text data, an LLM-driven genetic search to…

Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, et al.
Published
Oct 2025
Citations
42
Code
595 stars
21

International Conference on Automated Software Engineering

LongCodeZip: Compress Long Context for Code Language Models

LongCodeZip is a training-free, model-agnostic, plug-and-play framework for compressing long code contexts for code LLMs. It uses a dual-stage strategy: coarse-grained compression selects function-level chunks ranked by conditional perplexity relative to the instruction, and fine-grained compression segments retained functions into blocks via…

Yuling Shi, Yichun Qian, Hongyu Zhang, Beijun Shen, et al.
Published
Oct 2025
Citations
39
Code
163 stars
22

arXiv.org

FineVision: Open Data Is All You Need

FineVision is a large-scale, open corpus of 24 million samples (17 million images, 89 million turns, 9.5 billion answer tokens) for training vision-language models. It unifies over 200 public sources into 185 subsets via a semi-automated, human-in-the-loop pipeline that includes schema mapping, cleaning, de-duplication, and decontamination against 66…

Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, et al.
Published
Oct 2025
Citations
38
Code
Not linked
23

arXiv.org

LLaDA2.1: Speeding Up Text Diffusion via Token Editing

LLaDA2.1 introduces a novel decoding scheme for discrete diffusion language models (dLLMs) that combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, controlled by dual probability thresholds. This allows two operating modes: Speedy Mode (S Mode) lowers the M2T threshold for faster generation, relying on T2T to correct errors, while Quality Mode…

Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, et al.
Published
Feb 2026
Citations
38
Code
501 stars
24

Together AI

FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

FlashAttention-4 addresses the asymmetric hardware scaling of NVIDIA Blackwell GPUs (B200/GB200), where tensor core throughput doubles compared to Hopper, but shared memory bandwidth and exponential unit throughput do not scale accordingly. This shifts bottlenecks to non-matmul operations. The paper introduces techniques to mitigate these bottlenecks: (1)…

Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timothy Liu, et al.
Published
Mar 2026
Citations
36
Code
Not linked
25

arXiv.org

Cache-to-Cache: Direct Semantic Communication Between Large Language Models

This paper introduces Cache-to-Cache (C2C), a new paradigm for direct semantic communication between large language models (LLMs) that bypasses text-based interaction. C2C projects and fuses the KV-cache of a source model (Sharer) into that of a target model (Receiver) using a neural cache fuser with a learnable gating mechanism. Oracle experiments showed…

Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, et al.
Published
Oct 2025
Citations
35
Code
425 stars
26

NVIDIA

TiDAR: Think in Diffusion, Talk in Autoregression

TiDAR is a sequence-level hybrid architecture that combines diffusion and autoregressive (AR) language modeling in a single forward pass using structured attention masks. It drafts tokens in parallel via diffusion (thinking) and samples final outputs autoregressively (talking), exploiting free GPU compute slots to achieve high throughput without quality…

Jingyu Liu, Xin Dong, Zhifan Ye, Rishabh Mehta, et al.
Published
Nov 2025
Citations
33
Code
Not linked
27

arXiv.org

MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods

MMFineReason is a large-scale multimodal reasoning dataset with 1.8M samples and 5.1B solution tokens, built via a three-stage pipeline: data aggregation and standardization, CoT distillation from Qwen3-VL-235B-A22B-Thinking, and quality/difficulty-based selection. The dataset covers mathematics (79.4%), science (13.8%), puzzle/game (4.6%), and general/OCR…

Honglin Lin, Zheng Liu, Yun Zhu, Chonghan Qin, et al.
Published
Jan 2026
Citations
33
Code
6 stars
28

Independent research

Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation

Ling 2.0 is a series of reasoning-oriented language foundation models from Inclusion AI, scaling from 16B to 1T parameters under a unified high-sparsity Mixture-of-Experts (MoE) paradigm. The series includes three non-thinking instruct models: Ling-mini-2.0 (16B total, 1.4B activated), Ling-flash-2.0 (103B total, 6.1B activated), and Ling-1T (1T total, 51B…

Ling Team, Ang Li, Ben Liu, Binbin Hu, et al.
Published
Oct 2025
Citations
30
Code
Not linked
29

Annual Meeting of the Association for Computational Linguistics

RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation

The paper introduces RubricHub, a large-scale (~110k) multi-domain rubric dataset, generated via an automated Coarse-to-Fine framework to address limitations in existing rubric-based evaluation for open-ended tasks. The framework comprises three stages: principle-guided and response-grounded generation, multi-model aggregation, and difficulty evolution,…

Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, et al.
Published
Jan 2026
Citations
29
Code
66 stars
30

Independent research

Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

Step 3.5 Flash is a sparse Mixture-of-Experts (MoE) language model with 196B total parameters and 11B active parameters, designed to deliver frontier-level agentic intelligence with high computational efficiency. Its architecture uses a 3:1 interleaved Sliding Window/Full Attention layout, head-wise gated attention, and Multi-Token Prediction (MTP-3) to…

Ailin Huang, Ang Li, Aobo Kong, Bin Wang, et al.
Published
Feb 2026
Citations
25
Code
2.1K stars
31

arXiv.org

Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs

The paper introduces Honey-Data-15M, a 15-million-sample SFT dataset for fully open multimodal large language models (MLLMs), built to address data noise and a lack of complex reasoning data. It also presents HoneyPipe, a transparent data curation pipeline built on the DataStudio framework, which uses rule-based and model-based filtering, dual-level…

Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, et al.
Published
Oct 2025
Citations
25
Code
Not linked
32

arXiv.org

QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management

QwenLong-L1.5 is a long-context reasoning model built on Qwen3-30B-A3B-Thinking, introduced by Alibaba's Tongyi Lab. It achieves performance comparable to GPT-5 and Gemini-2.5-Pro on long-context benchmarks, surpassing its baseline by 9.90 points on average. The model's post-training recipe includes three key innovations: (1) a data synthesis pipeline that…

Weizhou Shen, Ziyi Yang, Chenliang Li, Zhiyuan Lu, et al.
Published
Dec 2025
Citations
23
Code
550 stars
33

arXiv.org

LFM2 Technical Report

LFM2 is a family of Liquid Foundation Models designed for efficient on-device deployment, using a hardware-in-the-loop architecture search to create a hybrid backbone of gated short convolutions and a few grouped-query attention blocks. The family includes dense models (350M–2.6B) and an 8.3B MoE variant (1.5B active), all with 32K context. Pre-trained on…

Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, et al.
Published
Nov 2025
Citations
23
Code
Not linked
34

arXiv.org

DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI

DataFlow is a unified, LLM-driven framework for data preparation in the era of data-centric AI, addressing the fragmentation of ad-hoc scripts and workflows. It provides a PyTorch-style API with nearly 200 reusable operators and six domain-general pipelines (text, math, code, Text-to-SQL, agentic RAG, knowledge extraction). The framework includes a global…

Hao Liang, Xiaochen Ma, Zhou Liu, Zhen Hao Wong, et al.
Published
Dec 2025
Citations
19
Code
7.2K stars
35

arXiv.org

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

This paper systematically compares integer (INT) and floating-point (FP) low-bit quantization formats for LLMs across different granularities. The authors introduce a theoretical framework based on quantization signal-to-noise ratio (QSNR) and crest factor, revealing a performance crossover: FP excels at coarse granularities, but at fine-grained block-wise…

Mengzhao Chen, Meng Wu, Hui Jin, Zhihang Yuan, et al.
Published
Oct 2025
Citations
18
Code
81 stars
36

Google DeepMind

Gemma 4 Technical Report

Gemma 4 is a new family of open-weight, natively multimodal language models from Google DeepMind, featuring dense (2.3B, 4.5B, 12B, 31B) and Mixture-of-Experts (26B total, 3.8B active) architectures. Key innovations include a thinking mode for reasoning traces, an encoder-free architecture for the 12B model that processes raw audio and image patches, and…

Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, et al.
Published
Jul 2026
Citations
18
Code
Not linked
37

arXiv.org

Set Block Decoding is a Language Model Inference Accelerator

Set Block Decoding (SBD) is a new inference acceleration paradigm for autoregressive language models that integrates next token prediction (NTP) with masked token prediction (MATP) in a single architecture. SBD allows sampling multiple, non-consecutive future tokens in parallel, using solvers from discrete diffusion literature like the Entropy Bounded (EB)…

Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haziza, et al.
Published
Sep 2025
Citations
18
Code
Not linked
38

arXiv.org

Speed Always Wins: A Survey on Efficient Architectures for Large Language Models

This survey systematically reviews efficient architectures for Large Language Models (LLMs), addressing the computational bottlenecks of the standard Transformer, particularly the quadratic complexity of self-attention and the high costs of large Feed-Forward Networks. It categorizes recent innovations into seven main areas: linear sequence modeling (e.g.,…

Weigao Sun, Jiaxi Hu, Yucheng Zhou, Jusen Du, et al.
Published
Aug 2025
Citations
17
Code
406 stars
39

Together AI

CDLM: Consistency Diffusion Language Models for Faster Sampling

CDLM (Consistency Diffusion Language Models) is a training-based method to accelerate inference in Diffusion Language Models (DLMs) by addressing two bottlenecks: excessive refinement steps and incompatibility with KV caching. It integrates consistency modeling to reduce sampling steps via multi-token finalization and enforces a block-wise causal attention…

Minseo Kim, Chenfeng Xu, Coleman Hooper, Harman M. Singh, et al.
Published
Nov 2025
Citations
17
Code
41 stars
40

Z.ai / GLM

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

IndexCache is a method to accelerate sparse attention in large language models by exploiting cross-layer redundancy in token selection. DeepSeek Sparse Attention (DSA) uses a lightning indexer at each layer to select top-k tokens, but this indexer still has O(L^2) complexity. IndexCache partitions layers into a small set of Full (F) layers that run their…

Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, et al.
Published
Mar 2026
Citations
16
Code
131 stars
41

arXiv.org

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

BeyondWeb is a synthetic data generation framework for LLM pretraining that outperforms existing synthetic datasets like Cosmopedia and Nemotron-Synth by up to 5.1 and 2.6 percentage points, respectively, across 14 benchmarks. It achieves up to 7.7x faster training than open web data and 2.7x faster than Nemotron-Synth. A 3B model trained on BeyondWeb for…

DatologyAI, :, Pratyush Maini, Vineeth Dorna, et al.
Published
Aug 2025
Citations
16
Code
Not linked
42

NVIDIA

On Data Engineering for Scaling LLM Terminal Capabilities

This paper addresses the lack of transparency in training data for terminal-capable LLMs by introducing Terminal-Task-Gen, a synthetic task generation pipeline, and Terminal-Corpus, a large-scale open-source dataset. The authors fine-tune Qwen3 models (8B, 14B, 32B) to create Nemotron-Terminal, achieving substantial gains on Terminal-Bench 2.0: 8B improves…

Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, et al.
Published
Feb 2026
Citations
15
Code
Not linked
43

arXiv.org

SLA2: Sparse-Linear Attention with Learnable Routing and QAT

SLA2 is a sparse-linear attention method for diffusion models that addresses two limitations of the prior SLA approach: a heuristic split between sparse and linear branches, and a mismatch between SLA's output and the intended sparse-linear decomposition. SLA2 introduces a learnable router that dynamically selects which attention computations use sparse or…

Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, et al.
Published
Feb 2026
Citations
15
Code
328 stars
44

NVIDIA

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

TriAttention is a KV cache compression method for long-context LLM reasoning. It exploits the observation that pre-RoPE Q/K vectors are highly concentrated around fixed non-zero centers, which remain stable across positions and contexts. This concentration causes attention to follow predictable distance preferences, which can be modeled as a trigonometric…

Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, et al.
Published
Apr 2026
Citations
14
Code
838 stars
45

arXiv.org

ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding

ReFusion is a masked diffusion language model that integrates sequence reorganization into a causal attention framework to overcome the limitations of standard masked diffusion models (MDMs), which suffer from high computational overhead due to the lack of Key-Value (KV) caching and incoherent generation from intractable token combination learning.…

Jia-Nan Li, Jian Guan, Wei Wu, Chongxuan Li
Published
Dec 2025
Citations
14
Code
63 stars
46

arXiv.org

Next-Embedding Prediction Makes Strong Vision Learners

The paper introduces Next-Embedding Predictive Autoregression (NEPA), a self-supervised visual pretraining method that trains a Vision Transformer to predict future patch embeddings from past ones, using causal masking and stop-gradient, without pixel reconstruction, discrete tokens, or contrastive losses. NEPA achieves 83.8% and 85.3% top-1 accuracy on…

Sihan Xu, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, et al.
Published
Dec 2025
Citations
13
Code
340 stars
47

Together AI

ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

ThunderAgent is a program-aware agentic inference system that improves throughput for multi-turn LLM agent workflows by abstracting them as agentic programs with metadata (ID, context length, tool environments, placement, phase, status). It introduces a program-aware scheduler with state-aware pausing and dynamic migration across GPU nodes to reduce KV…

Hao Kang, Ziyang Li, Xinyu Yang, Weili Xu, et al.
Published
Feb 2026
Citations
13
Code
Not linked
48

Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)

dLLM: Simple Diffusion Language Modeling

The paper introduces dLLM, an open-source framework that standardizes the training, inference, and evaluation of diffusion language models (DLMs). It provides modular trainers for Masked Diffusion (MDLM) and Block Diffusion (BD3LM), a plug-and-play sampler abstraction supporting efficient decoding like Fast-dLLM, and a unified evaluation pipeline that…

Zhanhui Zhou, Lingjie Chen, Hanghang Tong, Dawn Song
Published
Feb 2026
Citations
12
Code
Not linked
49

DeepSeek

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

DSpark is a speculative decoding framework that combines a semi-autoregressive draft model with confidence-scheduled verification to accelerate LLM inference. The draft model uses a parallel backbone (based on DFlash) to generate tokens in one pass, followed by a lightweight sequential head (Markov or RNN) to model inter-token dependencies, mitigating…

Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, et al.
Published
Jul 2026
Citations
12
Code
6.9K stars
50

arXiv.org

Scaling Embeddings Outperforms Scaling Experts in Language Models

This technical report from Meituan's LongCat team investigates scaling embeddings as an alternative to scaling experts in Mixture-of-Experts (MoE) language models. The authors find that N-gram Embedding, which augments token embeddings with hashed n-gram sub-tables, achieves a superior Pareto frontier compared to increasing expert count in specific…

Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, et al.
Published
Jan 2026
Citations
11
Code
Not linked