The Year in AI Papers
The year/Topics/Systems and efficiency

Topic area

Systems and efficiency

Every collection across systems and efficiency.

Papers
143
Research labs
10
Official code
80

1–50 of 143 papers in this topic area

01

Independent research

Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

This paper introduces Memory Decoder at Scale, scaling parametric long-term memory models up to 6.9B parameters and pretraining them on 300B tokens. To handle the computational bottleneck of constructing kNN distributions over 207B tokens, the authors develop a distributed Faiss pipeline using embedding compression, index sharding, and parallel search,…

Rubin Wei, Jiaqi Cao, Jiarui Wang, Junming Zhang, et al.
Published
Jul 2026
Citations
0
Code
13 stars
Read summary Original
02

Independent research

Metis: Memory Foundation Model

The paper introduces memory foundation models, which internalize memory into the backbone of large foundation models, contrasting with external memory modules like RAG. The authors formalize native memory via a persistent, dynamically evolving memory state within the model's parameters and native memory procedures that autonomously store and utilize…

Zeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang, et al.
Published
Jul 2026
Citations
0
Code
94 stars
Read summary Original
03

Moonshot AI

Kimi K3: Open Frontier Intelligence

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 104 billion activated parameters, native vision, and a 1-million-token context window. It uses Kimi Delta Attention, Attention Residuals, and Stable LatentMoE (16 of 896 routed experts active per token), achieving about 2.5× scaling efficiency over Kimi K2. Post-training includes…

Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, et al.
Published
Jul 2026
Citations
2
Code
8.1K stars
Read summary Original
04

Independent research

SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

This technical report presents SLAI T-Rex, a full-stack framework for post-training the DeepSeek-V4 model family on Ascend SuperPOD. System-level optimizations (parallelism, communication, memory, kernels) increased Model FLOPs Utilization (MFU) from 11.67% to 34.22%, a 2.93x improvement. For Operations Research (OR) specialization, a solver-grounded…

Dongfang Li, Xiaodong Luo, Ruoyu Sun, Xuhui Chen, et al.
Published
Jul 2026
Citations
0
Code
32 stars
Read summary Original
05

Independent research

Loop the Loopies!

The paper introduces the Loopie series, two Mixture-of-Experts (MoE) language models: Loopie-20B-A2B and Loopie-6B-A0.6B, which use a novel 'layer-loop' recurrence pattern where each layer is applied twice before moving to the next, unlike prior 'model-loop' approaches. The key contribution is the Loopie Recipe, a compute-matched scaling method that halves…

Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, et al.
Published
Jul 2026
Citations
0
Code
Not linked
Read summary Original
06

Independent research

xHC: Expanded Hyper-Connections

The paper introduces xHC (Expanded Hyper-Connections), a method to scale residual-stream expansion in Transformer models beyond the previous limit of N=4 streams. Existing Hyper-Connections (HC) methods, particularly Manifold-Constrained HC (mHC), show diminishing returns and rising costs when scaling N due to two bottlenecks: insufficient write-back…

Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, et al.
Published
Jul 2026
Citations
0
Code
60 stars
Read summary Original
07

DeepSeek

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

DSpark is a speculative decoding framework that combines a semi-autoregressive draft model with confidence-scheduled verification to accelerate LLM inference. The draft model uses a parallel backbone (based on DFlash) to generate tokens in one pass, followed by a lightweight sequential head (Markov or RNN) to model inter-token dependencies, mitigating…

Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, et al.
Published
Jul 2026
Citations
12
Code
6.9K stars
Read summary Original
08

Independent research

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

The paper introduces Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism for large language models that learns chunk selection end-to-end under the language-modeling loss. HiLS factorizes attention hierarchically: each query attends independently to retrieved chunks and fuses outputs using retrieval scores, which are…

Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, et al.
Published
Jul 2026
Citations
1
Code
138 stars
Read summary Original
09

Google DeepMind

Gemma 4 Technical Report

Gemma 4 is a new family of open-weight, natively multimodal language models from Google DeepMind, featuring dense (2.3B, 4.5B, 12B, 31B) and Mixture-of-Experts (26B total, 3.8B active) architectures. Key innovations include a thinking mode for reasoning traces, an encoder-free architecture for the 12B model that processes raw audio and image patches, and…

Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, et al.
Published
Jul 2026
Citations
18
Code
Not linked
Read summary Original
10

Independent research

Program-as-Weights: A Programming Paradigm for Fuzzy Functions

The paper introduces Program-as-Weights (PAW), a paradigm for implementing fuzzy functions (tasks like log filtering or intent classification) by compiling natural-language specifications into compact neural programs. A 4B compiler, trained on the new FuzzyBench dataset (10M examples), emits a hybrid program: a discrete pseudo-program (paraphrase plus…

Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, et al.
Published
Jul 2026
Citations
0
Code
Not linked
Read summary Original
11

Independent research

BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decoding

BlockPilot introduces a sample-adaptive policy for diffusion-based speculative decoding, addressing the inefficiency of fixed block sizes. The authors observe that the optimal block size varies across samples but concentrates near the training block size, enabling a lightweight classification approach. BlockPilot predicts the block size from the last…

Hao Zhang, Yiming Hu, Yong Wang, Mingqiao Mo, et al.
Published
Jun 2026
Citations
0
Code
Not linked
Read summary Original
12

Independent research

DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

DataClaw0 is a framework for intent-conditioned multimodal data tailoring, converting raw, noisy streams (e.g., tutorial videos, embodied trajectories, GUI logs) into schema-aligned, evidence-grounded training instances. The authors formalize this as a learnable capability and train models at 4B, 9B, and 27B scales, comparing a jointly trained omni model…

Cong Wan, Zeyu Guo, Zijian Cai, Jiangyang Li, et al.
Published
Jun 2026
Citations
0
Code
117 stars
Read summary Original
13

Independent research

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

The paper introduces Grouped Query Experts (GQE), a mixture-of-experts layer applied to grouped-query attention (GQA) in Transformers. GQE routes each token to k query-head experts within each GQA group, while keeping all key-value (KV) heads dense and unchanged, preserving GQA's KV-cache benefits and reducing active query-head computation. The method…

Vishesh Tripathi, Abhay Kumar
Published
Jun 2026
Citations
0
Code
Not linked
Read summary Original
14

Independent research

PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models

The paper introduces PerceptionDLM, a multimodal diffusion language model for efficient parallel region perception. It builds on PerceptionDLM-Base, a strong diffusion-based vision-language model, and adds region prompting, RoI-aligned feature replay, and structured attention masking to generate captions for multiple image regions simultaneously in a…

Yueyi Sun, Yuhao Wang, Jason Li, Ye Tian, et al.
Published
Jun 2026
Citations
0
Code
77 stars
Read summary Original
15

Independent research

Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

The report introduces Ling-2.6 and Ring-2.6, a family of trillion-parameter models for efficient agentic intelligence. Ling-2.6 is optimized for instant responses and high token efficiency, while Ring-2.6 targets deeper reasoning and advanced agentic workflows. Instead of training from scratch, the models upgrade the Ling-2.0 base via architectural…

Ang Li, Ben Liu, Bin Han, Bin Hu, et al.
Published
Jun 2026
Citations
2
Code
Not linked
Read summary Original
16

MiniMax

MiniMax Sparse Attention

MiniMax Sparse Attention (MSA) is a blockwise sparse attention mechanism built on Grouped Query Attention (GQA) to address the quadratic cost of softmax attention in ultra-long contexts. A lightweight Index Branch scores key-value blocks and selects a Top-k subset per GQA group, while the Main Branch computes exact block-sparse attention over selected…

Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, et al.
Published
Jun 2026
Citations
7
Code
442 stars
Read summary Original
17

Independent research

Redesign Mixture-of-Experts Routers with Manifold Power Iteration

This paper introduces Manifold Power Iteration (MPI), a redesign of Mixture-of-Experts (MoE) routers to align each router row with the principal singular direction of its associated expert's weight matrix. The method uses a 'Power-then-Retract' paradigm: a single power iteration step on router weights followed by L2 norm retraction to ensure stability.…

Songhao Wu, Ang Lv, Ruobing Xie, Yankai Lin
Published
Jun 2026
Citations
0
Code
Not linked
Read summary Original
18

Independent research

FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

This report introduces FlashMemory-DeepSeek-V4 (FM-DS-V4), which uses Lookahead Sparse Attention (LSA) to reduce GPU memory for ultra-long context LLM serving. LSA adds a Neural Memory Indexer to DeepSeek-V4-Flash, predicting and fetching only critical KV chunks every 64 steps, instead of keeping the full KV cache. The indexer is trained independently as a…

Yan Wang, Qifan Zhang, Jiachen Yu, Tian Liang, et al.
Published
Jun 2026
Citations
2
Code
101 stars
Read summary Original
19

Independent research

Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

Code2LoRA is a hypernetwork framework that generates repository-specific LoRA adapters for code language models, injecting repository knowledge with zero inference-time token overhead. It has two usage scenarios: Code2LoRA-Static maps a single repository snapshot to an adapter for stable codebases, and Code2LoRA-Evo maintains an adapter via a GRU hidden…

Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie
Published
Jun 2026
Citations
1
Code
Not linked
Read summary Original
20

Independent research

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

The paper introduces KVarN, a calibration-free KV-cache quantization method that mitigates error accumulation during autoregressive decoding in long-horizon reasoning tasks. It identifies that quantization errors accumulate across timesteps, driven primarily by incorrect token scales, which cause outlier errors that disproportionately degrade end-to-end…

Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, et al.
Published
Jun 2026
Citations
3
Code
452 stars
Read summary Original
21

Independent research

On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters

This paper argues that parameter-efficient fine-tuning (PEFT), especially LoRA, can scale from a single shared foundation model to millions of persistent personal model instances. The authors propose a three-axis framework: Scale Up (strengthening the shared base model), Scale Down (shrinking the local adaptive state), and Scale Out (sustaining a large…

Mind Lab, :, Vin Bo, Song Cao, et al.
Published
Jun 2026
Citations
2
Code
Not linked
Read summary Original
22

Independent research

OCC-RAG: Optimal Cognitive Core for Faithful Question Answering

The paper introduces OCC-RAG, a family of small language models (SLMs) specialized for faithful, context-grounded question answering (QA). The models are mid-trained from Qwen3 base models (0.6B and 1.7B) on a synthetic corpus of over 3 million QA examples generated via a novel pipeline. The corpus includes single-hop, multi-hop (single- and…

Maksim Savkin, Mikhail Goncharov, Alexander Gambashidze, Alla Chepurova, et al.
Published
May 2026
Citations
0
Code
43 stars
Read summary Original
23

arXiv.org

Mellum2 Technical Report

Mellum 2 is an open-weight 12B-parameter Mixture-of-Experts (MoE) language model with 2.5B active parameters per token, designed for software engineering tasks. It uses 64 experts with 8 active, Grouped-Query Attention with 4 KV heads, Sliding Window Attention on 3 of every 4 layers, and a Multi-Token Prediction head for speculative decoding. Pre-training…

Marko Kojic, Ivan Bondyrev, Aral de Moor, Joseph Shtok, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
24

arXiv.org

Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

Domino is a speculative decoding framework that decouples causal dependency modeling from expensive autoregressive draft execution. It uses a parallel draft backbone (based on DFlash) to generate preliminary draft distributions for the entire block, then applies a lightweight Domino head—comprising a GRU causal encoder and a low-rank correction head—to…

Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, et al.
Published
May 2026
Citations
8
Code
128 stars
Read summary Original
25

arXiv.org

ACC: Compiling Agent Trajectories for Long-Context Training

The paper introduces Agent Context Compilation (ACC), a method that converts multi-turn agent trajectories (from search, software engineering, and SQL agents) into long-context QA training pairs. Standard agent SFT masks tool responses, creating a supervision blind spot where evidence scattered across turns is unused. ACC compiles tool responses and…

Qisheng Su, Zhen Fang, Shiting Huang, Yu Zeng, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
26

arXiv.org

HRM-Text: Efficient Pretraining Beyond Scaling

HRM-Text introduces a Hierarchical Recurrent Model (HRM) that decouples computation into slow strategic and fast execution layers, inspired by biological multi-timescale processing. Trained from scratch on only 40B unique tokens (60B total with repetition) and a $1,500 budget, a 1B-parameter model achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP,…

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, et al.
Published
May 2026
Citations
3
Code
1.8K stars
Read summary Original
27

Independent research

DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

DataPrep-Bench is the first unified benchmark for evaluating LLMs as training data preparators, covering two complementary capabilities: data construction (transforming raw sources into supervised training data) and data quality evaluation (predicting downstream training utility of candidate datasets). The benchmark is grounded in downstream performance…

Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
28

Together AI

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

OSCAR is an INT2 KV-cache quantization method that uses attention-aware covariance structures, estimated offline, to derive fixed rotations and clipping thresholds, aligning quantization with downstream attention. It composes rotations as R = U · H_Had · P_br, where U is the eigenbasis of query-aware (Q^T Q) or score-aware (V^T S^T S V) covariance, H_Had…

Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, et al.
Published
May 2026
Citations
2
Code
555 stars
Read summary Original
29

arXiv.org

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

RTPurbo is a method that transforms full-attention large language models into highly sparse models with minimal adaptation, requiring only a few hundred training steps. It is based on three observations: only a small subset of attention heads (retrieval heads) truly need full long-context processing; long-range retrieval is governed by a low-dimensional…

Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, et al.
Published
May 2026
Citations
1
Code
Not linked
Read summary Original
30

arXiv.org

Darwin Family: MRI-Trust-Weighted Evolutionary Merging for Training-Free Scaling of Language-Model Reasoning

The Darwin Family framework enables training-free evolutionary merging of large language models via gradient-free weight-space recombination, improving reasoning performance without additional training. It introduces a 14-dimensional adaptive merge genome, MRI-Trust Fusion (balancing diagnostic layer-importance signals with evolutionary search via a…

Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
31

arXiv.org

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

This paper presents a systematic study of long-context continued pre-training (LongPT) for large vision-language models (LVLMs), extending a 7B model from 32K to 128K context. The authors find that long-document VQA is substantially more effective than OCR transcription for training. Key findings include: balanced sequence-length distributions outperform…

Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
32

arXiv.org

MinT: Managed Infrastructure for Training and Serving Millions of LLMs

MinT is a managed infrastructure system for Low-Rank Adaptation (LoRA) post-training and online serving, designed for settings where many trained policies are produced over a small number of expensive base-model deployments. Instead of materializing full checkpoints, MinT keeps the base model resident and moves exported LoRA adapter revisions through…

Mind Lab, :, Song Cao, Vic Cao, et al.
Published
May 2026
Citations
1
Code
Not linked
Read summary Original
33

arXiv.org

$δ$-mem: Efficient Online Memory for Large Language Models

δ-mem is a lightweight memory mechanism that augments a frozen full-attention LLM with a compact online state of associative memory, updated via delta-rule learning. It compresses past information into a fixed-size state matrix (e.g., 8×8) and uses its readout to generate low-rank corrections to the backbone's attention computation, avoiding full…

Jingdi Lei, Di Zhang, Junxian Li, Weida Wang, et al.
Published
May 2026
Citations
3
Code
253 stars
Read summary Original
34

arXiv.org

MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image

MulTaBench is a benchmark of 40 datasets (20 image-tabular, 20 text-tabular) designed for Multimodal Tabular Learning (MMTL). It addresses the limitation of existing benchmarks that focus on modality co-occurrence, which masks the benefits of task-specific tuning. The authors propose two criteria for dataset inclusion: Joint Signal (each modality provides…

Alan Arazi, Eilam Shapira, Shoham Grunblat, Mor Ventura, et al.
Published
May 2026
Citations
2
Code
Not linked
Read summary Original
35

arXiv.org

K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

The paper introduces K12-KGraph, a curriculum-aligned knowledge graph built from official Chinese People's Education Press textbooks, covering mathematics, physics, chemistry, and biology across primary, middle, and high school. It includes nine node types and fourteen relation types, capturing both curriculum structure and visual grounding. From this…

Hao Liang, Qihan Lin, Zhaoyang Han, Xiaochen Ma, et al.
Published
May 2026
Citations
0
Code
314 stars
Read summary Original
36

arXiv.org

Continuous Latent Diffusion Language Model

Cola DLM is a hierarchical latent diffusion language model that decomposes text generation into global semantic modeling in a continuous latent space and local textual realization via a conditional decoder. It uses a Text VAE to map text to latent variables, a block-causal DiT to model the latent prior via flow matching, and a decoder for text generation.…

Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, et al.
Published
May 2026
Citations
4
Code
271 stars
Read summary Original
37

arXiv.org

Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers

This paper identifies a failure mode in ultra-deep Diffusion Transformers (DiTs) called Mean Mode Screaming (MMS), where token representations homogenize and centered variation is suppressed, leading to training collapse. The mechanism involves a mean-coherent gradient shock that opens residual branches, causing a mean-dominated state. The authors propose…

Pengqi Lu
Published
May 2026
Citations
0
Code
9 stars
Read summary Original
38

arXiv.org

Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora

The paper introduces Programming with Data (ProDa), a paradigm that maps the LLM data-engineering lifecycle onto software development by using a shared three-level knowledge structure (L1 concepts, L2 relations, L3 reasoning chains) extracted from raw corpora. This structure serves as the common foundation for both training data synthesis and benchmark…

Chenkai Pan, Xinglong Xu, Yuhang Xu, Yujun Wu, et al.
Published
Apr 2026
Citations
1
Code
162 stars
Read summary Original
39

arXiv.org

LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model

LLaDA2.0-Uni is a unified discrete diffusion large language model (dLLM) that integrates multimodal understanding and generation in a single framework. Its architecture combines a SigLIP-VQ semantic tokenizer, a 16B MoE dLLM backbone, and a diffusion decoder. The tokenizer converts images into discrete semantic tokens, enabling block-level masked diffusion…

Inclusion AI, Tiwei Bie, Haoxing Chen, Tieyuan Chen, et al.
Published
Apr 2026
Citations
5
Code
772 stars
Read summary Original
40

arXiv.org

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

This paper systematically investigates on-policy distillation (OPD) for large language models, identifying conditions for success, the token-level mechanism, and practical remedies. The authors find that OPD succeeds only when the student and teacher share compatible thinking patterns and the teacher offers genuinely new knowledge beyond the student's…

Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, et al.
Published
Apr 2026
Citations
145
Code
892 stars
Read summary Original
41

arXiv.org

MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping

The paper introduces MegaStyle, a scalable data curation pipeline for constructing a large-scale, high-quality style dataset (MegaStyle-1.4M) with intra-style consistency and inter-style diversity. It leverages the consistent text-to-image style mapping of large generative models like Qwen-Image to generate style pairs from curated prompts. The pipeline…

Junyao Gao, Sibo Liu, Jiaxing Li, Yanan Sun, et al.
Published
Apr 2026
Citations
2
Code
131 stars
Read summary Original
42

arXiv.org

DMax: Aggressive Parallel Decoding for dLLMs

DMax is a new paradigm for diffusion language models (dLLMs) that enables aggressive parallel decoding while preserving generation quality by mitigating error accumulation. It reformulates decoding as a progressive self-refinement from mask embeddings to token embeddings, using two key components: On-Policy Uniform Training (OPUT) and Soft Parallel…

Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, et al.
Published
Apr 2026
Citations
8
Code
128 stars
Read summary Original
43

NVIDIA

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

TriAttention is a KV cache compression method for long-context LLM reasoning. It exploits the observation that pre-RoPE Q/K vectors are highly concentrated around fixed non-zero centers, which remain stable across positions and contexts. This concentration causes attention to follow predictable distance preferences, which can be modeled as a trigonometric…

Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, et al.
Published
Apr 2026
Citations
14
Code
838 stars
Read summary Original
44

arXiv.org

Adam's Law: Textual Frequency Law on Large Language Models

The paper proposes the Textual Frequency Law (TFL), which states that for LLMs, higher-frequency textual data should be preferred for both prompting and fine-tuning when meaning is kept constant. The framework includes Textual Frequency Distillation (TFD) to enhance frequency estimation via story completion, and Curriculum Textual Frequency Training (CTFT)…

Hongyuan Adam Lu, Z. L., Victor Wei, Zefan Zhang, et al.
Published
Apr 2026
Citations
3
Code
Not linked
Read summary Original
45

arXiv.org

Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development

This survey reviews over 1,000 open-access medical imaging datasets released between 2000 and 2025, analyzing their modalities, tasks, anatomical regions, and limitations. It finds the landscape is fragmented, small-scale, and unevenly distributed, with 2D images dominating, pathology and X-ray being the most common modalities, and classification and…

Zhongying Deng, Cheng Tang, Ziyan Huang, Jiashi Lin, et al.
Published
Mar 2026
Citations
2
Code
472 stars
Read summary Original
46

arXiv.org

TAPS: Task Aware Proposal Distributions for Speculative Sampling

This paper investigates how the training distribution of draft models affects speculative decoding performance, using HASS and EAGLE-2 drafters trained on MathInstruct, ShareGPT, and mixed data, evaluated on MT-Bench, GSM8K, MATH-500, and SVAMP. Results show task-specific training yields clear specialization: MathInstruct-trained drafts excel on reasoning…

Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna, Hasan Abed Al Kader Hammoud, et al.
Published
Mar 2026
Citations
0
Code
8 stars
Read summary Original
47

arXiv.org

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

DataFlex is a unified data-centric dynamic training framework built on LLaMA-Factory, designed to treat data as a first-class optimization variable. It integrates three paradigms—data selection, data mixture optimization, and data reweighting—through modular trainer abstractions (Select, Mix, Weight Trainers) and pluggable components (selectors, mixers,…

Hao Liang, Zhengyang Zhao, Meiyi Qiang, Mingrui Chen, et al.
Published
Mar 2026
Citations
3
Code
1.9K stars
Read summary Original
48

arXiv.org

SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

SpecEyes is an agentic-level speculative acceleration framework for multimodal large language models (MLLMs) that addresses the sequential bottleneck of tool-use loops. It uses a lightweight, tool-free model to speculatively answer queries that do not require deep agentic reasoning, bypassing the expensive tool-calling pipeline. The framework consists of…

Haoyu Huang, Jinfa Huang, Zhongwei Wan, Xiawu Zheng, et al.
Published
Mar 2026
Citations
1
Code
63 stars
Read summary Original
49

Qwen

HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning

The paper introduces HopChain, a scalable framework for synthesizing multi-hop vision-language reasoning data for reinforcement learning with verifiable rewards (RLVR) training of vision-language models (VLMs). The authors identify that long chain-of-thought (CoT) reasoning in VLMs suffers from diverse, compounding failure modes (perception, reasoning,…

Shenzhi Wang, Shixuan Liu, Jing Zhou, Chang Gao, et al.
Published
Mar 2026
Citations
2
Code
Not linked
Read summary Original
50

arXiv.org

Mixture-of-Depths Attention

The paper introduces Mixture-of-Depths Attention (MoDA), a mechanism for large language models that allows each attention head to attend to both sequence KV pairs at the current layer and depth KV pairs from preceding layers, addressing information dilution in deep Transformers. MoDA uses a unified softmax over sequence and depth attention, and a…

Lianghui Zhu, Yuxin Fang, Bencheng Liao, Shijie Wang, et al.
Published
Mar 2026
Citations
3
Code
274 stars
Read summary Original