Research collection
Novel model architectures as the contribution: attention alternatives, linear and state-space models, hybrid or recurrent designs, biologically motivated architectures, and architectural scaling studies.
Moonshot AI
Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 104 billion activated parameters, native vision, and a 1-million-token context window. It uses Kimi Delta Attention, Attention Residuals, and Stable LatentMoE (16 of 896 routed experts active per token), achieving about 2.5× scaling efficiency over Kimi K2. Post-training includes…
Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, et al.Independent research
The paper introduces the Loopie series, two Mixture-of-Experts (MoE) language models: Loopie-20B-A2B and Loopie-6B-A0.6B, which use a novel 'layer-loop' recurrence pattern where each layer is applied twice before moving to the next, unlike prior 'model-loop' approaches. The key contribution is the Loopie Recipe, a compute-matched scaling method that halves…
Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, et al.Independent research
The paper introduces xHC (Expanded Hyper-Connections), a method to scale residual-stream expansion in Transformer models beyond the previous limit of N=4 streams. Existing Hyper-Connections (HC) methods, particularly Manifold-Constrained HC (mHC), show diminishing returns and rising costs when scaling N due to two bottlenecks: insufficient write-back…
Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, et al.Google DeepMind
Gemma 4 is a new family of open-weight, natively multimodal language models from Google DeepMind, featuring dense (2.3B, 4.5B, 12B, 31B) and Mixture-of-Experts (26B total, 3.8B active) architectures. Key innovations include a thinking mode for reasoning traces, an encoder-free architecture for the 12B model that processes raw audio and image patches, and…
Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, et al.Independent research
The paper introduces Grouped Query Experts (GQE), a mixture-of-experts layer applied to grouped-query attention (GQA) in Transformers. GQE routes each token to k query-head experts within each GQA group, while keeping all key-value (KV) heads dense and unchanged, preserving GQA's KV-cache benefits and reducing active query-head computation. The method…
Vishesh Tripathi, Abhay KumarIndependent research
The report introduces Ling-2.6 and Ring-2.6, a family of trillion-parameter models for efficient agentic intelligence. Ling-2.6 is optimized for instant responses and high token efficiency, while Ring-2.6 targets deeper reasoning and advanced agentic workflows. Instead of training from scratch, the models upgrade the Ling-2.0 base via architectural…
Ang Li, Ben Liu, Bin Han, Bin Hu, et al.Independent research
This paper introduces Manifold Power Iteration (MPI), a redesign of Mixture-of-Experts (MoE) routers to align each router row with the principal singular direction of its associated expert's weight matrix. The method uses a 'Power-then-Retract' paradigm: a single power iteration step on router weights followed by L2 norm retraction to ensure stability.…
Songhao Wu, Ang Lv, Ruobing Xie, Yankai LinarXiv.org
Mellum 2 is an open-weight 12B-parameter Mixture-of-Experts (MoE) language model with 2.5B active parameters per token, designed for software engineering tasks. It uses 64 experts with 8 active, Grouped-Query Attention with 4 KV heads, Sliding Window Attention on 3 of every 4 layers, and a Multi-Token Prediction head for speculative decoding. Pre-training…
Marko Kojic, Ivan Bondyrev, Aral de Moor, Joseph Shtok, et al.arXiv.org
HRM-Text introduces a Hierarchical Recurrent Model (HRM) that decouples computation into slow strategic and fast execution layers, inspired by biological multi-timescale processing. Trained from scratch on only 40B unique tokens (60B total with repetition) and a $1,500 budget, a 1B-parameter model achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP,…
Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, et al.arXiv.org
The Darwin Family framework enables training-free evolutionary merging of large language models via gradient-free weight-space recombination, improving reasoning performance without additional training. It introduces a 14-dimensional adaptive merge genome, MRI-Trust Fusion (balancing diagnostic layer-importance signals with evolutionary search via a…
Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, et al.arXiv.org
This paper identifies a failure mode in ultra-deep Diffusion Transformers (DiTs) called Mean Mode Screaming (MMS), where token representations homogenize and centered variation is suppressed, leading to training collapse. The mechanism involves a mean-coherent gradient shock that opens residual branches, causing a mean-dominated state. The authors propose…
Pengqi LuarXiv.org
The paper introduces Mixture-of-Depths Attention (MoDA), a mechanism for large language models that allows each attention head to attend to both sequence KV pairs at the current layer and depth KV pairs from preceding layers, addressing information dilution in deep Transformers. MoDA uses a unified softmax over sequence and depth attention, and a…
Lianghui Zhu, Yuxin Fang, Bencheng Liao, Shijie Wang, et al.Moonshot AI
The paper proposes Attention Residuals (AttnRes), replacing fixed unit-weight residual connections in LLMs with learned, input-dependent softmax attention over preceding layer outputs. This addresses PreNorm dilution, where hidden-state magnitudes grow with depth, diluting layer contributions. A scalable variant, Block AttnRes, partitions layers into…
Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, et al.Together AI
Mamba-3 introduces three methodological improvements to state space models (SSMs) for better quality, capability, and inference efficiency. First, exponential-trapezoidal discretization generalizes prior heuristics, enabling a more expressive recurrence that can replace the short causal convolution. Second, complex-valued state transitions, implemented via…
Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, et al.arXiv.org
HyTRec is a generative recommendation framework designed to efficiently model ultra-long user behavior sequences. It addresses the trade-off between linear attention's efficiency and softmax attention's precision by using a hybrid architecture: a linear attention branch (Temporal-Aware Delta Network, TADN) processes long-term history, while a softmax…
Lei Xin, Yuhao Zheng, Ke Cheng, Changjiang Jiang, et al.Independent research
Step 3.5 Flash is a sparse Mixture-of-Experts (MoE) language model with 196B total parameters and 11B active parameters, designed to deliver frontier-level agentic intelligence with high computational efficiency. Its architecture uses a 3:1 interleaved Sliding Window/Full Attention layout, head-wise gated attention, and Multi-Token Prediction (MTP-3) to…
Ailin Huang, Ang Li, Aobo Kong, Bin Wang, et al.arXiv.org
This technical report from Meituan's LongCat team investigates scaling embeddings as an alternative to scaling experts in Mixture-of-Experts (MoE) language models. The authors find that N-gram Embedding, which augments token embeddings with hashed n-gram sub-tables, achieves a superior Pareto frontier compared to increasing expert count in specific…
Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, et al.arXiv.org
The paper introduces Multi-Head Linear Attention (MHLA), a linear attention mechanism that restores expressivity by partitioning tokens into blocks (heads) along the token dimension, computing local key-value summaries, and mixing them with query-specific learned coefficients. This addresses 'global context collapse' in standard linear attention, which…
Kewei Zhang, Ye Huang, Yufan Deng, Jincheng Yu, et al.DeepSeek
The paper introduces Engram, a conditional memory module that augments Mixture-of-Experts (MoE) large language models with static N-gram lookup tables, providing a complementary sparsity axis to conditional computation. Engram uses hashed N-gram embeddings with tokenizer compression, multi-head hashing, and context-aware gating to retrieve static knowledge…
Xin Cheng, Rui Tian, Wangding Zeng, Damai Dai, et al.Independent research
Solar Open is a 102B-parameter bilingual Mixture-of-Experts language model developed by Upstage to address data scarcity for underserved languages, focusing on Korean. The model uses a 196,608-token BPE tokenizer, a sparse MoE architecture with 12B active parameters, and was trained on 20 trillion tokens. Key innovations include synthesizing 4.5T tokens of…
Sungrae Park, Sanghoon Kim, Jungho Cho, Gyoungjin Gim, et al.Independent research
K-EXAONE is a 236B-parameter Mixture-of-Experts (MoE) language model developed by LG AI Research, activating 23B parameters per token. It supports a 256K-token context window and six languages: Korean, English, Spanish, German, Japanese, and Vietnamese. The model uses a hybrid attention mechanism (global and sliding window) and a fine-grained sparse MoE…
Eunbi Choi, Kibong Choi, Seokhee Hong, Junwon Hwang, et al.DeepSeek
The paper introduces Manifold-Constrained Hyper-Connections (mHC), a framework that addresses training instability and scalability issues in Hyper-Connections (HC) by projecting residual connection matrices onto the Birkhoff polytope (doubly stochastic matrices) using the Sinkhorn-Knopp algorithm. This restores the identity mapping property, ensuring norm…
Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, et al.arXiv.org
The paper introduces the expert-router coupling (ERC) loss, a lightweight auxiliary loss for Mixture-of-Experts (MoE) models that aligns router decisions with expert capabilities. The method treats each router embedding as a proxy token for the tokens routed to that expert, perturbs it with bounded noise, and feeds it through all experts to compute…
Ang Lv, Jin Ma, Yiyuan Ma, Siyuan QiaoarXiv.org
The paper introduces Next-Embedding Predictive Autoregression (NEPA), a self-supervised visual pretraining method that trains a Vision Transformer to predict future patch embeddings from past ones, using causal masking and stop-gradient, without pixel reconstruction, discrete tokens, or contrastive losses. NEPA achieves 83.8% and 85.3% top-1 accuracy on…
Sihan Xu, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, et al.DeepSeek
DeepSeek-V3.2 introduces three key innovations to close the gap with closed-source models: DeepSeek Sparse Attention (DSA) reduces computational complexity from O(L^2) to O(Lk) while preserving long-context performance; a scalable reinforcement learning (RL) framework with a post-training compute budget exceeding 10% of pre-training cost enables…
DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, et al.arXiv.org
LFM2 is a family of Liquid Foundation Models designed for efficient on-device deployment, using a hardware-in-the-loop architecture search to create a hybrid backbone of gated short convolutions and a few grouped-query attention blocks. The family includes dense models (350M–2.6B) and an 8.3B MoE variant (1.5B active), all with 32K context. Pre-trained on…
Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, et al.arXiv.org
This paper introduces Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress K tokens into a single continuous vector, reducing the number of autoregressive steps by a factor of K. The authors develop a likelihood-free…
Chenze Shao, Darren Li, Fandong Meng, Jie ZhouMoonshot AI
Kimi Linear is a hybrid linear attention architecture that outperforms full attention under fair comparisons across short-context, long-context, and reinforcement learning (RL) scaling regimes. Its core, Kimi Delta Attention (KDA), extends Gated DeltaNet with a finer-grained, channel-wise gating mechanism, improving memory control and enabling efficient…
Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, et al.Independent research
Ling 2.0 is a series of reasoning-oriented language foundation models from Inclusion AI, scaling from 16B to 1T parameters under a unified high-sparsity Mixture-of-Experts (MoE) paradigm. The series includes three non-thinking instruct models: Ling-mini-2.0 (16B total, 1.4B activated), Ling-flash-2.0 (103B total, 6.1B activated), and Ling-1T (1T total, 51B…
Ling Team, Ang Li, Ben Liu, Binbin Hu, et al.arXiv.org
This technical report introduces the Ring-linear model series, comprising Ring-mini-linear-2.0 (16B total, 1.6B active parameters) and Ring-flash-linear-2.0 (104B total, 7.4B active). Both use a hybrid architecture that combines linear and softmax attention to reduce I/O and computational costs in long-context scenarios, cutting inference cost to 1/10 of a…
Ling Team, Bin Han, Caizhi Tang, Chen Liang, et al.arXiv.org
UniMoE-Audio is a unified speech and music generation model that addresses task conflict and data imbalance through a Dynamic-Capacity Mixture-of-Experts (MoE) framework. The architecture introduces a Top-P routing strategy for dynamic expert allocation and a hybrid expert design with routed, shared, and null experts. To handle data imbalance, a…
Zhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng, et al.arXiv.org
This paper introduces Cache-to-Cache (C2C), a new paradigm for direct semantic communication between large language models (LLMs) that bypasses text-based interaction. C2C projects and fuses the KV-cache of a source model (Sharer) into that of a target model (Receiver) using a neural cache fuser with a learnable gating mechanism. Oracle experiments showed…
Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, et al.arXiv.org
The paper introduces Dragon Hatchling (BDH), a new Large Language Model architecture based on a scale-free, biologically inspired network of n locally-interacting neuron particles. BDH couples strong theoretical foundations and inherent interpretability without sacrificing Transformer-like performance. It is a practical, performant attention-based state…
Adrian Kosowski, Przemysław Uznański, Jan Chorowski, Zuzanna Stamirowska, et al.arXiv.org
This survey systematically reviews efficient architectures for Large Language Models (LLMs), addressing the computational bottlenecks of the standard Transformer, particularly the quadratic complexity of self-attention and the high costs of large Feed-Forward Networks. It categorizes recent innovations into seven main areas: linear sequence modeling (e.g.,…
Weigao Sun, Jiaxi Hu, Yucheng Zhou, Jusen Du, et al.Z.ai / GLM
GLM-4.5 is an open-source Mixture-of-Experts (MoE) large language model with 355B total and 32B activated parameters, developed by Zhipu AI and Tsinghua University. It features a hybrid reasoning method supporting both thinking and direct response modes. Trained on 23T tokens with multi-stage pre-training and mid-training (including repo-level code,…
GLM-4. 5 Team, :, Aohan Zeng, Xin Lv, et al.
