K-EXAONE Technical Report

K-EXAONE is a 236B-parameter Mixture-of-Experts (MoE) language model developed by LG AI Research, activating 23B parameters per token.

It supports a 256K-token context window and six languages: Korean, English, Spanish, German, Japanese, and Vietnamese.

The model uses a hybrid attention mechanism (global and sliding window) and a fine-grained sparse MoE with 128 experts, activating top-8 plus one shared expert.

It incorporates QK Norm, SWA-only RoPE, and a Multi-Token Prediction (MTP) module for faster inference.

The tokenizer was expanded to 150K vocabulary with SuperBPE.

Pre-training used 11T tokens with FP8 precision and the Muon optimizer.

Context length was extended from 8K to 256K in two stages using rehearsal, synthetic reasoning, and long-document datasets.

Post-training included SFT, reinforcement learning with verifiable rewards (AGAPO), and preference learning (GROUPER).

Evaluations show competitive performance against similar-sized models, excelling in Korean benchmarks (e.g., KOBALT 61.8) and safety (KGC-SAFETY 96.1).

The model is released under a proprietary license with restrictions on commercial distribution.