Ministral 3

The Ministral 3 series introduces three parameter-efficient dense language models (3B, 8B, and 14B) derived from the Mistral Small 3.1 24B parent model via Cascade Distillation, an iterative pruning and distillation technique.

Each size has base, instruct, and reasoning variants, all with vision capabilities and Apache 2.0 licensing.

The models support up to 256k token contexts (128k for reasoning).

Cascade Distillation prunes the parent model using layer importance (activation norm ratios), PCA-based hidden dimension reduction, and feedforward importance scoring, then distills from the parent's logits in short (16k) and long (256k) context stages.

Post-training for instruct models uses SFT and Online DPO (ODPO); reasoning models use SFT with chain-of-thought, GRPO, and ODPO.

Key findings: a stronger teacher does not improve pretraining distillation, but post-trained and preference-optimized teachers are better; the 14B Base closely matches Mistral Small 3.1 while being 40% smaller.

Benchmarks show competitive or superior performance versus Qwen 3 and Gemma 3 at similar sizes, with the 14B Instruct scoring 55.1 on Arena Hard and 90.4 on MATH, and reasoning models outperforming Qwen 3 counterparts on AIME 2024/2025 and LiveCodeBench.