AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

AraLingBench is a fully human-annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs).

It consists of 150 expert-designed multiple-choice questions across five categories: grammar, morphology, spelling, reading comprehension, and syntax.

The benchmark was constructed by five Arabic linguistics experts from the American University of Beirut through a four-phase process including question generation, difficulty filtering, expert quality control, and difficulty annotation.

Evaluation of 35 Arabic and bilingual LLMs revealed that current models show strong surface-level proficiency but struggle with deeper grammatical and syntactic reasoning.

Top performers (Yehia-7B, ALLaM-7B) achieved 72-74% average accuracy, while syntax was the most challenging category.

Correlations showed grammar and morphology are tightly linked (r=0.83), while syntax remains independent.

Cross-benchmark analysis found that high performance on knowledge-based benchmarks like ArabicMMLU does not guarantee linguistic competence, with some models showing negative correlation with retrieval-augmented evaluation.

Human-annotated difficulty levels did not always align with model performance, as some models performed better on hard questions than medium ones.

AraLingBench provides a diagnostic framework to distinguish genuine linguistic understanding from memorization or pattern recognition.