Paper 2508.13491
From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 4
- GitHub
- 3 stars
01 In brief
Summary
The paper introduces FinCDM, the first cognitive diagnosis evaluation framework for financial large language models (LLMs), moving beyond aggregate scores to assess knowledge-skill level proficiency.
It constructs CPA-KQA, a dataset of 210 expert-annotated questions derived from the CPA exam, covering 70 financial concepts, with high inter-annotator agreement (Krippendorff's alpha 0.937).
FinCDM uses non-negative matrix co-factorization to estimate model mastery across concepts.
Evaluations on 30 LLMs reveal that models with similar overall scores differ markedly in concept-level mastery, expose coverage gaps in existing benchmarks (e.g., FinEval), and identify behavioral clusters.
The framework outperforms neural and graph-based CDMs in accuracy (0.9379), AUC (0.9873), and RMSE (0.2314).
A case study on Claude 3.5 confirmed diagnosed knowledge gaps in tax concepts (F3, F5) with expert agreement of 0.80.
02 From the paper
Abstract
Large Language Models (LLMs) have shown promise for financial applications, yet their suitability for this high-stakes domain remains largely unproven due to inadequacies in existing benchmarks. Existing benchmarks solely rely on score-level evaluation, summarizing performance with a single score that obscures the nuanced understanding of what models truly know and their precise limitations. They also rely on datasets that cover only a narrow subset of financial concepts, while overlooking other essentials for real-world applications. To address these gaps, we introduce FinCDM, the first cognitive diagnosis evaluation framework tailored for financial LLMs, enabling the evaluation of LLMs at the knowledge-skill level, identifying what financial skills and knowledge they have or lack based on their response patterns across skill-tagged tasks, rather than a single aggregated number. We construct CPA-KQA, the first cognitively informed financial evaluation dataset derived from the Certified Public Accountant (CPA) examination, with comprehensive coverage of real-world accounting and financial skills. It is rigorously annotated by domain experts, who author, validate, and annotate questions with high inter-annotator agreement and fine-grained knowledge labels. Our extensive experiments on 30 proprietary, open-source, and domain-specific LLMs show that FinCDM reveals hidden knowledge gaps, identifies under-tested areas such as tax and regulatory reasoning overlooked by traditional benchmarks, and uncovers behavioral clusters among models. FinCDM introduces a new paradigm for financial LLM evaluation by enabling interpretable, skill-aware diagnosis that supports more trustworthy and targeted model development, and all datasets and evaluation scripts will be publicly released to support further research.