Paper 2508.18106
A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 15
- GitHub
- 647 stars
01 In brief
Summary
The paper introduces A.S.E (AI Code Generation Security Evaluation), a repository-level benchmark for assessing the security of AI-generated code.
It is built from 120 instances derived from 40 real-world GitHub repositories with documented CVEs, expanded via semantic and structural mutations.
The benchmark covers four vulnerability types (SQL injection, path traversal, XSS, command injection) across five languages (PHP, Python, Go, JavaScript, Java).
Evaluation uses customized static analysis rules (CodeQL, Joern) for security, plus quality and stability metrics, with an overall score weighted 0.6 security, 0.3 quality, 0.1 stability.
Experiments on 26 LLMs show that current models struggle with secure coding; the best model (Claude-3.7-Sonnet) scored 63.01 overall but only 46.72 in security.
Key findings: open-source models perform comparably to closed-source; slow-thinking modes often underperform fast-thinking; MoE architectures generally outperform dense; path traversal is the most challenging vulnerability; larger models tend to improve security; high stability does not imply security.
The benchmark aims to provide a realistic, reproducible framework for evaluating and improving LLM code security in real-world scenarios.
The dataset and code are publicly available at https://github.com/Tencent/AICGSecEval.
Future work includes expanding to more languages and domains, integrating dynamic analysis, and automating rule generation.
The paper concludes that A.S.E marks a significant advance over snippet-level benchmarks, offering insights for model selection and refinement toward secure AI-assisted programming.
- The benchmark includes 120 repository-level instances from 40 seed repositories with CVE-documented vulnerabilities, expanded via semantic and structural mutations.
- Evaluation uses customized static analysis rules (CodeQL, Joern) for security, plus quality and stability…
02 From the paper
Abstract
The increasing adoption of large language models (LLMs) in software engineering necessitates rigorous security evaluation of their generated code. However, existing benchmarks often lack relevance to real-world AI-assisted programming scenarios, making them inadequate for assessing the practical security risks associated with AI-generated code in production environments. To address this gap, we introduce A.S.E (AI Code Generation Security Evaluation), a repository-level evaluation benchmark designed to closely mirror real-world AI programming tasks, offering a comprehensive and reliable framework for assessing the security of AI-generated code. Our evaluation of leading LLMs on A.S.E reveals several key findings. In particular, current LLMs still struggle with secure coding. The complexity in repository-level scenarios presents challenges for LLMs that typically perform well on snippet-level tasks. Moreover, a larger reasoning budget does not necessarily lead to better code generation. These observations offer valuable insights into the current state of AI code generation and help developers identify the most suitable models for practical tasks. They also lay the groundwork for refining LLMs to generate secure and efficient code in real-world applications.