Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering

ABSTRACT

This paper systematically analyzes how agentic AI systems are evaluated in software engineering contexts. It reviews 18 recent papers from top venues such as ICSE and FSE to identify evaluation patterns and shortcomings. The authors highlight the lack of reproducibility, inconsistent metrics, and weak benchmarking practices. They propose a structured framework for evaluation that emphasizes explainability and repeatability. The work aims to establish more rigorous standards for validating agentic systems.