OpenAI 发布了一份详细审计报告，指出广泛使用的编程基准测试 SWE-Bench Pro 存在显著问题 [1]。该报告揭示约 30% 的测试任务存在问题，导致无法准确反映模型的真实能力 [1]。主要缺陷类别包括测试标准过于严格、提示词定义不清以及误导性提示等 [1]。此外，审计还发现部分任务的测试覆盖率较低 [1]。
此次评估工作通过自动化管道和五名软件工程师的人工复核共同完成 [1]。基于上述发现，OpenAI 已撤回此前推荐采用 SWE-Bench Pro 的建议 [1]。报告强调，高质量的评估数据对于模型部署和安全决策至关重要 [1]。
OpenAI has withdrawn its recommendation to use the widely adopted coding benchmark test SWE-Bench Pro following a detailed audit that identified serious defects in approximately 30% of its tasks [1]. The comprehensive review, conducted using automated pipelines and manual verification by five software engineers, revealed issues such as overly strict tests, unclear prompt definitions, low coverage, and misleading prompts [1]. These flaws mean the benchmark fails to accurately reflect a model's true capabilities [1]. Emphasizing that data quality is critical for model deployment and safety decisions, OpenAI stated it can no longer endorse SWE-Bench Pro based on these findings [1].