Databricks 在其拥有数百万行代码的庞大库上进行了内部基准测试，旨在评估不同人工智能（AI）编码代理在真实工程任务中的表现及成本效益 [1]。研究结果表明，单纯依据 Token 成本进行决策具有误导性，必须通过任务级基准测试来优化模型选择策略 [1]。
测试结果将模型划分为三个能力层级：最智能的模型虽然能有效解决复杂问题但成本高昂；而中等或低智能模型在处理常见日常任务时则表现出更高的效率与更低的成本 [1]。具体而言，GLM 模型在输出质量上与 Opus 4.8 统计持平，其每任务成本为 1.28 美元（Opus 4.8 为 1.94 美元），因此被确定为日常开发的主力驱动模型 [1]。相比之下，Sonnet 5 模型的 Token 成本约为 Opus 4.8 的六分之一，但由于其推理效率较低导致单次任务总成本高达 2.09 美元（高于 Opus 的 1.94 美元），且任务完成率低出约 6 个百分点 [1]。
此外，不同的执行框架（Harness）对同一模型的成本影响显著。其中，Pi Harness 因具备更优的上下文管理能力，减少了任务运行次数从而有效降低了整体成本 [1]。
Databricks conducted an internal benchmarking study evaluating the performance and cost efficiency of various AI coding agents across its multi-million line codebase [1]. The research aimed to determine optimal model selection strategies for real-world engineering tasks, moving beyond simple token-based metrics which can be misleading [1]. Results categorized models into three capability tiers: highly intelligent models capable of solving complex problems but at a high cost; medium and low-intelligence models that are efficient and affordable for common tasks [1].
Based on the study's findings, GLM was identified as the primary driver model for daily development due to its statistical parity in quality with Opus 4.8 while offering superior value [1]. Specifically, GLM achieved a cost of $1.28 per task compared to $1.94 per task for Opus 4.8 [1]. In contrast, Sonnet 5 demonstrated approximately 1.7 times lower token costs than Opus 4.8; however, its low inference efficiency resulted in a higher single-task cost of $2.09 and a completion rate six percentage points lower than that of Opus .
The study also highlighted the significant impact of different harnesses on model performance and expenses [1]. The Pi Harness proved particularly effective by optimizing context management, which reduced the number of tasks required to run and consequently lowered overall costs .