This research paper assesses the current state of AI agent benchmarking, highlighting critical flaws hindering real-world applicability. The authors identify shortcomings in existing benchmarks, including a narrow focus on accuracy without considering cost, conflation of model and downstream developer needs, inadequate holdout sets leading to overfitting, and a lack of standardization impacting reproducibility. They propose a framework to address these issues, advocating for cost-controlled evaluations, joint optimization of accuracy and cost, distinct benchmarking for model and downstream developers, and standardized evaluation practices to foster the development of truly useful AI agents. Their analysis uses case studies on several prominent benchmarks to illustrate the identified problems and proposed solutions. The ultimate goal is to improve the rigor and reliability of AI agent evaluation.