Anthropic adds build-eval and hillclimb tuning commands to Claude Code
On September 28 Anthropic published a tuning guide introducing two Claude Code entry points, /claude-api build-eval and /claude-api hillclimb (Claude Code v2.1.259 or later). The first turns "what counts as a good answer" into repeatable test cases and scoring rules; the second lets Claude propose one change per round — to prompts, skill files, tool descriptions or model config — keeping it only if the evaluation improves and reverting otherwise, with a hidden held-out set used to check for overfitting. Anthropic says an internal customer-support evaluation improved decision accuracy from 78.6% to 90.5% on 14 held-out tickets while cutting model call cost to roughly one fifth.
It turns prompt tuning from manual trial and error into an automatable, revertible loop, which matters most to solo developers paying their own API bills.
设想一下:
你用Claude Code手搓了一个小应用,顺便给它接了个AI客服。
本想让它自动解答用户「怎么登录」「会员服务有啥内容」之类的基础问题,好让自己当个甩手掌柜。
结果客服确实能聊天了,但你却沦为了它的全职陪练:
回答不准,赶紧补一句提示词;解释太啰嗦,再加一条限制。
更让人崩溃的是,每改一句提示词,你都得把常见问题重新问一遍,生怕它顾此失彼,刚修好一个问题,却把原本答对的又改坏了。