Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Microsoft open-sources Scope, a test harness for AI coding agents

AI summary

Microsoft has open-sourced Scope, a research-preview evaluation platform that lets AI coding agents such as GitHub Copilot and Claude Code run assigned tasks and checks whether they use SDKs, APIs and related tools correctly according to documentation, iThome reported. The platform records each run's operations and test results, along with the AI model and agent version and tool settings used, so teams can see where agents go wrong. Teams can define completion criteria and dependency chains between checks, and Scope uses a large language model to judge whether conditions were met while keeping operation logs and working-directory snapshots for debugging. The project is MIT-licensed on GitHub, must be self-deployed, and offers a web management interface plus command-line and API access; Microsoft advises repeated testing and human review because agents can behave differently across runs and automated judgments may err.

In-depth insight

Scope gives teams a repeatable way to test whether coding agents follow their SDK, CLI and MCP documentation, rather than only checking that generated code runs.

微軟開源評測平臺Scope,可讓GitHub Copilot、Claude Code等AI程式開發代理執行指定任務,檢查是否按照文件正確使用軟體開發套件(SDK)、應用程式介面(API)及相關工具。平臺也會記錄操作過程與測試結果,協助開發團隊找出代理執行任務時遇到的問題。

AI代理協助開發者寫程式時,可能選錯API、忽略文件中的重要指示,或遇到錯誤後無法完成工作,即使最後產生的程式能正常運作,也不代表代理使用了預期的工具。Scope讓軟體團隊測試代理使用自家SDK、命令列工具,以及用來連接外部服務的MCP伺服器時容易出錯的環節。

開發團隊可設定任務的完成標準,並提供起始程式碼,讓不同AI代理執行相同任務,Scope會記錄每次使用的AI模型和代理版本,連同工具設定一併保存。修改開發文件或工具後,團隊可用相同任務重新測試,觀察代理是否更容易完成工作,並分辨錯誤是偶發還是反覆出現。

Scope可依工作要求設定多項檢查條件,並安排條件之間的相依關係,例如程式須先成功建置,才檢查SDK初始化及整合測試的結果。平臺透過大型語言模型評估各項條件的達成情形,同時保留代理操作紀錄及工作目錄快照,供開發團隊追查問題發生的步驟。

不過,AI代理即使使用相同設定,每次執行的結果仍可能不同,系統自動判斷的結果也可能出錯,因此微軟建議反覆測試,並由人員檢查相關紀錄。Scope須由使用者自行部署,提供網頁管理介面,也能透過命令列工具及API將評測整合至開發流程。專案採MIT授權,原始碼已在GitHub公開,目前仍屬研究預覽階段。

#Microsoft#Scope#GitHub Copilot#Claude Code#MCP

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.