Governing What an Agent Learns
In an opinion article originally published on Tim O’Brien’s Medium blog and republished with his permission, O’Brien argues that agent security must govern not only identities and credentials but also the concepts, relationships, and evidence an agent can accept. He cites OpenAI evaluations of the Hugging Face incident and METR’s investigation, proposing “semantic and epistemic governance” to prevent learning from silently expanding an agent’s effective authority.
The article argues that access controls alone are insufficient because changes to an agent’s world model can change what it is effectively authorized to do, even when its underlying permissions remain unchanged.
This source provides an excerpt. Continue reading on the original site.