创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

OpenAI 在 GPT-5.6 回复上部署激活分类器

AI 总结

OpenAI 表示,自 GPT-5.6 起,已在所有助手回复上运行激活分类器(又称探针)。这种轻量级模型读取大模型内部残差流激活,用于识别失配或其他不良活动。探针支持流式决策,可在违反政策的生成过程中中止回复。OpenAI 称,与体积大得多的文本分类器相比,探针更抗越狱且运行效率更高。其架构采用 MLP、QV-attention 和指数移动平均,通常使用最多100万条由最强推理模型标注的合成及去标识消息进行训练。OpenAI 表示,每个新的预训练基座模型都需要重新训练探针,而且当模型权重偏离训练检查点时,探针性能可能下降。相关性能数据来自内部实验,未必能代表实际部署的分类器。

深度见解

OpenAI 表示,探针已成为其安全与对齐战略的重要组成部分,并已部署用于最强模型的内部和外部流量。它们提供可在模型生成回复期间介入的白盒监控,与文本分类器、基于模型的审查、自动调查和人工响应形成互补。

Introduction

At OpenAI, we train models to behave in a safe and aligned way. Despite recent progress in this direction [OpenAI, 2026b], there remain cases when adversarial pressure, rare circumstances, or training failures can degrade safety and alignment. For this reason, we adopt a ‘defense in depth’ approach: we deploy models with additional monitors that aim to detect such cases and stop further model action. Specifically, we use text classifiers that run on user prompts and assistant completions for this purpose. Starting from GPT-5.6 [OpenAI, 2026a], we have added another layer of defense to this stack: activation classifiers that run on all assistant completions.

Activation classifiers, or probes for short, are lightweight ML models that use an LLM’s own internal activations — its “neural patterns” — to classify model activity as being misaligned or otherwise undesirable (e.g., user activity having harmful intent). Probes offer multiple advantages compared to external text-based classifiers: they enable streaming decisions which can block a policy-violating model turn midway; are more robust to jailbreaks than significantly bigger text classifiers; and are significantly more efficient to run. Furthermore, probes’ accuracy generally scales in tandem with model capability: probes trained on the activations of more capable models are themselves more capable classifiers.

#OpenAI#GPT-5.6#activation classifiers

原文约 8 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。