Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Contract-Summary Service Uses Staged Rollout to Migrate from M1 to M2

AI summary

The contract-summary service team reported migrating its main model from M1 to M2 through 10%, 50%, and 100% traffic stages over nine days. During the rollout, M2 produced a 24% downstream parsing failure rate because of a changed date format, and its peak p99 latency exceeded seven seconds when concurrent-request limits caused queuing; the team fixed both issues, increased concurrency capacity, and added automatic fallback from M2 to M1 after HTTP 429 responses. The team said monthly model costs fell from 3,300 yuan to 2,000 yuan after the migration, while the 41 affected contracts were automatically retried.

In-depth insight

The report illustrates how staged traffic, downstream parsing checks, tail-latency monitoring, and automatic fallback can reduce the operational risks of switching production LLM models, although the measurements come from one first-person source.

十月中旬,我们要把合同摘要服务的主模型从 M1 换成 M2。M2 便宜 40%,离线评测准确率还高了 1.5 个点。

换模型这件事,团队之前做过一次,方式是周五晚上直接改配置。周一早上法务部来了三个人,说摘要里的金额单位全错了——M2 在某类合同上会把「万元」写成「元」。那次回滚花了半天,返工了两百多份摘要。

这一次,我们按发布普通服务的方式做了灰度:10%、50%、100% 三档,每档跑够时间、看够指标再往下走。前后九天,期间发现并修掉了两个离线评测没测出来的问题。

一、为什么模型切换必须灰度

离线评测再全,也只是一个固定的测试集。线上有三样东西是测试集没有的:

  • 真实的输入分布。 我们的测试集 600 份合同,线上一周就有 4,000 多份,类型更杂。
  • 真实的负载。 新模型在高并发下的延迟、限流表现,离线测不出来。
  • 下游的反应。 摘要格式变一点点,下游的解析脚本、人的阅读习惯都可能出问题。

#Contract-summary service#M1#M2#OpenAI-compatible API

Original · 4 min readRead the original
Group chat

No comments yet. Start the conversation.