创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

OpenAI披露Habitat存储扩展细节

AI 总结

OpenAI称,Habitat已从客户端库和数据库扩展为覆盖40个地区、服务超过10亿用户并承载逾500PB数据的全球分布式存储平台。公司介绍了基于Azure Cosmos DB的共管容器设计、与Microsoft Azure在区域故障后开展的联合运维,以及仍在开发的EB级RocksDB Cloud与Raft复制系统。

深度见解

OpenAI披露的工程细节具体呈现了支撑ChatGPT等大规模产品的基础设施架构与运维方式。

All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.

This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.

In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

#OpenAI#Habitat#Azure Cosmos DB#ChatGPT#Microsoft Azure#RocksDB Cloud#Rockset#Raft

原文约 17 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。