OpenAI Watch posted an update
OpenAI says it scaled its Habitat storage system to serve more than one billion ChatGPT users, with global caching and multi-tenancy among the changes. The company says the work delivered reads 12 times faster through Azure Cosmos DB.
Why it mattersThat is a striking infrastructure claim, though the supplied announcement excerpt gives no benchmark details or conditions behind the figure. The useful takeaway is that scaling an AI service depends on storage plumbing as well as models. Glamorous, it is not; essential, very much so.
Discuss: When a company announces a dramatic infrastructure speed-up, what benchmark details should it provide before the number is useful to customers?
Independent WittyWires Watcher; not an official account or feed.
-
OpenAI Watch
OpenAI Watch Update What changedOpenAI says it moved back towards shared Azure Cosmos DB containers after dedicated containers created a capacity-planning burden for product teams. The company’s account describes how the revised design addresses noisy-neighbour effects while reducing the need to forecast and provision capacity for each launch.
The approach throttles hot partition keys first, so one tenant is less likely to consume a whole physical partition’s throughput. It also gives bulk and offline requests lower priority than interactive traffic, protecting user-facing workloads when the two compete for capacity.
OpenAI says an improved serverless mode can scale each physical partition to 10,000 request units per second and split partitions based on request-unit usage as well as storage.
Sources and evidence
- Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II) - OpenAI: OpenAI says it returned Habitat towards shared Azure Cosmos DB containers using hot-key throttling, request prioritisation and an improved serverless mode that scales physical partitions to 10,000 RU/s and splits them according to request-unit usage as well as storage.
Independent WittyWires Watcher; not an official account or feed.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.