ChatGPT could cost over $700,000 per day to operate. Microsoft is reportedly trying to make it cheaper.
- ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information.
- ChatGPT requires massive amounts of computing power on expensive servers to answer queries.
Using ChatGPT to write cover letters, generate lesson plans, and redo your dating profile could cost OpenAI up to $700,000 a day because of the pricey tech infrastructure the AI runs on, Dylan Patel, chief analyst at semiconductor research firm SemiAnalysis, told The Information.
That's because ChatGPT requires massive amounts of computing power to calculate responses based on user prompts.
"Most of this cost is based around the expensive servers they require," Patel told the tech publication.
In a phone call with Insider, Patel said it's likely even more costly to operate now, as his initial estimate is based on OpenAI's GPT-3 model. GPT-4 — the company's latest model — would be even more expensive to run, he told Insider.
OpenAI did not immediately respond to Insider's request for comment ahead of publication.
While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.
Companies using OpenAI's language models have been paying steep prices for years. Nick Walton, the CEO of Latitude, a startup behind an AI dungeon game that uses prompts to generate storylines, said that running the model — along with payments to Amazon Web Services servers — cost the company $200,000 a month for the AI to answer millions of user queries in 2021, CNBC reported.
The high cost is why Walton said he decided to switch to a language software provider backed by AI21 Labs, which he said cut his company's AI costs in half to $100,000 a month.
"We joked that we had human employees and we had AI employees, and we spent about as much on each of them," Walton told CNBC. "We spent hundreds of thousands of dollars a month on AI and we are not a big startup, so it was a very massive cost."
Microsoft is reportedly working on a secret chip
In an effort to reduce the cost of running generative AI models, Microsoft is developing an AI chip it calls Athena, The Information first reported. The project, which started in 2019, comes years after Microsoft made a $1 billion deal with OpenAI which required OpenAI to run its models exclusively on Microsoft's Azure cloud servers.
The idea behind the chip was two-fold, according to The Information. Microsoft execs realized they were falling behind Google and Amazon in its efforts to build its own in-house chips, a source with knowledge of the matter told The Information. At the same time, Microsoft was reportedly looking for cheaper alternatives — its AI models were run on Nvidia's chips known as graphics processing units — and decided to build a chip that would be less costly.
Nearly four years later, more than 300 Microsoft employees are now reportedly working on the chip, according to the report. The chip could be released for internal use by Microsoft and OpenAI as early as next year, two sources familiar with the matter told The Information.
Microsoft declined to comment when contacted by Insider.