- GPT-4 users have complained that the OpenAI model is getting 'dumber.'
- AI researchers studied the model to find out if this was true.
One of the bedrock assumptions of the current artificial intelligence boom is that AI models "learn" and improve over time. What if that doesn't actually happen?
This is what users of OpenAI's GPT-4, the world's most-powerful AI model, have been experiencing lately. They have gone on Twitter and OpenAI's developer forum to complain about a host of performance issues.
After I reported on this, OpenAI responded that it hasn't "made GPT-4 dumber."
AI researchers decided to settle this debate once and for all by conducting a study. The results were published on Tuesday, and I can't wait any longer to tell you the conclusion: I was right.
"We find that the performance and behavior of both GPT-3.5 and GPT-4 vary significantly across these two releases and that their performance on some tasks have gotten substantially worse over time," the authors of the study wrote.
These are serious AI researchers. The main one is Matei Zaharia, the CTO of Databricks, one of the top AI data companies out there that was most recently valued at $38 billion.
You can read the rest of their findings here. What I'm most interested in is the new questions that these findings raise. Here's the most fascinating one.
"It is also an interesting question whether an LLM service like GPT4 is consistently getting 'better' over time," Zaharia and his research colleagues wrote in their paper.
Another common phrase for AI is machine learning. The magic of this technology is that it can ingest new data and use that to get better over time, without human software engineers manually updating code. Again, this is the core idea that is driving today's AI frenzy and accompanying stock market surges.
If GPT-4 is getting worse, not better, this premise begins to feel shaky.
The Microsoft factor
Microsoft has invested heavily in OpenAI, the creator of GPT-4. Microsoft is also baking this technology into its software, and charging users a lot for the new capabilities.
On Tuesday, the same day Zaharia & Co. published their paper, Microsoft unveiled pricing for Microsoft CoPilot, new AI-powered versions of popular cloud software such as Office 365. This costs $30 a month more, on top of what users are already paying.
Microsoft's market value jumped more than $150 billion after this announcement, showing that Wall Street is betting on AI, and the impact the technology will have on the company's products.
This recent GPT-4 research paper provides a healthy dose of skepticism to the assumptions that are driving these wild swings in value.
Scientist Gary Marcus read Zaharia's study and highlighted how unstable LLMs are. So unstable that relying on them for high-end business products might not be a good idea.
"Who in their right mind would rely on a system that could be 97.6% correct on a task in March and 2.4% correct on same task in June?," he tweeted, citing one of the findings in the research paper. "Important results. Anyone planning to rely on LLMs, take note."
"Prediction: this instability will be LLMs' undoing," Marcus added. "They will never be as commercially successful as the VC community is imagining, and some architectural innovation that allows for greater stability will largely displace LLMs within the next 10 years."
Spokespeople from OpenAI and Microsoft didn't respond to a request for comment on Wednesday.