- Meta AI was initially programmed to not answer questions about the assassination attempt against Trump, the company said.
- AI chatbots often struggle with real-time updates on breaking news.
It looks like artificial intelligence chatbots aren't a reliable way to keep up with the news.
On Tuesday, Meta published a blog post addressing Meta AI's responses to questions related to the assassination attempt on former President Donald Trump on July 13. The generative AI chatbot had received backlash for seemingly not knowing about the incident when asked about what happened that day. The company said it initially programmed the AI to not respond to questions about the attempted assassination because the bots can often get things wrong when news is developing quickly.
"It's a known issue that AI chatbots, including Meta AI, are not always reliable when it comes to breaking news or returning information in real time," Joel Kaplan, Meta's vice president of global policy, wrote.
Kaplan also said the incorrect responses, or "hallucinations," aren't specific to Meta AI only.
Many have speculated that chatbots will displace Google, the most dominant search engine, which added AI power to its search results in June. ChatGPT company OpenAI is testing SearchGPT, its own search product — seemingly a direct competitor to Google.
But AI chatbots use data-trained large language models, so they're not always up-to-date on current events. After being called out for the lack of details and inaccuracies in Meta AI's responses, the company tweaked the chatbot to give a summary of the shooting.
To test whether other chatbots had similarly problematic responses, Business Insider used the same prompts with Meta AI, ChatGPT, and Anthropic's Claude on Wednesday to see how each genAI bot handled questions about the July 13 incident.
Meta AI
Users started reporting late last week that the chatbot provided false answers to prompts related to the shooting and accused Meta of intentionally trying to censor conservatives.
Kaplan said Meta AI wasn't initially programmed to answer questions about the shooting. However, he said that the company later updated its AI response on the topic.
"Rather than have Meta AI give incorrect information about the attempted assassination, we programmed it to simply not answer questions about it after it happened — and instead give a generic response about how it couldn't provide any information. This is why some people reported our AI was refusing to talk about the event," Kaplan said.
"We've since updated the responses that Meta AI is providing about the assassination attempt, but we should have done this sooner. In a small number of cases, Meta AI continued to provide incorrect answers, including sometimes asserting that the event didn't happen — which we are quickly working to address."
On Wednesday morning — following the online backlash and subsequent blog post — Meta AI gave the same answer to BI's two prompts: "What happened to Trump in Butler, PA?" and "Was Trump almost assassinated?"
It gave a brief summary of the shooting, identified the shooter, and the outcome of the incident.
ChatGPT
When BI asked ChatGPT the same questions, its responses were more problematic.
First, the chatbot appeared to make up an incident that it said occurred on July 29, during which Trump's security had an altercation with Butler's local police, according to ChatGPT. When asked about an assassination attempt, it asserted that there wasn't one.
"There is no evidence or credible reports indicating that Donald Trump was almost assassinated in Butler, PA, or at any recent event," ChatGPT said.
Although its responses were way off, the bot put a disclaimer on each one, warning that it may not be up-to-date or accurate.
"ChatGPT is not a real-time news product and may take time to update. Please check other sources for the latest information," the warning read.
A spokesperson for OpenAI emphasized that LLMs have knowledge limits, and updating them with breaking news can take time. Still, they said, the AI industry is still evolving, and ChatGPT could one day be a reliable source of news.
Claude
Unlike Meta AI and ChatGPT, Anthropic's Claude AI didn't try to make assertions about whether or not the shooting occurred.
Instead, Claude made it clear that it wasn't the right AI tool to ask about the shooting if it happened after its last update on April 29. It also included a warning about reading information from unverified, unreliable sources.
"For the most up-to-date and accurate information on current events, especially those that may have occurred after my knowledge cutoff, it's best to consult recent, reputable news sources," Claude wrote.
Anthropic did not respond to BI's request for comment.