It's not just Google — closer inspection reveals Bing's AI also flubbed the facts in its big reveal

• Dmitri Brereton said Bing's new AI chatbot "got some answers completely wrong" during its demo.
• The AI and search engine researcher said that the tech is "definitely not ready for launch"

It's not just Google's AI chatbot Bard that's messing up.

Earlier this week, Dmitri Brereton, an AI and search engine researcher, pointed out that Bing's new AI chatbot made a number of errors during its demo — and said that it's "definitely not ready for launch."

As part of Microsoft's unveiling of the new tech, Bing's AI was asked to list the pros and cons of the three best-selling pet vacuums. The chatbot made a list for the "Bissel Pet Hair Eraser Handheld Vacuum," with cons including its noise level and short cord.

The problem? When Brereton compared what the chatbot said to the article it linked to as a source, he noticed that that review didn't mention the vacuum's noisiness. Plus, it's cordless.

"I hope Bing AI enjoys being sued for libel," he wrote.

In another example, Bing was prompted to create a five-day itinerary for a trip to Mexico City and was asked for nightlife suggestions. It responded with a descriptive list of bars and clubs.

After cross-referencing the bot's answers with his own research, Brereton found some of the descriptions the bot spit out were wrong. In one case, the bot recommended going to a bar's website to book a reservation and check out its menu — but neither reservations nor the menu are available on the bar's site. For two other bars, the bot said there were no reviews online. There are, in fact, hundreds for one and thousands for the other.

The most egregious mistake Bing made during its chatbot demo, Brereton said, was fabricating numbers after it was asked about the key takeaways from Gap's Q3 2022 financial report.

The technology mislabeled some of the numbers, like the adjusted gross margin, and other values, like diluted earnings per share, were "completely made up."

Bing's summary of Lululemon's Q3 2022 earnings report also contained errors.

"I am shocked that the Bing team created this pre-recorded demo filled with inaccurate information, and confidently presented it to the world as if it were good," Brereton concluded.

"We're aware of this report and have analyzed its findings in our efforts to improve this experience," a Microsoft spokesperson told Insider. "We recognize that there is still work to be done and are expecting that the system may make mistakes during this preview period, which is why the feedback is critical so we can learn and help the models get better."

Brereton isn't the only user who has caught Bing's mistakes.

A writer at The Verge asked Bing to list the movies showing in a particular London neighborhood. The bot answered with "Spider-Man: No Way Home" and "The Matrix Resurrections" — two old movies that were found to not actually be playing.

The AI arms race may lead to the spread of misinformation

Brereton's observations come as Big Tech companies like Google and Microsoft enter an AI arms race.

Google and Microsoft both demoed new chatbots to the public last week. Though Microsoft was first to release its new technology, Google plans to release its chatbot, Bard, in just a few weeks. (An ad for Google's new tech showed the company's chatbot making an error of its own.)

Even before Bing's errors were revealed, Gary Marcus, a former professor of neural science at New York University, noted in a blog post how the two companies were being pit against each other. Microsoft's demo "was presented as a revolution," while Google's demo was presented as a "disaster," he wrote.

The intense pressure to roll out these tools has led some in tech — including John Hennessy, the chairman of Alphabet — to say that they are being released prematurely. Google employees said the announcement of its tech was "botched" and "rushed."

The hasty releases may come at the cost of spreading misinformation, especially if people expect quick, accurate answers on search engines.

While Brereton told Insider that generative AI search engines like the new Bing can be "quite transformative," he noted that releasing it prematurely "could lead to big problems."

"It's more dangerous with a search engine that millions of people trust for correct answers starts confidently dishing out garbage," Brereton said. "Bing tries to solve this by warning people the answers are inaccurate. But they know and we know that no one is going to listen to that."