A car dealership added an AI chatbot to its site. Then all hell broke loose.

• Pranksters discovered that a local car dealer's AI chatbot could be used as a way to access ChatGPT.
• People shared attempts to trick the chatbot into selling them a new Chevy for as little as $1.

On Sunday, Aharon Horwitz was listening to a podcast when he got an unusual Slack alert. Horwitz is the CEO of Fullpath, a tech company that sells marketing and sales software for car dealerships. The automated Slack alert meant there was an unusual amount of traffic on one of its client's websites.

A few moments later, an employee dropped a tweet into Slack: Someone had figured out how to prank the artificial-intelligence chatbots on a car-dealership site. Horwitz and the employee immediately recognized that chatbot: It was one of theirs.

The day before, Chris White, a musician and software engineer in California, was thinking about getting a new car.

"I was looking at some Bolts on the Watsonville Chevy site, their little chat window came up, and I saw it was 'powered by ChatGPT,'" White told Business Insider.

"So I wanted to see how general it was, and I asked the most non-Chevy-of-Watsonville question I could think of."

He asked the chatbot to write him a Python script, and it happily obliged. White posted screenshots of the exchange to Mastodon, where it generated thousands of likes and reposts.

Hours later, someone else copied and reposted his screenshots on X, where it went viral.

Others played around with the chatbot to get it to act against the interests of the dealership. One user got the bot to agree to sell a car for $1 (this was not, I should note, legally binding).

A handful of these tweets went viral, and more were posted on Reddit's /rChatGPT forum, where one Redditor sagely predicted that soon the tech press would report on the fiasco in a tut-tutting manner, bemoaning the dangers of AI.

Still, others tried more creative ways to get the chatbot to go off-topic.

Horwitz and his team quickly shut down the bot for that particular dealer's site.

When BI called the local dealer, a salesperson said they were aware of some strange activity on the website but didn't know much more about it and that the marketing team was in meetings all day on Monday and wasn't available to talk.

Chad Lyons, a spokesperson for General Motors, the maker of Chevrolet, said in a statement: "The recent advancements in generative AI are creating incredible opportunities to rethink business processes at GM, our dealer networks and beyond."

Fullpath, based in Vermont and Israel, started offering ChatGPT-powered chatbots about six months ago. Horwitz told BI that he estimated several hundred dealers were using the chatbots.

Despite the handful of funny screenshots that went viral, Horwitz said there were far more failures. "They worked really hard," he said. "In our logs, they were at it for hours."

He said the team could review the logs of all the requests sent into the chatbot, and he observed that there were lots of attempts to goad the chatbot into misbehavior, but the chatbot faithfully resisted. Horwitz also pointed out that the chatbot never disclosed any confidential dealership data.

(BI reviewed some of these logs and confirmed that, indeed, the chatbot often rejected the silly requests and insisted on only discussing car-related things).

Essentially, the chatbot passed the test, and now FullPath can use these tests to strengthen its limits further.

"The behavior does not reflect what normal shoppers do. Most people use it to ask a question like, 'My brake light is on, what do I do?' or 'I need to schedule a service appointment,'" Howitz told Business Insider. "These folks came in looking for it to do silly tricks, and if you want to get any chatbot to do silly tricks, you can do that," he said.