Scientists found more than 1,000 AI spam bots trying to scam people and steal their social media profiles — and regulators can't keep up
- Research by Indiana University academics details how bad actors are taking advantage of AI.
- The study found over a thousand of bot accounts promoting scams on X, formerly Twitter.
People keep finding new uses for ChatGPT. One of the latest is flooding social media with spam bots and AI-generated content that could further degrade the quality of information on the internet.
A new study shared last month by researchers at Indiana University's Observatory on Social Media details how malicious actors are taking advantage of OpenAI's chatbot ChatGPT, which became the fastest-growing consumer AI application ever this February.
The research, conducted by Kai-Cheng Yang, a computational social science researcher, and Filippo Menczer, a computer-science professor, found that ChatGPT's ability to generate authoritative-looking text is being used to run "botnets" on X, formerly Twitter.
What are botnets and why are they bad?
Botnets are networks of hundreds of harmful bots and spam campaigns on social media that can go undetected by current anti-spam filters. They are deployed for many reasons — in this case, the botnets are promoting fraudulent cryptocurrencies and NFTs.
The bot accounts attempt to convince people to invest in fake cryptocurrencies and may even steal from their existing crypto wallets.
The network Yang and Menczer discovered on X housed over 1,000 active bots. The bots followed and responded to each other's posts with ChatGPT outputs, often publishing selfies stolen from human profiles to build fake personas.
The rise of social media gave bad actors a cheap way to reach a large audience and monetize false or misleading content, Menczer said. New AI tools "further lower the cost to generate false but credible content at scale, defeating the already weak moderation defenses of social-media platforms," he said.
In the past few years, social-media bots — accounts that are wholly or partly controlled by software — have been routinely deployed to amplify misinformation about events, from elections to public-health crises such as COVID.
These social-media bots have been easy to pick out because of their robotic behavior and unconvincing fake personas.
But generative-AI tools like ChatGPT produce humanlike text and media in seconds. "The advancement of AI tools will distort the idea of online information permanently," Yang told Insider.
The AI bots in the network uncovered by the researchers mainly posted about fraudulent crypto and NFT campaigns and promoted suspicious websites on similar topics, which themselves were likely written with ChatGPT, the survey says.
Entire websites can be made up of AI-generated misinformation
Beyond social media, ChatGPT-like tools have generated spammy generic news websites, many of which publish blatant falsehoods.
NewsGuard, a private company that rates news and information websites' reliability, has so far found more than 400 such AI-generated websites as part of its ongoing audits since April.
These websites generate advertising revenue from automated ad tech that purchases ad space on websites regardless of their quality or nature.
We can still detect AI-generated spam — for now
Both NewsGuard and the paper's researchers were separately able to unearth AI-generated spam content using an obvious tell that chatbots currently have.
When ChatGPT can't produce an answer because the prompt is against OpenAI's policies or concerns private information, it generates a canned response such as, "I'm sorry, but I cannot comply with this request."
Researchers look for when these responses slip out in an automated bot's content, whether on a webpage or in a tweet. They use the phrases to uncover parts of a broader bot campaign and from there discover the rest of the spam network.
However, experts fear that as chatbots get better at mimicking humans, such telltale signs will disappear and make it much harder to track AI-generated content.
Wei Xu, a computer-science professor at the Georgia Institute of Technology, told Insider that it will become harder to detect and filter AI-generated content while more malicious users exploit it, creating a vicious cycle of AI-generated content that's increasingly hard to detect.
Xu's concerns could be a reality soon. Europol, the EU's law-enforcement agency, predicts 90% of internet content will be AI-generated by 2026.
Without regulation, as long as there are more incentives and low costs for creating AI-generated content, bad actors will always be far ahead of those trying to shut it down, added Xu.
"It's similar to plastic bottled water. We know it is a disaster for the environment, but as long as it is cheap without a big penalty, it will widely exist," said Xu.
Regulators are sprinting to catch up with the deluge of spam
The AI content detectors available, such as ZeroGPT and the OpenAI AI Text Classifier, are unreliable and often can't accurately tell the difference between human-generated and AI-generated content, a recent study by European academics found. OpenAI's detection service had such a low accuracy rate that the firm decided to shut it down.
In July, the Biden administration announced that big players in AI, including Google, Microsoft, and OpenAI, had given commitments to the White House about engineering guardrails to reduce AI risk.
One such measure was tagging AI-generated content with a hidden label to help people distinguish it from content made by humans, per the White House.
But Menczer, the coauthor of the Indiana University research, said these safeguards would likely not stop bad actors who ignored them.
Yang said that tracking suspects' social-media activity patterns, whether they have a history of spreading false claims and how diverse in language and content their previous posts are, is a more reliable way to identify bots.
People will have to be even more skeptical of anything they encounter online, said Yang, adding that generative AI will "have a profound impact on the information ecosystem."