From transphobia to Ted Kaczynski: How TikTok's algorithm enables far-right self-radicalization

• Social media's role in radicalizing extremists has drastically increased over the last several years.
• Some far-right TikTokers employ a meme-like format in their content to dodge content moderation.

A recent study from left-leaning nonprofit watchdog Media Matters found that if a TikTok user solely interacts with transphobic content and creators, the social networking app's algorithm will gradually begin to populate their "For You" page with white supremacist, antisemitic, and far-right videos, as well as calls for violence.

Launched in 2016 by Chinese tech startup ByteDance, TikTok saw a surge in user growth throughout the COVID-19 pandemic and acquired 1 billion users across the world in five years, many of which are teenagers and young adults.

In 2020, the app classified more than a third of its daily users as 14 years old or younger, The New York Times reported. A former TikTok employee noted that videos and accounts made by children who appeared younger than the app's minimum age requirement of 13 were allowed to remain online for weeks, The Times reported, raising questions about measures taken by the platform to protect its users from misinformation, hate speech, and even violent content.

In the experiment, researchers from Media Matters created a dummy account, interacted with anti-trans content, and then evaluated the first 400 videos fed to the account. Some of the videos were removed before they could be analyzed, while others were sponsored advertisements unrelated to the study. Of the remaining 360 videos, researchers found:

• 103 contained anti-trans and/or homophobic narratives
• 42 were misogynistic
• 29 contained racist narratives or white supremacist messaging
• 14 endorsed violence

"While nearly 400 may sound like a large number of videos, if a user watches videos for an average of 20 seconds each, they could consume 400 videos in just over two hours. A user could feasibly download the app at breakfast and be fed overtly white supremacist and neo-Nazi content before lunch," the study concluded.

The far-right movement has historically embraced anti-trans rhetoric, and right-wing recruiters know that "softer ideas" like transphobia can be used to introduce newcomers to more extreme beliefs, Melody Devries, a Ryerson University PhD candidate who studies far-right recruitment and mobilization, told Insider.

"The videos that start people down the rabbit hole are things that are, unfortunately, prejudices that are not considered that extreme in society," Devries said.

Unforeseen consequences of the digital age

Before the rise of social media, individuals predominately formed their beliefs through real-world networks of relationships with parents, family members, and friends. Social media platforms, however, gave individuals the ability to expand these social networks by building communities in online environments.

The rapid expansion and evolution of digital spaces have transposed extremist content and ideologies from niche corners of the Internet to platforms that are frequented by billions of users.

"Now, Facebook, Instagram, Twitter, all of our communications platforms that we think of as sort of the most easy to use can be the starting point [of radicalization]. And then a person can move into more layered applications that are harder to penetrate," Thomas Holt, a professor and director of the Michigan State University school of criminal justice, told Insider.

According to the National Consortium for the Study of Terrorism and Responses to Terrorism (NCSTRT), social media's role in extremism has drastically increased over the last several years.

In 2012, only 48% of extremists listed in Profiles of Individual Radicalization in the United States (PIRUS), an NCSTRT dataset, said that social media played a role in their radicalization. By 2016, 86.75% of PIRUS-listed extremists used social media in their radicalization process, according to an NCSTRT research brief.

Holt mentioned Facebook, Instagram, and Twitter, all of which are either a decade or more than a decade old. But in the past five years, TikTok has become one of the fastest-growing social media platforms of all time, known for its powerful algorithm that serves up highly tailored videos.

It has more than 100 million daily users in the US, according to CNBC, and has recently become the focus of more scrutiny surrounding its algorithm, which Black and LGBTQ creators have said censors their voices and perpetuates targeted harassment.

How Big Tech streamlined self-radicalization

Because social media profit models rely heavily on user engagement, most companies choose to take the proverbial "middle road" when moderating content in order to avoid accusations of censorship from either side of the political spectrum and, ultimately, damaging their bottom line, according to Devries.

"The fact that those platforms are totally fine with that, because that's their profit motive, and that's their design, I think is a problem and obviously contributes to how right-wing communication is transformed," Devries told Insider.

Subpar content moderation has allowed implicit extremist content to largely remain on platforms, sometimes reaching up to millions of users. Many of the extremist TikTok videos analyzed by Media Matters employed a "memetic format," or utilized the platform's unique combination of audio, video, and text to evade violating community guidelines.

For example, several of the videos populated to the FYP of the researchers' dummy account used a sound called "Teddy," which quotes the first line of "Unabomber" Ted Kaczynski's manifesto: "The industrial revolution and its consequences have been a disaster for the human race."

The sound, which has been used in more than 1,200 videos, has become popular on right-wing TikTok.

"In the videos we reviewed, it was frequently paired with montages of screenshots of LGBTQ people livestreaming on TikTok. These videos not only use audio that pays homage to a terrorist, but they also promote the harassment of LGBTQ TikTok users," Media Matters researchers wrote.

While the "Teddy" sound might not explicitly violate the platform's guidelines, videos using it frequently communicate hateful, dangerous, and even violent messages when taking into consideration the full piece of content, including other components like visuals and text.

The Internet has become a critical resource for extremist groups and loopholes around community guidelines allow them to promote their ideologies to larger audiences in subtle and convincing ways, according to Holt's research in Deviant Behavior.

"Whether [viewers] initially believe it or not, over time, these interactions with that content slowly kind of chips away at their ideological belief system and builds up a new one that's based around the ideas presented in this content," Devries said.

Stopping online interactions with extremist content

The impacts of disinformation, misinformation, and radicalization propagated by social media — insurrections, national security issues, and even genocide — have been felt throughout the globe for years.

"It's not just the US. Every country is being impacted in some way by the use of and misuse of social media platforms for disinformation, misinformation, or radicalization. There's an inherent need for better regulation, better management of platforms, and, to the extent that it can be provided, transparency around reporting and removal," Holt stold Insider.

However, Devries added, it's not about presenting counter-facts; the interactions themselves need to be stopped.

In her ethnographic analysis of far-right Facebook spaces, Devries has seen the platform add infographics warning that a post contains misinformation in an attempt to moderate content, an approach that she sees as counterintuitive.

"Not only are folks interacting with the false content itself, they're interacting with the fact that Facebook has tried to censor it. So that infographic itself becomes another piece of content that they can interact with and pull into their ideology," Devries told Insider.

When asked for comment, a Facebook spokesperson maintained that the company tries to give the maximum number of people a positive experience on Facebook and takes steps to keep people safe, including allocating $5 billion over the next fiscal year for safety and security.

When a Wall Street Journal investigation exposed how Facebook proliferated real-world harms by failing to moderate hate speech and misinformation, the company acknowledged in a September 2021 blog that it "didn't address safety and security challenges early enough in the product development process."

Rather than pursuing reactive solutions like content moderation, Holt proposes that social media companies mitigate online extremism on their platforms by implementing solutions like those used to remove child sexual exploitation content.

Tools like Microsoft's PhotoDNA are used to stop online recirculation of child sexual exploitation content by creating a "hash," which functions as a sort of digital fingerprint that can be compared against a database of illegal images compiled by watchdog organizations and companies, according to Microsoft.

If this kind of technology was overlayed against social media, Holt said it could be automated to take down content associated with extremism or violent ideologies.

Still, this solution relies on social media platforms making internal changes. In the meantime, Holt advocates for better public education on these platforms and how to use them responsibly.

"Yeah, the cat is out of the bag. I don't know how we roll it back and minimize our use of social media. So instead, it seems like we have to get better at educating the public, particularly young people, to understand, 'Here's how the platforms work, here's what may be there,'" Holt told Insider.

Ultimately, both Holt and Devries agree that more research is needed to analyze how newer platforms like TikTok are used to mobilize extremists and radicalize newcomers into their ideology, as well as discover solutions to minimize and counteract the fallout.

TikTok told Insider that all of the content cited in the Media Matters study was removed from the platform for violating its hateful behavior policy. Additionally, the company outlined anti-abuse efforts that it has built into its product, including its addition of new controls that allow users to delete or report multiple comments at once and block accounts in bulk.

Still, Eric Han, head of US safety for TikTok, said in an October statement that harassment and hate speech are "highly nuanced and contextual issues that can be challenging to detect and moderate correctly every time."

"To help maintain a safe and supportive environment for our community, and teens in particular, we work every day to learn, adapt, and strengthen our policies and practices," said TikTok's Q2 2021 transparency report.