Facebook reportedly measures 'violence and incitement trends' — and it spiked by 45% during the week of the election

• Facebook measures how posts on its platform could lead to real-world violence through an internal tool that tracks "violence and incitement trends," BuzzFeed News reported Thursday.
• The metric, which analyzes potentially dangerous search terms and hashtags, has spiked 45% during the week of the election, according to BuzzFeed News.
• Facebook has taken actions against a number of accounts, posts, and groups for spreading election misinformation and calling for violence, but some things are still slipping through the cracks.
• Facebook has also repeatedly been accused of helping fuel real-world violence around the world — from ethnic cleanings in Myanmar to shootings of anti-police-brutality protesters in Kenosha, Wisconsin.

For years, misinformation researchers, civil and human rights groups, nongovernmental organizations, governments, journalists, and everyday users have tried to alert Facebook to instances of its platform being used to incite real-world violence.

On Thursday, BuzzFeed News reported that Facebook is also tracking those instances through an internal metric called "violence and incitement trends," and that, as President Donald Trump's close allies openly call for violence over the election results, those trends are surging.

The metric analyzes hashtags and search terms used across Facebook over the past five days to churn out a score that estimates the likelihood of potential real-world violence — and that scored climbed 45%, from 400 to 580, during election week, according to BuzzFeed News.

Employees' discussion of the terms that have been spiking in popularity recently, such as #DrainTheSwamp, #Trump2020, and #KeepAmericaGreat as well as QAnon affiliated hashtags, implied that Facebook has taken steps to suppress some potentially dangerous trends, but also that there are many gray area terms that don't explicitly call for violence but still could be correlated with real-world harm.

Facebook did not respond to a request for comment on this story, but a spokesperson told BuzzFeed News that Facebook is "staying vigilant in detecting content that could incite violence during this time of heightened uncertainty."

Facebook has been racing this week to remove or limit the spread of content that violates its policies against election misinformation or inciting violence, but it hasn't always been able to catch violations before they go viral.

Despite temporarily turning off political group recommendations leading up to the election, Facebook allowed a group that made false election claims and 'worrying calls for violence' to amass more than 365,000 members organically in just two days before it removed the group.

Facebook has faced increasing pressure both internally and externally to take more aggressive action on misinformation, especially around the election, and the company has rolled out a slew of new policies and tactics in recent months.

Facebook has tightened its rules against voter suppression and intimidation (including calls to violence), cracked down on disinformation networks, and spun up its election operations center, which has been be actively working with law enforcement to spot foreign influence efforts. Facebook has also rolled out a system for flagging posts for review by moderators before they go viral, temporarily turned off political group recommendations, and continues to enlist third-party fact checkers.

Even with these changes, however, misinformation researchers remain skeptical that Facebook is dealing with the issue effectively.

"I'm not sure they have a handle on it at all," Nina Jankowicz, a researcher at the Woodrow Wilson International Center for Scholars, told BuzzFeed News, saying that suppressing hashtags is an inadequate way to deal with violent-prone groups because the root cause is Facebook's business model, which relies on keeping these types of groups using its platform in the first place.

Facebook also has a long history of being accused of contributing to, and even exacerbating real-world violence, from helping fuel genocide against the Rohingya in Myanmar to refusing to take action against groups openly calling for violence against anti-police-brutality protesters in Kenosha, Wisconsin, where three were eventually fatally shot — despite receiving hundreds of warnings from users.