Facebook Is To Blame For A Wave Of 'Dark Traffic' Swamping The Web But It's Working On A Fix

Lara O'ReillyDec 5, 2014, 18:43 IST

Flickr CC/Christopher"Dark social" could become less of a problem as Facebook is working on a fix.

So-called "dark traffic" is a huge problem for publishers and website owners. It's useful to be able to look at your analytics, but when a huge proportion of your referral traffic is listed as "direct" (and aside from a homepage, it's very unlikely users are typing in full URLs to their browser to go directly to a page) it's difficult to know where to focus in order to boost your audience. Now new research from analytics firm Chartbeat, as well as confirmation from major publishers, confirms that Facebook's mobile apps are largely responsible for the swathes of dark traffic being directed toward websites. We already knew Facebook is the principal source of social referral traffic for the majority of digital publishers. But Facebook has been underselling itself. Fortunately for publishers and website owners, it appears that Facebook is working on a fix. The term "dark social" was coined by Alexis C. Madrigal, a senior editor at the Atlantic, who found more than half of the website's referral traffic was seemingly untraceable. It mostly refers to when links have been shared via an online chat, email or app rather than through a browser or specific social app through which referrals can easily be tracked. Chartbeat has found in some cases dark social accounts for 65% of certain websites' traffic, but averages at about a third across its network. Around 10 - 15% of The Guardian's traffic is dark traffic. Most analytics platforms (Google Analytics, Adobe Omniture, Chartbeat) have a huge section in their reports called "direct," which counts all those apps that users use to click on links "direct to a site." Facebook is one of them - alongside other apps like Reddit, Gmail and IM apps. Twitter on other hand, has a special "t.co" URL shortener which means Twitter referrals are always easily trackable.

The problem with Facebook not using a shortener like this is that a huge proportion of its users access the site only via the mobile app. We know 703 million people visit the site via their mobiles daily, and presumably the majority of those do so via the app. That's a huge wave of referral traffic going unidentified. Chartbeat, which from this week began tracking Facebook users more effectively using a different method (more on that later,) has published a blog post that clearly marks out the Facebook/dark social impact. Last month the analytics company began to look at time series data for specific articles to identify patterns in traffic and correlate that with the "dark social" number.

Take a look at the chart below. The dark traffic spikes at exactly the same time the article began going big on Reddit, and then again when it started picking up pace on Facebook.

Chartbeat Madrigal also wrote in a blog post published this week that he too conducted a test earlier this year. He set up an Atlantic post "deep in the sand of time" so only people who were part of the test would find it. He then posted a link to the story on his Facebook page and told his friends only to click to it if they were using the mobile app. He then looked at the referrers (of which he knew 100% were coming from Facebook). A few people showed up on his Adobe Omniture analytics report as coming from Facebook.com, but the rest showed up as "type/bookmarked." For the most part, those that had been clicking on their mobiles had gone untraced: Alexis Madrigal Figuring that those numbers would be tightly correlated with the overall number of Facebook mobile visitors, Madrigal compared Facebook mobile referrals with dark social. The correlation was pretty comparable. Alexis Madrigal As a result of its tests, Chartbeat has started looking more closely at the "user agent," a tag users leave when they visit a website which identifies the type of browser they use, operating system and usually reveals some information about where they originate from. Most analytics systems don't dig deeply into this type of information. Some publishers have developed their own methods to dig further into their website analytics to uncover the "user agent," which reveals Facebook as the source of the direct referral traffic. Both The Guardian and BuzzFeed said on Twitter last night they have been digging further into user agent data for a while now. But that takes both time and expertise that most publishers don't have. Malcolm Coles, Trinity Mirror (owner of UK newspapers The Daily Mirror and The People) digital director (who is soon to take up the same role at the Telegraph Media Group, owner of The Daily Telegraph,) told Business Insider that he has been told by analytics companies that Facebook is "finally" fixing the problem. A separate tweet from digital analytics company Parse.ly suggested Facebook has a fix in the latest updates to its mobile apps. Business Insider contacted Facebook for further details, but the company said it had nothing to share. "The main effect has been that people have underestimated the impact Facebook has on their traffic and that people have maybe been overestimating the impact of Twitter," Coles said. But he adds that until every app owner - Reddit, Whatsapp email browsers and so on - do the same thing by creating bespoke automatic URL shorteners, the problem of "dark social" won't go away any time soon.

Cookies on the Business Insider India website

Facebook Is To Blame For A Wave Of 'Dark Traffic' Swamping The Web But It's Working On A Fix

Take a look at the chart below. The dark traffic spikes at exactly the same time the article began going big on Reddit, and then again when it started picking up pace on Facebook.