Beware The 'Vanity Metrics' Used To Boost Tech Startups

Boonsri Dickinson, Business InsiderAn employee of FormspringSAN FRANCISCO (Reuters) - A growing group of social and mobile Web services are poised to become the next Facebook Inc. Just ask them.

Online startups with eye-popping statistics that purport to show their popularity have captivated the attention of the media and investors. Companies boast of everything from skyrocketing Web page views to user sign-ups to shared digital photos as measures of their success.

But unlike the financial numbers used to evaluate more mature companies, the so-called "vanity metrics" now in vogue are largely self-reported and often vague enough to defy definition. With valuations for high-flying startups hitting nosebleed levels, some hear echoes of the obsession with "eyeballs," or raw numbers of website visitors, that defined the dot-com boom of the late 1990s but quickly proved to be an illusory measure of value.

"I wouldn't buy a pizzeria just because the phone is ringing off the hook," David Cowan, a partner at Bessemer Venture Partners, said in a telephone interview. "First, I have to understand who is calling, why, from where and can we really sell them pizza.

"People have to be careful about the extent to which they use these metrics as indicators of value," he said.

The issue came into focus last week when Snapchat, the hot mobile photo and video messaging startup, announced that its users send 400 million "snaps" every day just as reports surfaced that it had turned down a $3 billion acquisition offer from Facebook.

The $3 billion offer shocked many given that the company, which lets smartphone users send vanishing pictures to each other, does not have any revenue. Some also wondered what the 400 million snaps actually referred to, and how they compared to the amount of photos shared on Facebook.

If a Snapchat user sends the same photo to 10 people, does Snapchat count it as 10 snaps or one snap, a widely discussed article asked last week in online publication BuzzFeed.

A Snapchat representative told Reuters by email that the 400 million snaps includes both photos and videos that users received but did not provider further clarification. The company said in a tweet last week that 88 percent of snaps are sent to one recipient.

Many of the self-reported user metrics are "black boxes," said Joe Beninato, a Silicon Valley entrepreneur who has founded and sold several Internet companies.

Of course, 400 million of anything is a big number and a sign that the company has achieved some level of success, he said. But he said a single metric alone might not tell the whole story and could even mask problems, such as a high user churn rate.

Pinterest, a social media scrapbook service whose last round of funding valued it at $3.8 billion, is only beginning to experiment with monetization. The company does not disclose its total number of users, but recently said that the service has 1.5 million travel-related "Pins" per day. The company, which declined to comment, does not publicly provide any data about its total number of users or their activity, though it ranks among the 50 most-visited websites in the United States, according to online measurement firm comScore.

Among the most the questionable metrics that are now commonplace are a company's number of registered users, which doesn't account for duplicate and dormant accounts; a messaging service's volume of messages, which could have been sent by either a very small, or a very large, number of users; and installations of mobile apps, which includes installations by customers who did not like a service and quit using it.

Even longstanding Web traffic metrics such as page views and unique visitors remain surprisingly open to interpretation, owing to divergent measurement techniques and standards.

The number of "active" users, defined as people who access a Web service at least once a month, could include automated software that looks like real users, driving up website page views and disseminating paid advertising links on social networks.

"Sometimes these companies don't take out the 'bots,' and if you take out the bots, then suddenly 40 percent of their traffic goes away," said Hans Swildens, the founding partner at Industry Ventures.

MOM, POP AND BOTS

For Web and mobile startup companies, proving robust growth in users and time spent with the product is critical to attracting media coverage, recruiting engineering talent and securing funding.

"I've been in plenty of meetings where people are discussing what information to release to the press to get the maximum impact, something that sounds good but doesn't give away too much information to competitors," Eric Ries, who is the author of the book "The Lean Startup" and has worked at several online companies, said by phone.

Choosing the right metric to release is a key decision within Web startups these days, said one public relations executive who advises Internet companies and asked that his name not be used.

"There's a lot of pressure to talk about these type of figures," he said. "Many companies are scrambling to help people understand the scope of their business."

A good metric is one that a company has confidence will grow reliably and remain the most relevant snapshot of the product's popularity going forward. But he noted that in some cases the practice has "devolved into this exercise of trying to appear big when you're not really big. It's almost this masquerade these days."

Venture capitalists, who typically evaluate hundreds of startups every year, say they know how to cut through the superficial metrics to find out how a company is really doing.

VCs look at the ratios of daily users to weekly users and monthly users to get a feel for how many repeat users a company has. They strip out growth that comes from unsustainable practices, such as online services that tap into a user's list of contacts to send "spam" messages, and they try to determine how "viral" a service is by measuring how many of a user's friends join.

But with the recent lifting of a long-standing ban on private company investment solicitation and a proposed "crowdfunding" rule that would allow mom-and-pop investors to bet on startup companies, consumer advocates fear less sophisticated investors could easily be misled.

"It's a very selective and dressed-up version of the facts," Barbara Roper, the director of investor protection at the Consumer Federation of America, an association of nonprofit groups based in Washington, referring to the user metrics that companies provide, said by phone. "There's no standard of measurement. There's no comparability."

Even experienced investors can make mistakes. Viddy, a social video app, raised $30 million in funding in 2012 as investors jostled to get a piece of the company that was at one point adding more than 1 million new users a day.

"Their user-growth chart didn't go to a hockey stick, it went straight up a wall," an investor in Viddy, who asked to remain anonymous because he was not authorized to discuss the situation, said in a phone interview.

What many people didn't realize at the time, he said, was that Viddy's hyper-growth was mainly due to being one of the first video apps to take advantage of new distribution features on Facebook, rather than anything unique about the product itself. When Facebook suddenly tightened that distribution channel, Viddy's momentum came to a screeching halt.

Formspring, a social network which at one point was reported to have "2 billion answers to questions" posted on its website, shut down in March and was relaunched as Spring.me under a new management team.

Flud, a mobile news reader, was adding one user every four seconds shortly after its launch in 2010 - but closed shop in August.

THE SMELL TEST

Bessemer's Cowan notes that some metrics really do have value, pointing to his firm's decision to invest in professional social network LinkedIn Corp six years ago.

The fact that many LinkedIn users were inviting their friends to join the service, and that LinkedIn user profiles were appearing high up in search results on Google Inc's search engine suggested that the service was spreading in a meaningful way, he said.

The company was in its early days of generating revenue in 2007 - but the real growth potential was reflected in some of its nonfinancial metrics, Cowan said.

(With additional reporting by Gerry Shih; Editing by Jonathan Oatis)