Playing Moneyball with Startups — the Metrics Dilemma

How do VCs answer the question-“Should I invest in this Startup?”

Before I get to that, I’ll quickly digress to share the story of a remarkable sporting achievement that gave more impetus to a hitherto, underused method for evaluation of startups.

Back in 2002, the Oakland Athletics baseball team created history by finishing first in their division.

No mean feat, especially when you are an underdog, with severe financial limitations and having lost 3 of your best players to the elite teams. And then to top it all, going on to set an American League record for the longest win streak with 20 consecutive wins.

More interestingly, the way in which the Oakland A’s achieved all of this was so incredible and unique at the same time, that it inspired a book (which went on to become a bestseller) and a Hollywood movie starring Brad Pitt that received 6 Academy Award nominations including one for Best Picture. The movie’s screenplay, was written by no less than Aaron Sorkin (of The Social Network and Steve Jobs fame).

The book details how Billy Beane, Oakland Athletics’ General Manager, and the brain behind what came to be known as the ‘Moneyball’ strategy catapulted them to unprecedented success with 4 more division crowns.

Moneyball in a nutshell
In sports, the Moneyball model is about finding undervalued (or under-the-radar) players and buying them on the cheap to put together a competitive team.

It gets you the maximum bang for your buck using data and detailed analysis to identify value, which your non-moneyball competitors are yet to spot. They do this by measuring in-game activity of players to make cost-effective purchase decisions through comparison and heavy use of statistics.

It has been used extensively in baseball, basketball and football, with remarkable success.

Moneyball and Consensus
In his recent article titled, The Selection Dilemma, Kae Capital’s Shubhankar Bhattacharya, talks about Indian skipper, MS Dhoni, getting selected into national team despite non-consensus amongst selectors (such scenes play out quite often in the movie Moneyball as well).

In the context of Dhoni’s selection, it could be argued that it might not have been entirely, an emotional decision. Perhaps, those selectors who pushed his case for inclusion were playing Moneyball in their heads.

Let me elaborate…

Although conventional wisdom suggests relying on batting averages or runs scored to determine who the better batsmen is, the Moneyball approach would not consider it to be an effective metric at all. Depending on the format of the game (test/ ODI/ T20), it would consider other, more effective metrics, given the team’s requirement or objective for a particular position. In this case, that happened to align with the stated reason by then captain Sourav Ganguly for Dhoni’s inclusion over fellow wicketkeeper-batman, Dinesh Karthik (who had scored more runs than Dhoni in domestic matches that season).

Ganguly mentioned that he needed a batsmen who could perform under pressure on a bigger stage. In Moneyball terms, that probably maps to higher strike rate, runs scored and partnerships stands specifically in the slog/ death overs or while batting second in internationals. Did Dhoni’s stats better Karthik’s in that regard? Unfortunately, I do not have the data. But, his inclusion was justified when he consistently finished tight games for the national side, filling in a precise requirement.

The Game VCs play – Selection or Moneyball?
Coming back to the article, Shubhankar, goes on to elaborate on how the job of a VC is akin to that of the selector, in identifying promising startups that have been overlooked or misunderstood and hence passed on by other investors. He further illustrates via a 2x2 matrix on how the most successful bets come from non-consensus investments that are not necessarily backed by data or even common sense.

But, if data can really provide actionable decisions, then why do VCs still largely rely on their ‘gut feeling’ or go by their ‘instinct’ before investing?

The Moneyball Dilemma
That’s because you cannot solely rely on statistics to tell you everything about a startup. Some things cannot be measured statistically or accurately — like a founder’s grit, or for that matter, the craziness of an idea!

Just like sometimes, the only way a coach/ manager/ selector can assess certain qualities of a player, is to watch him/her play, VCs like to assess founders too, over several months and many meetings before they take a bet.

Data scientists, however, would argue that everything can be measured — if you can see it and discriminate it based on those values, a stat can be made for it. According to them, a stat being unavailable only means that the stat doesn’t exist, yet!

They might have a point.

Given the massive advances in machine learning, the Moneyball approach can not only throw data points, but also help find the narrative in that data and its impact (there’s already AIVC — the world’s first Venture Capital fund powered entirely by Artificial Intelligence).

Coming back to the first question…
How does a VC answer the question, “Should I invest in this Startup”

I believe the answer will always be a mix of the emotional and the rational.

The emotional ‘Yes’ or ‘No’ determines the faith the VC has in the team and the idea; and the rational will always be a percentage that goes something like,

“There’s a 72.8% chance that the startup will attract follow-on funding”

or

“a 48.2% probability that this startup gets acquired”

or

“a 12.9% chance this startup will list publicly in the next 8 years.”

The latter, in fact, is based more on the evaluation of business models and traction gained. And in that, it aligns with those startups in the [Successful + Consensus] quadrant, which will almost always give less than ideal or maximum returns. The reason is because investors make a beeline to invest in those companies that seem to have everything going right. Thus providing diminishing returns as their value is driven up.

Having said that, investors and founders alike, need to move towards an increasingly advanced set of statistics and make it a bigger part of public discussion when it comes to determining performance of startups. That would mean the media would need to play an active role in the process by not glorifying vanity metrics such as GMV or number of app downloads or number of items sold. A case in point being the recent war of words between the CEOs of Flipkart and Amazon India over Dussera and Diwali sales figures.

Such efforts can also help reign in valuation bubbles. Overvaluing and over-investing, in startups or business models, that are fundamentally broken or flawed, can be thus controlled when the industry moves towards a ‘gold standard' for evaluation via metrics.

Also, such complex metrics would also help overcome the conundrum posed by Goodhart’s Law, as these metrics will be hard to game.

Final thoughts
In the end, both approaches try to find value where it is not immediately apparent. However, many questions remain to be answered as one system cannot completely replace the other. To choose one, would be ill-advised and plain foolish.

Perhaps, if VCs can evaluate their bets in retrospect and share the data, it could merit a follow-up article. If you had like to reach out in that regard (or even otherwise), I’m on Twitter and LinkedIn.