How The Microsoft Engine That Nailed The World Cup Is Using Your Facebook Status To Predict NFL Games
Cortana's NFL predictions come from a model developed by the company's Bing Predicts team.
When you search the term "NFL predictions" or search for a specific game on Bing, the results contain the Bing Predicts prediction with the team's percentage chance of winning. Windows phone users can ask Cortana who will win this weekend's Seahawks-Broncos game, for example, and it will tell them.
Here's what the predictions look like on the web:
The Bing Predicts NFL model, which a small group of computer scientists at Microsoft came up with, includes all the traditional variables that you'd expect it to include. Determinative factors like past results, home/road effect, turf/grass effect, weather, strength of schedule, and advanced offensive and defensive statistics are all in there.
But Bing Predicts includes a variable that other systems don't - public sentiment.
"It takes into account the wisdom of the crowd," is how Microsoft director of consumer communications Craig Beilinson described it.
Because of exclusive partnerships with Facebook and Twitter, Bing has access to a mountain of social data that they can use to determine the public perception of teams and games. By analyzing aggregate Facebook status updates and tweets, the model can quantify the public sentiment for or against a given team and factor that into its prediction. The idea is to catch things that the stats can't see - like injuries or inner turmoil.
Advanced statistical models are often held up as the antidote to public perception. These models are meant to be purely empirical, immune from human subjectivity. The Bing model is different. It sees the wisdom of crowds as a legitimate indicator. It sees public sentiment as the expression of unseen variables that ought to be incorporated in the model next to traditional indicators like rushing stats.
As we saw in Week 2, this can be dangerous. The Baltimore Ravens played the Pittsburgh Steelers on Thursday Night Football only days after cutting Ray Rice in chaotic fashion that week. While Microsoft wouldn't tell us how heavily public sentiment is weighted in the model's "secret sauce," anecdotal evidence suggests it affected the prediction.
Bing Predicts had the Steelers as a relatively significant 59.8% favorite. That prediction deviated from most other models. Las Vegas had the Ravens as a slight favorite, and Nate Silver's ELO model had the Ravens as a 54% favorite.
Overall, though, things are off to a decent start.
Cortana is 19-13 on the young season. Silver's model is 19-13 as well. Vegas favorites are 17-15. Time will tell if Cortana can duplicate its World Cup success.
Bing Predicts started as an experiment to see if Microsoft could predict the results of shows like "American Idol" and "The Voice" based on what people were talking about and searching for online.
After some success in the reality TV realm, Bing moved into trickier, more saturated predictive territory - sports.
It found success there too.
It correctly predicted all but one World Cup knockout stage game. It picked Germany to beat Brazil in the semifinal when every other statistical model had Brazil as the favorite. It nailed apparent toss-up games like Costa Rica-Greece and Uruguay-Colombia. Its only blemish was the third-place consolation game between Brazil and the Netherlands - which the underdog Netherlands won.
That success raised eyebrows. It's one thing to look at social data and figure out which "American Idol" singer the public likes, it's another to correctly predict a sporting event.
Microsoft still views Bing Predicts as something of an experiment. The company breaks down predictions into three categories: open votes ("American Idol"), limited votes (the Academy Awards), and things that happen (sports). The third category is the hardest to get right. Whereas Facebook statuses and tweets about "American Idol" has a relationship to the actual outcome of the show, public sentiment in sports isn't actionable.
The team is updating its model every week to try and get more accurate predictions, which we'll be tracking week-by-week.