 |
This post is part of the "Future of Business" series, which examines how cutting-edge technologies are rapidly reshaping our world, from how businesses run to how we live. "The Future of Business" is sponsored by SAP. See more posts in the series » |
It refers to the gobs of information being collected about us and our world through things like Twitter, Facebook, Google, smartphones, GPS locations, and tiny sensors built into everything.
But the question for most people is...so what? Why are techies and scientists so excited about this thing? After all, it's not like data didn't exist before, even in large quantities.
The short answer: big data is a new tech that can help us understand, and then predict, things like never before. Things like what we'll buy and when, when the prices of airline tickets and electronics are cheapest, which New York manholes are most likely to explode, or which tenement buildings most likely to catch fire.
Who wouldn't want to know stuff like that?
Big data can do this because it uses all information on a topic, instead of a smaller more manageable subset. Then it finds correlations, or things that tend to happen at the same time.
Correlations allow us to discover patterns and patterns allow us to predict the future, explains the authors of the book "Big Data: A Revolution that Will Transform How We Live, Work and Think," written by Oxford professor Viktor Mayer-Schonberger and big data journalist for the Economist, Kenneth Cukier.
The more data you have, the more correlations you can find and the more accurate the pattern and the prediction become.
Correlations are different from the classic method of making predictions: trying to find out "why" or "what caused something." That's known as "causation."
A correlation does not mean that one thing caused the other. For instance, earlier this year, MasterCard created its own big data division. After analyzing billions of records, it learned that when someone buys gas at around 4 p.m., the person will likely then spend $35-$50 on groceries or at a restaurant. The gas purchase didn't cause the food purchase.
"Predictions based on correlation lie at the heart of big data," the authors write.
Google is the biggest examples of a "big data" company that knows how to use correlations. It keeps records of everything we do with its online services and finds new ways to use that data. For instance, Google created a realtime flu tracker by watching where people searched for terms relating to illness and mapping that data from the Center for Disease Control.
A lot more of this is on its way. Here are some examples from the book of Google-like companies doing interesting things with big data.
- Oren Etzioni, creator of one of the first search engines, MetaCrawler, scraped data from a travel website to build a tool called Farecast that predicted when an airfare price was likely to go down. Microsoft bought Farecast for $115 million in 2008. Today, it's part of Bing.
- Then Etzioni turned around and created Decide.com to help people predict when prices will drop on electronics.
- PriceStats tracks the prices of millions of products in over 70 countries to keep tabs on inflation and the economy.
- AirSage gathers 15 billion geolocation records daily from multiple cell phone carriers. It uses this for traffic analysis to help city planners.
- Similarly, Inrix collects traffic data from the sensors in the cars themselves via BMW, Ford, Toyota, and others and uses that to help city planners model traffic flow.
- Kaggle crowdsources the process of writing a big data algorithm by running contests.
- TheNumbers.com collects detailed records on Hollywood films, from box office sales to who worked together on which films. Producers use it to predict how much money a film is likely to make.