Facebook's magic engine for crunching our data just got faster - and any company can use it
One way Facebook does this is with Presto, a tool it developed a little while back to make sense of its 300 petabytes (as in 300,000,000 gigabytes) and growing mountain of data. Existing database and analytics technology was way too slow, and so Facebook engineers got together and decided the only way forward was to build its own that worked like magic.
Today, 1,000 Facebook employees use Presto to crunch a petabyte (as in 1,000,000 gigabytes) of data every single day as the company tries to learn more about Facebook's users and figure out how to make the social network better.
It's designed to be fast and to take data from several different places from anywhere in a business, giving anybody the tools to analyze huge, huge amounts of data. It works with trendy databases like Hive and Cassandra, but Presto works with the more established kind of technologies like SQL too.
But the really cool part is that Facebook gives Presto away for free to anybody who wants it as open source. Dropbox, for example, uses Presto to analyze how people use its cloud storage, which is helpful when trying to diagnose problems or search for errors. Airbnb is also a Presto user.
Today, Facebook announced a whole mess of improvements to Presto, mostly focused around making it faster for software engineers to ask questions of the data, meaning they can do even more. That's not an easy goal, given how much data has to be parsed for meaningful answers at the size of a Facebook or a Dropbox.
"Improving query performance directly improves their productivity," writes Facebook engineer and Presto creator Dain Sundstrom in a blog post.
This isn't the first time Facebook has given away useful technology that could undercut traditional ways of doing things: The social network has been working hard on the Open Compute Project, which is trying to design a better and more efficient way to build data centers, with partners like Apple - much to Cisco's chagrin.