Here's how Descartes Labs built one of the world's most powerful supercomputing systems using only $5,000 and Amazon's cloud
- Descartes Labs' supercomputing system is one of the 500 most powerful commercial supercomputers in the world - and it's the only one on that list that is completely run on the cloud.
- Descartes Labs uses Amazon Web Services, and it says that its system can be cheaper than building and setting up physical supercomputing infrastructure.
- Across the industry, more customers are looking to the cloud for high-performance computing - the kind of thing that traditionally required a dedicated mainframe or other kind of supercomputer.
- Read more BI Prime stories here.
Twice a year for the last 26 years, TOP500 publishes a list of the 500 most powerful commercial supercomputers. This most recent list includes a historical first: satellite imagery and data company Descartes Labs rated on the list at #136 - making it the only one of its ilk to run completely on the cloud.
Descartes Labs' computer uses Amazon Web Services, the retailer's market-leading cloud platform. This system can model how people and companies use natural resources at a global scale, simulate weather, and measure earthquakes.
Just as impressively, to build this system, Descartes Labs spent a paltry $5000 on AWS's services - far less than it would take to build most of the rest of the supercomputers on the TOP500 list, which could cost tens of millions of dollars and take months of work just to get up and running.
Mike Warren, CTO and cofounder of Descartes Labs, says that there's always going to be a need for specialized supercomputing systems, especially at places like research labs. But Descartes' approach shows that there's a way to get access to this kind of computing horsepower, without having to spend millions.
"Our performance on the list as a demonstration of the capability of the cloud, but it's definitely not a claim that this is the right way for everyone to solve their specific problem," Warren told Business Insider. "It's certainly going to be an option that's valuable for some people."
Notably, Amazon itself has made the TOP500 list before. However, to achieve that feat, it used the retailer's own, internal platforms - not the version of Amazon Web Services that's available to the paying public.
Descartes, for comparison, achieved this feat on the real, actual AWS platform, facing the same limitations and costs as any other customer of the platform.
This all happens against the backdrop of the burgeoning market for high-performance computing, or HPC. Chirag Dekate, senior director and analyst at Gartner, says that the market for HPC in the cloud is poised to explode, as almost every company would prefer to avoid the high costs of the more traditional approach.
"Many companies are increasingly looking at cloud-based capabilities," Dekate told Business Insider. "They're trying to figure out how to make HPC work in the cloud. While the current market for HPC is small, over time, they are looking forward to creating a cloud based strategy."
'The cloud has grown up'
Warren spent most of his career in building supercomputers, as far back as the '90s. In 1998, Warren and his team at Los Alamos National Lab built a top-500 supercomputer using Linux.
"At the time, there was a lot of discussion around,' why did this scientist waste his time on a crazy idea of wiring together these machines,'" Warren said.
When he first started building supercomputers, the cloud didn't exist yet. Now, it's easy to set up computing infrastructure with little more than a credit card.
Warren believes that supercomputing applications will eventually migrate to the cloud. The advantage is that it's cheaper, and a company has more control over how they want to run and scale their applications.
"The cloud has grown up to a point where it can do these applications that were formerly the domain of supercomputers you would spend tens of millions of dollars on," Warren said.
"We basically demonstrated it was possible to do that in the world of the cloud where you don't have to own software and there's no upfront capital expense," he added.
Dave Bartoletti, vice president and principal analyst at Forrester, says that although it will still require specialized set-up and expertise, Descartes Labs' system will make engineers and data scientists will rethink how to best run demanding supercomputing applications.
"It shows how public cloud platforms are maturing to be able to handle an ever-increasing range of even the most compute-intensive workloads," Bartoletti told Business Insider.
