AR startup Blippar has launched the visual browser it says could be 'bigger than the internet itself'
On Thursday, Mitra revealed the fruits of his work.
"Blipparsphere" is a "visual browser" that uses machine learning to recognize real-world objects.
To use it, users fire up the iOS or Android Blippar app and point their smartphone camera at any object.
The app then uses Blippar's "proprietary knowledge graph" to analyze the object's characteristics and suggest what that it might be - based on its previous performance of recognizing similar objects - and then offer additional useful information about it from the web.
The suggestions appear in a circle on the middle of the smartphone screen, while other words related to the characteristics of that object or other similar objects buzz around it:
The aim is for the machine learning database to build a visual catalog of every object in the world - from animals and people that move around, to plants that often change their characteristics day-to-day, to logos and landmarks, and the millions of mundane objects.
The algorithm will also learn the user themselves, suggesting a cluster of objects that are likely relevant to that individual person, based on the kinds of things they have pointed their phone at before.
We tried it out for ourselves
Business Insider was given a demo of the Blipparsphere at the company's London office last month.
Sure enough, when I pointed the smartphone camera at a chair, the coffee machine, a spoon, a banana, and a television set, the app instantly recognized what the objects were.
Keen to ensure I was not being subjected to a controlled environment, with objects that have been tested dozens of times by Blippar's staff, I pulled out a pen from my bag. The app recognized it.
Someone nearby had their rucksack next to them on the table. The app instantly recognized the Nike logo on the pocket.
Pointing the smartphone camera out of the window, the app recognized the small objects speeding across London Bridge below were automobiles.
Mitra opened the fridge door and pulled out - seemingly at random - someone's lunch. The app recognized some of the different elements of the salad inside the lunchbox (through the transparent plastic casing) - lettuce, tomato, and even had a guess that there was also meat of some sort inside. Point the smartphone camera at sushi and words like "rice" and "fish" popped up, before it finally linked the two together and made an accurate guess.
Mitra explained that if a human could reasonably guess what a dish is, or what ingredients it contains, it's likely the Blippar app could do a good job of it too.
Mitra recalled a recent trip to Beijing when he did a walking tour. Mitra said he impressed the locals at the market, saying the app recognized different species of fish and enabled him to teach them about the history of the Peking duck.
The app did make mistakes on the way: It thought the microwave was a toilet because there was a kitchen roll stood next to it on the kitchen side. Sometimes it mistook a chair for a table. And factors such as shadow and distance affected its performance - but its guesses were still intelligent, as they were based on the accuracy of the results it had delivered to other users who had pointed their smartphones at objects with similar characteristics.
How Blippar got there
Impressed, but still somewhat cynically questioning whether I was being duped in some way, I asked Mitra how on earth a small company of 350 people that originally started out just five years ago in the augmented reality space had seemingly made a visual recognition breakthrough the likes of tech giants like Google and Microsoft have failed to crack.
Its augmented reality grounding certainly helped from a technology standpoint.
The Blippar app was founded in 2011 as a way to point your smartphone camera at a physical object to make it come to life in some way on your screen. It made its money by allowing brands to make their products "blippable" for ad campaigns, but in those five years, it also got very good at recognizing all different types of objects - from cars, to logos, to pieces of fruit.
Hypothetically, anyone who has used the Blippar app in the past, has helped the company's system learn how to make a realistic guess as to what an object might be.
Mitra believes it's also the culture of the business that allowed it to pivot towards building its visual discovery browser.
He said: "First of all: Focus. Large companies like Google can have thousands of engineers distributed on so many hobby projects. We have a dedicated focus."
He added that Blippar looks to hire "thinkers rather than executors".
He explains that with an analogy: "There are many doctors in this world, hundreds of thousands of doctors, but there are maybe 3,000 amazing brain surgeons. You cannot compare the two. There is a professional comparison, but not really, so we are after those brain surgeons. One of those is equal to hundreds of engineers elsewhere. They are handpicked. Our given mission has never been achieved before and that's why we are having breakthroughs."
Mitra said there's another key difference in his leadership style: "I'm very philosophical ... I'm willing to lose anything to achieve [our goal] and that's difficult to achieve in large companies."
Blippar has also been buoyed by $104 million in outside investment. In March, the company raised a $54 million Series D round led by Khazanah National Berhad, the strategic investment fund of the government of Malaysia, which has also previously invested in technology firms including Chinese e-commerce giant Alibaba and British travel search site Skyscanner.
What the Blipparshere could be used for
Mitra passionately bounces as he reels off all the potential use cases for the Blipparsphere: Traffic-flow management; it could help radiologists assess x-rays and diagnose health issues; it could be fed with pattern behavior and alert a CCTV officer when there was about to be a riot or a terror attack; it could be installed in self-driving cars - there are a lot of possibilities.
The subject Mitra is most passionate about is removing illiteracy as a barrier to development in third-world countries. Someone in an unprivileged part of the world could potentially use the app to determine whether the soil in front of them was fertile, whether a fruit was safe to eat, or how to use a piece of machinery.
"People have disrupted the automotive industry, the retail industry, people disrupted banking. This is like disrupting 5,000 years of social structures and bringing knowledge parity in the world," Mitra said.
Blippar is not there yet - the missing piece is accurate natural language translation within its image recognition engine - but Mitra adds: "We are getting there."
For the time being, Blippar will be looking to go back to its heritage to make money from the Blipparsphere: Making money from brands.
Nestle, for example, could pay Blippar to ensure the Blipparsphere always accurately identifies the make and model of its coffee makers and provides up-to-date information and an easy way to purchase the machine, or more coffee pods.
In the meantime, Mitra wants Blippar users to point their phones at as many objects as possible. The more people use the app, the more its machine learns, and the more intelligent the Blipparsphere will be.