Business Insider India has updated its Privacy and Cookie policy. We use cookies to ensure that we give you the better experience on our website. If you continue without changing your settings, we\'ll assume that you are happy to receive all cookies on the Business Insider India website. However, you can change your cookie setting at any time by clicking on our Cookie Policy at any time. You can also see our Privacy Policy.
Amazon's cashierless grocery store cameras were fooled by a trickster's wardrobe change - but he still got charged for an avocado left by the bathroom
Amazon's cashierless grocery store cameras were fooled by a trickster's wardrobe change - but he still got charged for an avocado left by the bathroom
Tyler SonnemakerFeb 27, 2020, 06:43 IST
Reuters/Louise HeavensAmazon's Go Grocery store in Seattle, Washington.
Amazon recently opened its first cashierless grocery store in Seattle, which uses cameras, sensors, and artificial intelligence to identify which items customers buy.
A journalist from Ars Technica tried to trick the cameras and see if he could nab some items without them noticing.
After ducking into a bathroom and putting on a jacket, he said he ended up not getting charged for several items.
But even Amazon is possessive of its avocados: It charged him for one he left outside the bathroom.
The Seattle store is a larger version of the Amazon Go stores the company launched in 2018, which are more similar to convenience stores.
Amazon Go Grocery, as the company calls it, is a larger version of the convenience-store-style Amazon Go stores it first launched in 2018, and sells everything from produce and meat to beer and wine. But instead of employees ringing up customers after they finish shopping, the store uses a network of cameras, sensors, and artificial intelligence to figure out what they're selecting in real-time, and then charges their Amazon account afterwards.
Ars Technica reporter Sam Machkovech took a trip to the experimental store to see if he could bag some fruits and veggies without its surveillance system noticing. While his creative produce-juggling efforts failed, he was able to pull off a wardrobe change in a bathroom that allowed him to temporarily escape from Amazon's watchful eyes.
Advertisement
For its part, Amazon said the system isn't meant to focus on people like Machkovech.
"At Amazon Go and Amazon Go Grocery, we spend the vast majority of our time thinking about improving the experience for the 99.9% of well-intentioned shoppers vs. designing a system that is focused on deterring the few bad actors," an Amazon spokesperson told Business Insider.
Here's a look at how the stores work - and how Machkovech was able to game the system.
Before entering an Amazon Go store, customers must scan their smartphone so Amazon can figure out who they are.
Customers must download the Amazon Go app, which generates a unique QR code that they scan upon entering the store. That way, Amazon can figure out who's inside, track them while shopping, and charge their account after they leave.
Amazon monitors customers' actions once they're inside using a complex and extensive array of cameras, sensors, and motion-detection software.
Amazon Go stores don't use facial recognition, according to TechCrunch. But they do use loads of cameras, sensors, and complex software to detect motion, identify objects, and analyze all the data they collect, helping them determine which items to place in customers' "virtual shopping carts."
The sensors are "the same types of technologies used in self-driving cars: computer vision, sensor fusion, and deep learning," according to Amazon's website.
Machkovech headed for the produce section, where he hoped the irregularity of the items might give him an edge in fooling the cameras.
Canned goods or boxed items have consistent shapes, sizes, weights, and labeling, but when it comes to produce, Machkovech noted that Amazon's system must make more nuanced decisions, such as figuring out whether an avocado is a medium or large one (he said they're priced differently) and identifying subtle differences between potatoes and yams.
Machkovech said he juggled veggies, rearranged bunches of bananas, and even swapped items behind his back, hoping his sleight of hand might overwhelm the cameras.
Amazon's cameras keep a close eye on customers while they shop, but fortunately not while they're using the restroom.
Machkovech eventually decided to duck into a restroom, and while the hallways leading there did have cameras, he didn't see any inside the restroom itself. Before going in, Machkovech left an avocado on a shelf where Amazon had encouraged shoppers to leave merchandise. Once inside, he hid a banana that had come loose during his fruit juggling in his backpack.
Machkovech also put on a jacket that had been in the bag and removed a pair of sunglasses, hoping the costume change might help him disappear. On the way out, he grabbed a few small non-produce items so he could test his theory.
Go stores don't have a typical checkout process, customers just place items in their bag and walk out when they're done.
Once done shopping, customers simply exit the store rather than going through a traditional checkout process. Amazon's software will note that they left, process the data it has from their visit, total up the items, and charge their Amazon account.
When Machkovech finally received his receipt, he said he discovered he had been charged for both the loose banana and avocado left outside the restroom, but not the items he took after changing.
His receipt also took several hours to process, with Amazon estimating Machkovech had spent more than two hours shopping, suggesting his wardrobe change had done the trick.
Amazon's Go stores are just part of its major push in recent years to grow its physical presence.
Machkovech said he chalked the experience up to Amazon "solving a problem that doesn't exist," arguing that people are perfectly capable of checking customers out.
But Amazon apparently disagrees, and is reportedly looking to open 3,000 Go stores by 2021 as part of its recent push into physical locations. That effort includes Amazon's acquisition of high-end grocery chain Whole Foods in 2017 for nearly $14 billion, as well as its expansion of book stores and "4-star" stores (locations that sell a small selection of highly rated items from Amazon's online inventory).
As for thwarting potential shoplifters as it expands its cashierless Go stores, Amazon told Business Insider that "overall our technology is highly accurate."
