Robots need some very specific instructions in order to successfully accomplish tasks. If you want a robot to bring you a beer while you recline on the couch, it needs to know what a beer is, that a beer is in a refrigerator, what a refrigerator looks like, how to open said fridge, and so on. It's tedious!
Humans can't possibly provide all this context with a simple verbal request, so Ashutosh Saxena and researchers at Cornell University's Robot Learning Lab aim to help bridge this communication gap with a project called "Tell Me Dave."
Saxena's robot, equipped with a 3-D camera, scans its environment and identifies the objects in it, using computer vision software previously developed in Saxena's lab. The robot has been trained to associate objects with their capabilities: A pan can be poured into or poured from; stoves can have other objects set on them, and can heat things. So the robot can identify the pan, locate the water faucet and stove and incorporate that information into its procedure. If you tell it to "heat water" it can use the stove or the microwave, depending on which is available. And it can carry out the same actions tomorrow if you've moved the pan, or even moved the robot to a different kitchen.
Put another way, instead of hard-coding numerous processes that are effortless for a human but quite elaborate for robots (like grabbing a beer from the fridge), Saxena and crew have taught their PR2 bot to recognize various objects after being taught their traits. A microwave can be used to heat things, a bowl can hold water, a tap dispenses water, and so on.
A quick verbal request to heat up a bowl of water, let's say, is now completely within a robot's purview despite the usually robot-beating lack of info associated with such a statement - a robot now "knows" how to fill in the steps missing from the request.
As a demonstration of this ability, the video above demonstrates a robot filling a rather open-ended ice cream order: "Take some coffee in a cup. Add ice cream of your choice. Finally, add raspberry syrup to the mixture." This is more or less how we imagine we'd speak to a robot in a sci-fi movie. It takes the robot some time, but soon enough: affogato!