Researchers develop robotic AI system that learns by watching how-to videos

As artificial intelligence (AI) and robots continue to have an increasingly important role in our everyday lives, researchers at Cornell University have developed a new AI-powered robotic framework that allows robots to learn by watching how-to videos.

Specifically, this approach, called RHyME (Retrieval for Hybrid Imitation under Mismatched Execution), makes robots less finicky and more adaptive than before, empowering them to use their own memory and connect the dots when performing tasks they’ve seen only once, per a Tech Xplore report from April 22.

Learning by watching how-to videos

In the words of Kushal Kedia, a doctoral student in the field of computer science, and one of the study’s authors:

“One of the annoying things about working with robots is collecting so much data on the robot doing different tasks. (…) That’s not how humans do tasks. We look at other people as inspiration.”

For instance, a RHyME-equipped robot that has seen a video of a human picking up a mug from the counter and placing it in a nearby sink will scan through its library of videos and draw inspiration from similar actions, such as grasping a cup and lowering a utensil.

Historically, robots have required precise, step-by-step directions to carry out basic tasks and didn’t know what to do when things would go off-script, like if they dropped a tool or lost a screw. However, RHyME could substantially reduce the time, energy, and money spent on training them.

According to the researchers, this would enable robots to learn multi-step sequences while significantly reducing the amount of robot data needed for training, as RHyME only needs 30 minutes of it, already demonstrating a more than 50% increase in task success compared to previous methods.

Furthermore, this branch of machine learning, called ‘imitation learning,’ could spell massive implications for the adaptability of household robot assistants that might otherwise lack the wits to navigate the physical world and its countless unforeseen obstacles.