In a study, researchers from New York University, Meta, and the robotics company Hello Robot have developed a series of AI models that teach robots to complete basic tasks in new surroundings without further training or fine-tuning. The five AI models, called robot utility models (RUMs), allow machines to complete five separate tasks—opening doors and drawers and picking up tissues, bags, and cylindrical objects—in unfamiliar environments with a 90% success rate. 

The team hopes its findings will make it quicker and easier to teach robots new skills while helping them function within previously unseen domains. The approach could make deploying robots in our homes easier and cheaper.

According to Mahi Shafiullah, a PhD student at New York University who worked on the project, in the past, people have focused a lot on the problem of ‘How do we get robots to do everything?’ but not asking ‘How do we get robots to do the things that they do know how to do—everywhere?. “We looked at ‘How do you teach a robot to, say, open any door, anywhere?", he added.

Collecting data

Teaching robots new skills generally requires a lot of data, which is hard to come by. Because robotic training data needs to be collected physically—a time-consuming and expensive undertaking—it’s much harder to build and scale training databases for robots than it is for types of AI like large language models, which are trained on information scraped from the Internet.

To gather the data essential for teaching a robot a new skill faster, the researchers developed a new version of a tool they had used in previous research: an iPhone attached to a cheap reacher-grabber stick, the kind typically used to pick up trash. 

The team used the setup to record around 1,000 demonstrations in 40 different environments, including New York City and Jersey City homes, for each of the five tasks—some of which had been gathered as part of previous research. Then, they trained learning algorithms on the five data sets to create the five RUM models.

These models were deployed on Stretch, a robot with a wheeled unit, a tall pole, and a retractable arm holding an iPhone, to test how successfully they could execute the tasks in new environments without additional tweaking. Although they achieved a completion rate of 74.4%, the researchers were able to increase this to a 90% success rate when they took images from the iPhone and the robot’s head-mounted camera, gave them to OpenAI’s recent GPT-4o LLM model, and asked it if the task had been completed successfully. If GPT-4o said no, they simply reset the robot and tried again.

Overcoming challenges

A significant challenge facing roboticists is that training and testing their models in lab environments isn’t representative of what could happen in the real world. Research that helps machines behave more reliably in new settings is much welcomed, says Mohit Shridhar, a research scientist specializing in robotic manipulation who wasn’t involved in the work. 

The project could serve as a general recipe to build other utility robotics models for other tasks, helping to teach robots new skills with minimal extra work and making it easier for people who aren’t trained roboticists to deploy future robots in their homes, says Shafiullah.

MIT