Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have introduced an innovative approach for training precision home robots using a real-to-sim-to-real methodology. This breakthrough enables robots to be trained in simulations of scanned home environments, making customized household automation accessible to everyone.

Revolutionizing home automation

Time-consuming household chores are at the top of many automation wish lists. Many roboticists aim to develop hardware and software that allows robots to learn "generalist" policies—rules and strategies that work under all conditions. However, most homeowners are more interested in robots working perfectly in their environments. With this in mind, CSAIL researchers have developed a solution to efficiently train robust robot policies for particular settings.

Introducing RialTo

RialTo, the new system developed by MIT researchers Marcel Torne Villasevil and Pulkit Agrawal, is more complex than just scanning a room with a phone. It begins by scanning the target environment using tools like NeRFStudio, ARCode, or Polycam. Once the scene is reconstructed, users can upload it to RialTo’s interface to make detailed adjustments, add necessary joints to the robots, and more.

The refined scene is then exported into a simulator to develop policies based on real-world actions and observations, such as grabbing a cup on a counter. These real-world demonstrations are replicated in the simulation, providing valuable data for reinforcement learning.

Results and Testing

Testing showed that RialTo created strong policies for various tasks, improving 67 per cent over imitation learning with the same number of demonstrations. The tasks included:

• Opening a toaster.
• Placing a book on a shelf.
• Putting a plate on a rack.
• Placing a mug on a shelf.
• Opening a drawer.
• Opening a cabinet.

Researchers tested the system's performance under three increasing levels of difficulty: randomizing object poses, adding visual distractors, and applying physical disturbances during task execution. The system outperformed traditional imitation-learning methods when paired with real-world data, particularly in situations with significant visual distractions or physical disruptions.

Challenges and Future Improvements

Despite its success, RialTo currently requires three days to be fully trained. The team plans to improve the underlying algorithms and use foundation models to expedite this process. Simulation training has limitations, such as difficulty with sim-to-real transfer and simulating deformable objects or liquids.

Collaborative Effort and Support

This research was led by Marcel Torne Villasevil, with senior authors Abhishek Gupta, assistant professor at the University of Washington, and Pulkit Agrawal. Additional CSAIL contributors include EECS PhD student Anthony Simeonov SM ’22, research assistant Zechu Li, undergraduate student April Chan, and Tao Chen PhD ’24. The Improbable AI Lab and WEIRD Lab members provided valuable feedback and support.

The Sony Research Award, the U.S. government, and Hyundai Motor Co. supported the project, assisted by the WEIRD (Washington Embodied Intelligence and Robotics Development) Lab. The researchers presented their work at the Robotics: Science and Systems (RSS) conference earlier this month.

Looking Ahead

The CSAIL team continues to refine RialTo, aiming to make home automation robots more accessible and effective in customized household environments. With ongoing improvements and collaborations, the future of home automation looks promising, driven by cutting-edge research and innovation.

Source: Article

Image source: MIT News