Researchers at University of Washington have created an AI system that can create music using visual cues alone, reported UW news. ‘Audeo’ is an ML-enabled system that learns from a silent top-angle video of a pianist playing her instrument.

Audeo is able to recreate the melody through a series of steps: first by capturing the exact keys keys that are pressed and in what order, and then decoding nuances such as the emphasis given to each key. The researchers presented Audeo at the NeurIPS 2020 conference on December 8, 2020.

“To create music that sounds like it could be played in a musical performance was previously believed to be impossible,” said senior author Eli Shlizerman, an assistant professor in both the applied mathematics and the electrical and computer engineering departments. “An algorithm needs to figure out the cues, or ‘features,’ in the video frames that are related to generating music, and it needs to ‘imagine’ the sound that’s happening in between the video frames. It requires a system that is both precise and imaginative. The fact that we achieved music that sounded pretty good was a surprise.”

Audeo was trained and tested on YouTube videos of pianiast Paul Barton. The AI-generated music was correctly identified by music-recognition apps in 86% cases, as compared to 93% success rate of the training videos.

“The goal of this study was to see if artificial intelligence could generate music that was played by a pianist in a video recording — though we were not aiming to replicate Paul Barton because he is such a virtuoso,” Shlizerman said. “We hope that our study enables novel ways to interact with music. For example, one future application is that Audeo can be extended to a virtual piano with a camera recording just a person’s hands. Also, by placing a camera on top of a real piano, Audeo could potentially assist in new ways of teaching students how to play.”