Google's Latest Experiment Allows Your Paint Brush to Produce Musical Notes

Google has introduced a new tool — Paint with Music — that allows you to have a one of a kind experience. Launched in June this year, the platform offers an interactive experience, connecting two completely different but major art forms — painting and music composition. How does it work, you ask? 

Powered by Magenta's DDSP library (Differentiable Digital Signal Processing) and assisted by machine learning, the Paint with Music tool transforms your brush strokes into musical notes. Your paintbrush is essentially the tool that helps you create music. And yes, you can play it on any instrument from an elaborate list available. So from flute to saxophone, you have the option of painting your notes on a variety of sensorial canvases. These include "in the sky," "underwater," "on the street,"  and "on paper."

The canvas, machine learning (ML), and special sound effects on the platform come together to make your painting talk or produce music. Having said that, a little bit of understanding of musical notes will come in handy. The tool is fairly easy to use. On the Paint with Music website or the app, once you launch the experiment, Google gives you four canvas options to choose from. 

Once you have selected your favourite canvas, the next thing to factor in is the musical instrument you want your paintbrush to sound like every time you draw something. On the right bottom of your screen, you have the option to undo a note that you have drawn or even delete the entire painting and start afresh.

This is the second Google experiment that involves painting as the mainstay of the tool. In November 2020, Google AI's Chimera Painter enabled users to create a fleshed-out creature based on a rough sketch. The tool added features and textures online and this machine learning model was created with Unreal Engine and the GAN-based (generative adversarial network) ML algorithm. Even though the tool primarily focussed on the needs of game developers, anyone could use it and turn a rough doodle into a monster that looked real. 

“What if artists had a paintbrush that acted less like a tool and more like an assistant? A machine learning model acting as a paintbrush could reduce the amount of time necessary to create high-quality art without sacrificing artistic choices, perhaps even enhancing creativity,” the Google AI team wrote back then in a blog post, explaining the Chimera Painter demo.