Rochester researchers collaborate on project to leverage AI for music production

The interdisciplinary project includes an education and outreach component to better prepare future musicians

 A team of researchers from the University of Rochester’s Eastman School of Music, Warner School of Education & Human Development, Hajim School of Engineering & Applied Sciences, and School of Arts & Sciences, along with Northwestern University’s McCormick School of Engineering, are collaborating to leverage and better utilize Artificial Intelligence (AI) to empower musicians to produce and disseminate their art more effectively and independently.

This project, which received a $1.8M grant award from the National Science Foundation Future of Work at the Human-Technology Frontier program, builds on the preliminary work of an interdisciplinary team of 16 faculty, staff, and students across the fields of music, audio engineering, ethics, and education. Five principal investigators lead the team – the University of Rochester’s Raffaella Borasi (Warner School of Education), Rachel Roberts (Eastman School of Music, Institute for Music Leadership), Zhiyao Duan (Hajim School of Engineering & Applied Sciences), and Jonathan Herington (School of Arts & Sciences), and Northwestern University’s Bryan Pardo (McCormick School of Engineering). This grant marks the first sizable NSF-funded research project with deep collaboration between Eastman, Hajim, ASE, and Warner.

Raffaella Borasi, the lead principal investigator, says of the collaboration, “The most exciting and rewarding aspect of this project is the opportunity to work together with people contributing such different expertise to address the many complementary components of a complex problem.”

There is a growing demand for digital music content, including music for film, television, games, and advertising, as well as the traditional recorded music market. AI-powered tools have the potential to enable musicians to create digital music products on their own and at a very low cost in ways that are not yet possible. By eliminating their dependence on centralized music production, musicians will achieve greater creative freedom and reach larger audiences while increasing their chances of making a living with their art.

However, musicians must overcome many obstacles before realizing this potential. Despite significant advances in audio AI research, today, only a few musicians (even among electronica and hip-hop artists who have made the most use of technology in their work) are using AI-powered tools. Obstacles that need to be addressed are cost, limitations of currently available tools, specialized technical skills needed to use specific tools, as well as musician concerns about how the use of AI may impact their creativity, intellectual property, and professional identity.

Two essential things need to happen to unleash AI’s full potential in music production. First, more robust and user-friendly AI-powered tools to support various aspects of music production need to be created and disseminated. At the same time, researchers continue to create novel, deep-learning AI models, so this is only a step toward building user-friendly tools that most musicians can use. Second, we need to empower more musicians to use these new tools in creative and transformative ways in their everyday endeavors.

The project, titled “Toward an Ecosystem of Artificial-intelligence-powered Music Production” (TEAMuP), proposes to achieve these goals by working in the following complementary areas:

• Creating an open-access framework enabling musicians and AI researchers to collaborate in developing new music production solutions, which will run on Audacity (an open-source and free DAW that more than two million people have used to date).
• Understanding both advances and challenges that may arise from using AI for music production.
• Providing new learning opportunities to empower current and future musicians to utilize technology better.

After conducting and analyzing interviews and surveys with a diverse group of musicians, the University of Rochester team will also develop an education and outreach program to help practicing musicians incorporate deep learning tools into their workflow. The education component of TEAMuP includes a two-semester music and technology course, a summer camp designed for underrepresented students, and online instructional materials.

Through this project, the research team aims to increase the democratization of music production by helping musicians become self-sufficient in their music creation. The open-access framework they will develop will also accelerate research in audio engineering by providing a vehicle to effectively deploy and refine new AI models for music production. Insights gained from this project will be relevant for many other occupations at the human-technology frontier besides musicians. Those insights include how to create more ethical and user-friendly AI-powered products, develop the mindsets and skills needed by domain specialists to leverage technology, and understand the implications of the pandemic.