Piano & AI

The goal of the “Piano & AI” project is to develop and experiment a musical interaction loop between a piano player and an artificial entity. The project envisions a collaborative improvisation between the human pianist and the AI system. The project idea is illustrated in the figure below.

To enable the AI to listen to the pianist, two microphones capture the sound of the piano, and the signal is analyzed using various signal processing and machine learning tools. These tools extract musically relevant parameters such as onset time, pitch, BPM, loudness, and spectral features that characterize the timbre of the instrument. The tools have been trained and optimized on a database specifically recorded by Marco Mezquida for this project and the Google Magenta Maestro Database.

Using this analysis, the AI system synthesizes a musical accompaniment or response to the pianist in real-time. As the pianist hears the AI's musical response, he gets inspired by the AI-generated sound and interacts with the system as he would with another player in an improvisation. To support and structure concerts, the project has created several soundscapes or textures, each of which can use any of the extracted descriptors to react to the pianist's playing. These textures generate a wide range of sonic atmospheres.

Project development: 

The project commenced in mid-2021 as a collaborative effort  between Sónar, UPC and Betevé. Its primary objective was to create the inaugural concert of the "AI and Music Festival" (documentary excerpt). Alongside pianist Marco Mezquida, the technical development of the system was initially undertaken by Philippe Salembier, Josep M. Comajuncosas, Ivan Paz, and Joan Cañelles (Shelly). Most of the rehearsals were conducted at the Escola Superior de Música de Catalunya  (ESMUC). The inaugural concert took place on October 27, 2021, at the Auditori of Barcelona. Since then, the system has undergone further evolution under the supervision of Philippe Salembier and Josep M. Comajuncosas, and has been featured in several concerts throughout 2022 and 2023:

• Sónar Lisbon, Portugal, April 8 2022
• Palau de Pedralbes, Barcelona, May 16 2022, 
• Terrassa principal theater, Terrassa, June  2 2022
• Cosmocaixa, Barcelona, February 17 2023

Barcelona Auditori concert (October 27, 2021) 

Terrassa principal theater concert (June 2, 2022)

SonarMies Sessions (Barcelona, June 2023): "Reprocessing… Piano + AI"

In 2023, for Sónar’s 30th Anniversary, the project underwent a significant evolution through the “Reprocessing….. Piano & AI” event emphasizing openness and creative experimentation. This event took place at Barcelona's Mies van der Rohe Pavilion. The initiative started with a press presentation on June 13th, led by Marco Mezquida.

Marco Mezquida during the SonarMies press presentation

Subsequently, from June 15th to 17th, the Mies van der Rohe Pavilion hosted six public sessions, presented by Lluís Nacenta. Carles Viarnès, CLARAGUILAR, Marina Herlop, Francesco Tristano, Lluïsa Espigolé, and Adria González (Killin’Cactuz) engaged with the AI for the very first time. During the sessions, each artist discovered, dialogued with and even challenged the AI system. These sessions were full of innovations, sonic explorations and discoveries for the pianists, the technical team in charge of the system and the public. 

Summary of the SonarMies sessions with Carles Viarnès, CLARAGUILAR, Marina Herlop, Francesco Tristano, Lluïsa Espigolé and Killin'Cactuz

Recording of individual sessions:  

  .  

     Carles Viarnès                 CLARAGULAR                    Marina Herlop  

  .  

    Francesco Tristano                Lluïsa Espigolé                    Killn'Cactuz      

Fira Mediterrania (Manresa, October 2023): "Manel Camp, Piano + AI"

In October 2023, a session similar to the SonarMies sessions was organised by during the Fira Mediterrania. The Pianist Manel Camp was invited to interact for the first time with the Piano & AI system. The AI system had the opportunity to discover new musical grounds.  

Manel Camp, Fira Mediterrania