Musical Interaction Based on the Conductor Metaphor

Interface metaphors are often used in Human Computer Interaction (HCI) to exploit knowledge that users already have from other domains. A commonly used one in Digital Musical Instruments (DMIs) is the conductor metaphor. The simple idea behind it is to turn the computer into an orchestra that the user conducts with movements that resemble those of a real orchestra conductor. So far, many refinements have been proposed to provide more accurate or expressive control over different musical parameters. However, even though the orchestra conducting metaphor offers a good case for investigating several aspects of gesture-based interaction, the way in which users interact with these interfaces has not been explored in depth to improve their usability. The availability of commercial depth-sense cameras, which has stimulated the apparition of new DMIs based on this metaphor, also makes this missing in-depth exploration easier. This dissertation offers such analysis.

We theorize that part of the knowledge that users have from the domain that the interface metaphor replicates is user-specific. In this context, we argue that systems using an interface metaphor can see their usability improved by adapting to this user-specific knowledge. We propose strategies to design motion-sound mappings for DMIs that draw upon the conductor metaphor by adapting to personal nuances that can be automatically computed from spontaneous conducting movements.

For this, we first analyze the performance of a professional conductor in a concert, identifying descriptors than can be computationally extracted from motion capture data and that describe the relationships between the movement of the conductor and specific aspects of the performance potentially controllable in an interactive scenario. Then, we use these same techniques to build two systems that adapt to user-specific tendencies in two contexts. The first one allows to control tempo and dynamics with adaptations learned from analyzing conducting movements performed on top of fixed music. The second one provides control over articulation through gesture variation, the mapping being defined by each user through gesture variation examples. In both cases, we perform observation studies to guide the interface design and user studies with participants of different musical expertise to evaluate the usability of the systems.

In addition to the above, we study the potential of the conductor metaphor in a gaming context as a mean to raise interest for classical music. We developed Becoming the Maestro, a game that exploits state-of-the-art technologies that allow to interact with symphonic music content in new ways. We also perform a user study which shows the potential of the game to increase curiosity for classical music.

In summary, this thesis offers an in-depth exploration of interaction with interfaces based on the conductor metaphor, proposing strategies to improve their usability that span to other interface metaphor cases and to gesture-based interaction in general. These contributions are complemented by the data collected in all observation studies, which is made publicly available to the community.