About
Can you extract the emotional aspects of a piece of music to animate a face. Music-driven Emotionally Expressive Face (MusicFace) is a early stage project that creates “facial choreography” driven by musical input. In addition to its artistic uses, MusicFace can be used for creating visual effects in movies and animations, and also realistic characters in computer games and virtual worlds.

MusicFace starts by receiving the input music in MIDI format. Structure and expressive cues such as tempo, loudness, and timing are combined with emotion color cues from extracting harmonic tension and rhythm, which then can be translated into emotional states. A rule-based fuzzy system is responsible for “re-mapping” the aesthetic communication stream to facial emotions and actions (e.g. blinking).

MusicFace Animation:

Selected frames from the animation “Concerto for Virtual Strings and Faces” by Steve DiPaola – an early piece using musicFace where the facial animation was completely driven by an original musical composition. Structural and emotion data was extracted from the music via a fuzzy rule based system and remapped to emotional and structure “equivalents” in face space. The image above is also a frame that shows NPR techniques applied to the same animation.

Affective Communication Remapping. Affective forms are extracted from one
communication source (here: music) and then remapped to a new target destination (animated faces).

This work will soon be presented in more detailed form at EVA conference in London, where we will present the paper “Music-Driven Emotionally Expressive Faces”. It will also be realized in art work displayed on a 6 ft projected globe at the A.I.R Gallery in New York City and at the Siggraph Art Show 2004. Contact us for more details.

This first “realistic” piece (left) was created for and shown at New Forms Festival’03 in Vancouver. The “artistic” piece (right) is the same animation using non-photo realistic 3d rendering techniques which showcase how this technique can be used for “music paintings”, interactive and responsive set design (using projected imagery) for theatre and dance, and other music generated visual work.
Following is the original New Forms Festival proposal that speaks to our motivation:

New Forms Festival proposal

Following is the original New Forms Festival proposal that speaks to our motivation:
Concerto for Virtual Strings and Faces
– a presentation and screening

By Steve DiPaola & Scot Gresham-Lancaster
– concept, programming & animation by Steve DiPaola
– original score and special edit by Scot Gresham-Lancaster

Overview:
This is a collaborative piece between an electronic composer and an interactive 3D artist where an original electronic music composition will drive 3d virtual emotive faces in a collaborative re-mapping of the emotional and gestural state of both the music and the lifelike facial animation. It is based on a system which explores “face space” which has been used in several multidisciplinary collaborations (i.e. art, internet communication, electronic games, social science) but this New Forms proposal will mark the first collaborative experiment in the relationship of “choreography in human emotion face space” via artistically remapping/interpreting/converting the emotive stream of the musical instruments line of an original composition.

Background and Motivation:
For many years, Steve DiPaola has been creating and using a visual development system for exploring face space, both in terms of facial types and animated emotional expressions. See the following paper. Within this multi-dimensional space of faces, it would be possible to traverse a path from any face to any other face, morphing through faces along that path. Predecessors of this system have been used (via commissioned work) for best selling electronic game “The Sims” and in a collaboration with the video artist Nam June Paik and in music videos for Kraftwerk. This system has been extended into emotion and gesture space making it possible to build a space language of facial emotion and behavior.

The system allows us to traverse morphing face paths through face space, (both facial types and emotion types) with the ability to affect the trajectory of path (straight, curved) and the key points (faces) to target along the way. Music can be described as paths through tonal space with significant cultural biases described by certain paths or key points (i.e. notes, intervals, scales) determining whether certain motifs sound appealing. Can we similarly write a composition in face space? Imagine a representation of two intertwining paths through face space that use their relationships, both harmonic and discordant, to create a composition. Can we dance with faces, both moving through facial types as well as ebb and flowing through emotional gestures?

It is this idea – dancing or composing in face space that is the spark for this collaboration. We will be both be exploring our artistic domains (Scot’s electronic music composition and Steve’s interactive face-based emotion composition) to combine them into one synergistic movement, much as a dance choreographer and a composer would do. Except with our interactive knowledge based tools, we can choreograph an “living” emotive faces, creating a process that interprets and translates the emotion state of the composed music, where each instrument would drive an aspect of emotive faces.

For additional information see the musicface page on dipaola.org