AI Affective Virtual Human

Research Collaborators
Steve DiPaola , Nilay Ozge Yalcin , Ulysses Bernardet , Maryam Saberi , Michael Nixon

Our open source toolkit / cognitive research in AI 3D Virtual Human (embodied IVA :Intelligence Virtual Agents) : a real-time system that can respond emotionally (voice, facial animation, gesture, etc) to a user in front of it via a host of gestural, motion and bio- sensor systems, with several in lab AI systems able to detect speech from the user in front of it and give a coherent answers via speech, expression and gesture. The system uses Unity and SmartBody (USC) API who we have collaborated with for years.

The Research
Our affective real-time 3D AI virtual human setup with face emotion recognition, movement recognition and data glove recognition.

The growing success of dialogue systems research makes conversational agents a perfect candidate for becoming a standard in human computer interaction. The naturalness of communicative acts allows for providing a comfortable ground for the users to interact with. There have been many advances on using multiple communication channels in dialogue systems in the way of simulating humaneness in an artificial agent.

However, one issue is to be able to find a good balance of the intensity and frequency of multimodal affective feedback to guide the dialogue flow. The timing and the quality of the feedback can have varying effects on different users. We use conversational mirroring mechanisms to generate baselines for interaction, that can be used to dynamically guide the user towards the end goal.

Our ongoing research project studies human behavior while interacting with these assistive technologies using natural interaction methods to create an Embodied Conversational Agent (ECA) that enables users to efficiently achieve their goals. We have a working prototype of our ECA using Deep Learning AI and NLP (Natural Language) systems – the goal is to build up from this basic system a strong conversation process that can fully understand, analyze, build a user model and with it converse effectively with users.

Downloads and Links
PDF: BICA Journal  AI Avatar : Journal: A Computational Model of Empathy for Interactive Agents (BICA 18) Winner: Research Award Paper
PDF: Stanford Poster AI Avatar : Poster from Stanford VR and Behavioral change Conference 2017.
PDF: IVA 2016a Simulink Toolbox for Real-time VirtualCharacter Control
PDF: IVA 2016b  An Architecture for Biologically Grounded Real-time Reflexive Behavior
PDF: IVA 2015  A Framework for Exogenous and Endogenous Reflexive Behavior in Virtual Characters

Additional Media and Code
Media / Code Repository Repository of Code and Media for our RealAct system