AI Affective Virtual Human
Our open-source toolkit / cognitive research in AI 3D Virtual Human (embodied IVA : Intelligence Virtual Agents) : a real-time system that can converse with a human by sensing their emotions and conversation ( via facial emotion recognition, voice stress, semantics of the speech and words) and respond affectively, emotionally (voice, facial animation, gesture, etc) to a user in front of it via a host of gestural, motion and bio-sensor systems, with several in lab AI systems and give a coherent, personality-based conversational answers via speech, expression and gesture. The system uses Unity and SmartBody (USC) API who we have collaborated with for years. We use cognitive modeling, empathy modelling, NLP and a variety of AI-based modules in our system (see papers).
Our affective real-time 3D AI virtual human setup with face emotion recognition, movement recognition and data glove recognition. See overview video or specific videos or papers below.
The growing success of dialogue systems research makes conversational agents a perfect candidate for becoming a standard in human-computer interaction. The naturalness of communicative acts allows for providing a comfortable ground for the users to interact with. There have been many advances on using multiple communication channels in dialogue systems in the way of simulating humaneness in an artificial agent.
However, one issue is to be able to find a good balance of the intensity and frequency of multimodal affective feedback to guide the dialogue flow. The timing and quality of the feedback can have varying effects on different users. We use conversational mirroring mechanisms to generate baselines for interaction, that can be used to dynamically guide the user towards the end goal.
Our ongoing research project studies human behavior while interacting with these assistive technologies using natural interaction methods to create an Embodied Conversational Agent (ECA) that enables users to efficiently achieve their goals. We have ongoing studies and updates of our ECA using Deep Learning AI and NLP (Natural Language) systems – the goal is to build up from this basic system a strong conversation process that can fully understand, analyze, build a user model and with it converse effectively with users.
Downloads and Links
|Conference Proceeding Link||AI Avatar: BICA 2019 Conference: M-Path: A Conversational System for the Empathic Virtual Agent|
|Journal Link||AI Avatar: Artificial Intelligence Review 2019 Journal: Modeling empathy: building a link between affective and cognitive processes|
|PDF: Conference Paper||AI Avatar:8th International Conference on Affective Computing & Intelligent Interaction: Evaluating Empathy in Artificial Agents|
|PDF: Conference Paper||AI Avatar: 41st Annual Meeting of the Cognitive Science Society: Evaluating Levels of Emotional Contagion with an Embodied Conversational Agent|
|PDF: Conference Paper||AI Avatar: International Conference on Multimodal Interaction 2018 Extended Abstracts: Modeling Empathy in Embodied Conversational Agents|
|PDF: BICA Journal||AI Avatar : Journal: A Computational Model of Empathy for Interactive Agents (BICA 18) Winner: Research Award Paper|
|PDF: Stanford Poster||AI Avatar : Poster from Stanford VR and Behavioral change Conference 2017.|
|PDF: IVA 2016a||Simulink Toolbox for Real-time VirtualCharacter Control|
|PDF: IVA 2016b||An Architecture for Biologically Grounded Real-time Reflexive Behavior|
|PDF: IVA 2015||A Framework for Exogenous and Endogenous Reflexive Behavior in Virtual Characters|
|Additional Media and Code|
|GitHub: M-Path ECA agent M-Path||Repository of Code and Media for our RealAct system|
|Media / Code Repository||Repository of Code and Media for our RealAct system|