About
Our open-source toolkit / cognitive research in AI 3D Virtual Human (embodied IVA : Intelligence Virtual Agents) : a real-time system that can converse with a human by sensing their emotions and conversation ( via facial emotion recognition, voice stress, semantics of the speech and words) and respond affectively, emotionally (voice, facial animation, gesture, etc) to a user in front of it via a host of gestural, motion and bio-sensor systems, with several in lab AI systems and give a coherent, personality-based conversational answers via speech, expression and gesture. The system uses Unity and SmartBody (USC) API who we have collaborated with for years. We use cognitive modeling, empathy modelling, NLP and a variety of AI-based modules in our system (see papers).

The Research
Our affective real-time 3D AI virtual human setup with face emotion recognition, movement recognition and data glove recognition. See overview video or specific videos or papers below.

The growing success of dialogue systems research makes conversational agents a perfect candidate for becoming a standard in human-computer interaction. The naturalness of communicative acts allows for providing a comfortable ground for the users to interact with. There have been many advances on using multiple communication channels in dialogue systems in the way of simulating humaneness in an artificial agent.

However, one issue is to be able to find a good balance of the intensity and frequency of multimodal affective feedback to guide the dialogue flow. The timing and quality of the feedback can have varying effects on different users. We use conversational mirroring mechanisms to generate baselines for interaction, that can be used to dynamically guide the user towards the end goal.

Our ongoing research project studies human behavior while interacting with these assistive technologies using natural interaction methods to create an Embodied Conversational Agent (ECA) that enables users to efficiently achieve their goals. We have ongoing studies and updates of our ECA using Deep Learning AI and NLP (Natural Language) systems – the goal is to build up from this basic system a strong conversation process that can fully understand, analyze, build a user model and with it converse effectively with users.