Multimodal engagement level recognition in casual conversition

Abstract

Automatic detection of engagement in human-human and human-machine in dyadic and multiparty interaction scenarios is important in evaluating the success of communication. Engagement detection has a wide range of applications including monitoring in spoken dialogue systems, event detection in conversations, and user satisfaction detection using designated devices. However, pre-processing approaches and knowledge of relevant features for engagement level recognition (for example feature normalization and until now unexplored facial features) are required for a robust learning method to model engagement, as well as explore real-time automatic conversational engagement. In this research we sequentially 1/ build an engagement corpus, 2/ analyse the annotation of engagement, 3/ investigate useful non-verbal features and approaches to model engagement dynamically and sequentially. The final goal is to automatically detect and recognize engagement in a spontaneous human-human conversation and human-machine interaction. In this work, engagement is studied using non-verbal features, in particular, visual -- auditory features and machine learning methods for engagement level recognition. The proposed engagement prediction paradigm with advanced results, new features, novel learning models and speaker dependency effects are documented in this thesis.