Social Sensing Methods for Analysis of Dyadic Hospitality Encounters

First impressions are critical to professional interactions, especially in service industry like hospitality. In the service industry, customers often assess quality of service based on the behavior, perceived personality, and other attributes of the front-line service employees they interact with. Interpersonal communication during these interactions is thus key to determine customer satisfaction and perceived service quality. Given the importance of first impressions in hospitality, this thesis contributes to the implementation of a behavioral training framework for hospitality students with an aim of improving the impressions that other people make about them in workplaces. We outline the challenges associated with designing such a framework and embedding it in the everyday practice of a real hospitality school. These behavioral training sessions were recorded based on principles of unobtrusive measurements and social signal processing. We collect a dataset of 169 laboratory plays; job interviews and reception desk scenarios, for a total of 338 interactions. In our first study of the job interview scenario, we evaluate the relationship between automatically extracted verbal and nonverbal cues with the various manually annotated impressions of social variables in a correlation analysis. We then develop methods to automatically infer first impressions using verbal cues, nonverbal features and their combination. Our inference results indicate that nonverbal features outperform verbal cues in an inference task. Best inference performance is obtained by fusion of verbal and nonverbal cues. A gender based analysis reveals important differences between males and females in terms of nonverbal cues displayed and impressions formed. This result is corroborating previous findings in psychology. In our second study we investigate the reception desk interaction. We aimed to develop a computational framework to automatically infer perceived performance and skill variables using nonverbal and verbal behavior displayed. We also study the connections between receptionists’ impressions of Big-5 personality traits, attractiveness, and performance. Our results indicate the feasibility of inferring perceived job performance from nonverbal and verbal cues displayed. Furthermore, contrary to our hypothesis, perceived attractiveness had low predictive power of first impressions. We then conduct a cross-situation analysis to understand the impact of situation in the formation of first impressions. This is based on the truism in psychology that same people behave differently in different situations. A correlation analysis reveals connection between perceived variables and nonverbal cues displayed during job interviews, and perceived performance on the job. We develop a computational framework to infer the perceived performance and soft skills in the reception desk situation with nonverbal cues and linguistic style from the two interactions as predictors. The best inference performance is achieved by fusing nonverbal cues displayed during the reception desk setting and the human-rated interview scores. We observe that some behavioral cues (greater speaking turn duration and head nods) are positively correlated to higher ratings for all perceived variables across both situations. This is one of the major contributions of this thesis.