Facing Employers and Customers: What Do Gaze and Expressions Tell About Soft Skills?

Eye gaze and facial expressions are central to face-to-face social interactions. These behavioral cues and their connections to first impressions have been widely studied in psychology and computing literature, but limited to a single situation. Utilizing ubiquitous multimodal sensors coupled with advances in computer vision and machine learning, we investigate the connections between these behavioral cues and perceived soft skills in two diverse workplace situations (job interviews and reception desk). Pearson's correlation analysis shows a moderate connection between certain facial expressions, eye gaze cues and perceived soft skills in job interviews (r is an element of [-30,30]) and desk (r is an element of [20,36]) situations. Results of our computational framework to infer perceived soft skills indicates a low predictive power of eye gaze, facial expressions, and their combination in both interviews (R-2 is an element of [0.02,0.21]) and desk (R-2 is an element of [0.05, 0.15]) situations. Our work has important implications for employee training and behavioral feedback systems.