Reliability and Validity of Nonverbal Thin Slices in Social Interactions

Denise Frauendorfer, Laurent Son Nguyen
2015
Article

Résumé

Four studies investigated the reliability and validity of thin slices of nonverbal behavior from social interactions including (a) how well individual slices of a given behavior predict other slices in the same interaction; (b) how well a slice of a given behavior represents the entirety of that behavior within an interaction; (c) how long a slice is necessary to sufficiently represent the entirety of a behavior within an interaction; (d) which slices best capture the entirety of behavior, across different behaviors; and (e) which behaviors (of six measured behaviors) are best captured by slices. Notable findings included strong reliability and validity for thin slices of gaze and nods, and that a 1.5-min slice from the start of an interaction may adequately represent some behaviors. Results provide useful information to researchers making decisions about slice measurement of behavior.

Official source

https://infoscience.epfl.ch/record/218118?ln=fr

À propos de ce résultat

Cette page est générée automatiquement et peut contenir des informations qui ne sont pas correctes, complètes, à jour ou pertinentes par rapport à votre recherche. Il en va de même pour toutes les autres pages de ce site. Veillez à vérifier les informations auprès des sources officielles de l'EPFL.

Concepts associés

Chargement

Publications associées

Chargement

Computational Analysis Of Behavior In Employment Interviews And Video Resumes

Laurent Son Nguyen

Used in nearly every organization, employment interviews are a ubiquitous process where job applicants are evaluated by an employer for an open position. Consisting of an interpersonal interaction between at least one interviewer and a job applicant, they are used to assess interviewee knowledge, skills, abilities, and behavior in order to select the most suitable person for the job at hand. Because they require face-to-face interaction between at least two protagonists, they are inherently social, and all that recruiters have as a basis to forge their opinion is the applicant's behavior during the interview (in addition to his resume); in such settings, first impressions are known to play an important role. First impressions can be defined as snap judgments of others made based on a low amount of information. Interestingly, social psychology research has shown that humans are quite accurate at making inferences about others, even if the information is minimal. Social psychologists long studied job interviews, with the aim of understanding the relationships between behavior, interview outcomes, and job performance. Until recently, psychology studies relied on the use of time-intensive manual annotations by human observers. However, the advent of inexpensive audio and video sensors in the last decade, in conjunction with improved perceptual processing methods, has enabled the automatic and accurate extraction of behavioral cues, facilitating the conduct of social psychology studies. The use of automatically extracted nonverbal cues in combination with machine learning inference techniques has led to promising computational methods for the automatic prediction of individual and group social variables such as personality, emergent leadership, or dominance. In this thesis, we addressed the problem of automatically predicting hirability impressions from interview recordings by investigating three main aspects. First, we explored the use of state-of-the-art computational methods for the automatic extraction of nonverbal cues. As a rationale for selecting the behavioral features to be extracted, we reviewed the psychology literature for nonverbal cues which were shown to play a role in job interviews. While the main focus of this thesis is nonverbal behavior, we also investigated the use of verbal content and standard questionnaire outputs. Also, we did not limit ourselves to the use of existing techniques: we developed a multimodal nodding detection method based on previous findings in psychology stating that head gestures are conditioned on the speaking status of the person under analysis, and results showed that considering the speaking status improved the accuracy. Second, we investigated the use of supervised machine learning techniques for the prediction of hirability impressions in a regression task, and up to 36% of the variance could be explained, demonstrating that the automatic inference of hirability is a promising task. Finally, we analyzed the predictive validity of thin slices, short segments of interaction, and showed that short excerpts of job interviews could be predictive of the outcome, with up to 34% of the variance explained by nonverbal behavior extracted from thin slices. As another trend, online social media is changing the landscape of personnel recruitment. Until now, resumes were among the most widely used tools for the screening of job applicants. [...]

EPFL2015

Chargement

The computational modeling of face-to-face interactions using nonverbal behavioral cues is an emerging and relevant problem in social computing. Studying face-to-face interactions in small groups helps in understanding the basic processes of individual and group behavior; and improving team productivity and satisfaction in the modern workplace. Apart from the verbal channel, nonverbal behavioral cues form a rich communication channel through which people infer – often automatically and unconsciously – emotions, relationships, and traits of fellowmembers. There exists a solid body of knowledge about small groups and the multimodal nature of the nonverbal phenomenon in social psychology and nonverbal communication. However, the problem has only recently begun to be studied in the multimodal processing community. A recent trend is to analyze these interactions in the context of face-to-face group conversations, using multiple sensors and make inferences automatically without the need of a human expert. These problems can be formulated in a machine learning framework involving the extraction of relevant audio, video features and the design of supervised or unsupervised learning models. While attempting to bridge social psychology, perception, and machine learning, certain factors have to be considered. Firstly, various group conversation patterns emerge at different time-scales. For example, turn-taking patterns evolve over shorter time scales, whereas dominance or group-interest trends get established over larger time scales. Secondly, a set of audio and visual cues that are not only relevant but also robustly computable need to be chosen. Thirdly, unlike typical machine learning problems where ground truth is well defined, interaction modeling involves data annotation that needs to factor in inter-annotator variability. Finally, principled ways of integrating the multimodal cues have to be investigated. In the thesis, we have investigated individual social constructs in small groups like dominance and status (two facets of the so-called vertical dimension of social relations). In the first part of this work, we have investigated how dominance perceived by external observers can be estimated by different nonverbal audio and video cues, and affected by annotator variability, the estimationmethod, and the exact task involved. In the second part, we jointly study perceived dominance and role-based status to understand whether dominant people are the ones with high status and whether dominance and status in small-group conversations be automatically explained by the same nonverbal cues. We employ speaking activity, visual activity, and visual attention cues for both the works. In the second part of the thesis, we have investigated group social constructs using both supervised and unsupervised approaches. We first propose a novel framework to characterize groups. The two-layer framework consists of a individual layer and the group layer. At the individual layer, the floor-occupation patterns of the individuals are captured. At the group layer, the identity information of the individuals is not used. We define group cues by aggregating individual cues over time and person, and use them to classify group conversational contexts – cooperative vs competitive and brainstorming vs decision-making. We then propose a framework to discover group interaction patterns using probabilistic topicmodels. An objective evaluation of ourmethodology involving human judgment and multiple annotators, showed that the learned topics indeed are meaningful, and also that the discovered patterns resemble prototypical leadership styles – autocratic, participative, and free-rein – proposed in social psychology.

EPFL2011

Chargement

Machine Vision-Based Analysis of Gaze and Visual Context

Basilio Noris

Visual behavior, and specifically gaze directed at objects and people, is a fundamental cue for understanding how children with Autism Spectrum Disorder (ASD) experience and respond to social interaction. Indeed atypical behaviors such as averting the gaze from faces, looking out of the corner of the eyes, and having difficulties disengaging from non-social stimuli, are well known symptoms of this developmental disorder. However, studying these atypicalities presents several technical challenges, both at a hardware and software level. Traditionally this type of analysis is done by viewing video recordings of the child's interaction and manually rating gaze behavior episodes. This data collection procedure is often a time consuming process, and all the results have to be checked by multiple raters to ensure reliability. When automated systems are used, issues of intrusiveness, robustness and reliability have to be taken into consideration, and their impact on the behavior of the child needs to be accounted for. Moreover, children often do not cooperate to the same extent as an adult participant in the experimental process. This thesis addresses the problem of studying the visual behavior of children during dyadic interactions, using tools from machine learning and computer vision. One focus is on the technical aspects of how to measure the direction of the gaze, and how to recognize what the child is looking at. Another is on the development of semi-automated data analysis collection methods that can be efficiently checked and corrected by the experimenter. The concrete contribution of this work is a set of algorithms and software tools that allow firstly the measurement of multiple gaze factors and the quantification of attentional episodes to objects and people, and secondly the analysis of their distributions, extrema, and spatio-temporal correlations. These tools are applied to the specific case of children affected by ASD, which allows the assessment of how their gaze strategies differ from typically developing children. To measure the direction of the gaze of children in a robust and unobtrusive way, the main recording tool employed is the WearCam: a head-mounted camera developed in our laboratory, that records the eyes of the child and an image of the broad field of view. The reflected images of the eyes are mapped to the coordinates of the gaze in the field of view by means of data-driven appearance-based methods. This is a different approach to the standard methods for gaze tracking, as it makes no use of active lighting, and allows the gathering of data to be postponed until after the recording. This is especially useful as it relieves the children from actively participating in a calibration phase. The task of identifying at which object or person the child is looking is approached through a semi-automated analysis of the recordings. A computationally intensive face and object detection system provides a preliminary estimation, while a faster on-line method can be used by the experimenter to correct that estimation by providing active cues. Furthermore, to allow faster processing of the recordings, a set of features specific to the relevant objects is extracted. The combination of these approaches greatly reduces the time needed to collect the data, while ensuring that reliable data is obtained. The development of these tools facilitates the studying of the behavior of children during natural interactions with an adult. Our approach is able to obtain information on several gaze factors, such as the frequency and length of fixations to faces, how extensively the gaze explores the broad field of view and the use of central vs. peripheral vision. The experimental observations obtained from the comparison of 29 children with autism to 29 typically developing children provide, in two separate studies, quantitative measurements in support of known deficits, such as reduced gaze to faces, downcast gaze and shorter fixations to social stimuli; deficits that were so far documented only in a qualitative manner. More importantly, the results highlight a novel hypothesis, not yet documented, suggesting that ASD children may use a strategy of continuous peripheral viewing to overcome their visual processing difficulties. These observations thus provide an experimental contribution to clinical research in autism.

EPFL2011

Chargement

Computational Analysis Of Behavior In Employment Interviews And Video Resumes

Laurent Son Nguyen

EPFL2015

Machine Vision-Based Analysis of Gaze and Visual Context

Basilio Noris

EPFL2011