Simulation of nonverbal social interaction and small groups dynamics in virtual environments

How can the behaviour of humans who interact with other humans be simulated in virtual environments? This thesis investigates the issue by proposing a number of dedicated models, computer languages, software architectures, and specifications of computational components. It relies on a large knowledge base from the social sciences, which offers concepts, descriptions, and classifications that guided the research process. The simulation of nonverbal social interaction and group dynamics in virtual environments can be divided in two main research problems: (1) an action selection problem, where autonomous agents must be made capable of deciding when, with whom, and how they interact according to individual characteristics of themselves and others; and (2) a behavioural animation problem, where, on the basis of the selected interaction, 3D characters must realistically behave in their virtual environment and communicate nonverbally with others by automatically triggering appropriate actions such as facial expressions, gestures, and postural shifts. In order to introduce the problem of action selection in social environments, a high-level architecture for social agents, based on the sociological concepts of role, norm, and value, is first discussed. A model of action selection for members of small groups, based on proactive and reactive motivational components, is then presented. This model relies on a new tagbased language called Social Identity Markup Language (SIML), allowing the rich specification of agents' social identities and relationships. A complementary model controls the simulation of interpersonal relationship development within small groups. The interactions of these two models create a complex system exhibiting emergent properties for the generation of meaningful sequences of social interactions in the temporal dimension. To address the issues related to the visualization of nonverbal interactions, results are presented of an evaluation experiment aimed at identifying the application requirements through an analysis of how real people interact nonverbally in virtual environments. Based on these results, a number of components for MPEG-4 body animation, AML — a tag-based language for the seamless integration and synchronization of facial animation, body animation, and speech — and a high-level interaction visualization service for the VHD++ platform are described. This service simulates the proxemic and kinesic aspects of nonverbal social interactions, and comprises such functionalities as parametric postures, adapters and observation behaviours, the social avoidance of collisions, intelligent approach behaviours, and the calculation of suitable interaction distances and angles.

Mining Conversational Social Video

Joan Isaac Biel Tres

The ubiquity of social media in our daily life, the intense user participation, and the explo- sion of multimedia content have generated an extraordinary interest from computer and social scientists to investigate the traces left by users to understand human behavior online. From this perspective, YouTube can be seen as the largest collection of audiovisual human behavioral data, among which conversational video blogs (vlogs) are one of the basic formats. Conversational vlogs have evolved from the initial "chat from your bedroom" format to a rich form of expression and communication that is expanding to innovative applications where a more natural and engaging way of reaching audiences is either necessary or might be beneficial. This video genre, available online in huge quantities, is a unique scenario for the study and characterization of complex human behavior in social media, that contrarily to social networks, text blogs, and microblogs, has remained unexplored so far. The automatic behavioral understanding of conversational vlogs is a new domain for multimedia research. In short, the goal of our research is the understanding of the processes involved in this social media type, based not only on the verbal channel – what is said – but also on the nonverbal channel – how it is said. The nonverbal channel includes prosody, gaze, facial expression, posture, gesture, etc. and has been studied in depth in the field of nonverbal communication. While the study of vlogging contributes to user behavior research in social media, it also adds to a larger research agenda in social computing by analyzing behavioral data at scales not previously achievable in other scenarios. These type of analysis pose important challenges regarding the development and integration of methods for robust and tractable audiovisual processing. In this thesis, we address the problem of mining user behavior inside conversational videos by addressing three main aspects. First, we integrate state-of-the art audio processing and computer vision techniques to analyze conversational social video. While the initial focus of the thesis is the nonverbal aspect of vlogger behavior, we also investigate the verbal content. Second, we study some of the interpersonal and social processes that link vlogger behavior and vlog consumption in social media platforms such as YouTube. In this context, we examine the phenomenon of social attention in vlogs, and we investigate the use of crowdsourcing as a scalable method to annotate large multimodal corpora with interpersonal perception impres- sions. Finally, we propose a computational framework to predict interpersonal impressions automatically using multimedia analysis, crowdsourced impressions, and machine learning techniques. We anticipate that the work presented in this dissertation will motivate future work in social and behavioral sciences, media analysis, natural language processing, and affective and social computing applied to the large-scale analysis of human interaction in social video.

EPFL2013

Mining Conversational Social Video

Joan Isaac Biel Tres

The ubiquity of social media in our daily life, the intense user participation, and the explosion of multimedia content have generated an extraordinary interest from computer and social scientists to investigate the traces left by users to understand human behavior online. From this perspective, YouTube can be seen as the largest collection of audiovisual human behavioral data, among which conversational video blogs (vlogs) are one of the basic formats. Conversational vlogs have evolved from the initial "chat from your bedroom" format to a rich form of expression and communication that is expanding to innovative applications where a more natural and engaging way of reaching audiences is either necessary or might be beneficial. This video genre, available online in huge quantities, is a unique scenario for the study and characterization of complex human behavior in social media, that contrarily to social networks, text blogs, and microblogs, has remained unexplored so far. The automatic behavioral understanding of conversational vlogs is a new domain for multimedia research. In short, the goal of our research is the understanding of the processes involved in this social media type, based not only on the verbal channel – what is said – but also on the nonverbal channel – how it is said. The nonverbal channel includes prosody, gaze, facial expression, posture, gesture, etc. and has been studied in depth in the field of nonverbal communication. While the study of vlogging contributes to user behavior research in social media, it also adds to a larger research agenda in social computing by analyzing behavioral data at scales not previously achievable in other scenarios. These type of analysis pose important challenges regarding the development and integration of methods for robust and tractable audiovisual processing. In this thesis, we address the problem of mining user behavior inside conversational videos by addressing three main aspects. First, we integrate state-of-the art audio processing and computer vision techniques to analyze conversational social video. While the initial focus of the thesis is the nonverbal aspect of vlogger behavior, we also investigate the verbal content. Second, we study some of the interpersonal and social processes that link vlogger behavior and vlog consumption in social media platforms such as YouTube. In this context, we examine the phenomenon of social attention in vlogs, and we investigate the use of crowdsourcing as a scalable method to annotate large multimodal corpora with interpersonal perception impressions. Finally, we propose a computational framework to predict interpersonal impressions automatically using multimedia analysis, crowdsourced impressions, and machine learning techniques. We anticipate that the work presented in this dissertation will motivate future work in social and behavioral sciences, media analysis, natural language processing, and affective and social computing applied to the large-scale analysis of human interaction in social video.

EPFL2013

Social Interface and Interaction Design for Group Recommender Systems

Yu Chen

Group recommender systems suggest items of interest to a group of people. Traditionally, group recommenders provide recommendations by aggregation the group membersâ preferences. Nowadays, there is a trend of decentralized group recommendation process that leverages the group dynamics and reaches the recommendation goal by allowing group members to influence and persuade each other. So far, the research on group recommender systems mainly focuses on the how to optimize the preference aggregation and enhance the accuracy of recommendations. There is a lack of emphasis on the usersâ social experience, such as interpersonal relationship, emotion exchange, group dynamics, etc. We define the space where user-user interaction occurs in social software as social interfaces. In this thesis, we aim to design and evaluate social interfaces and interactions for group recommender systems. We start with surveying the state-of-the-art of user issues in group recommender systems and interface and interaction design in the broad sense of social applications. We present ten applications and their evaluation via user studies, which lead to a preliminary set of social interface and interaction design guidelines. Based on these guidelines, we develop group recommender systems to investigate the design issues. We then study social interfaces for group recommender systems. We present the design and development ofan experimental platformcalled GroupFun that recommends music to a group of users. We then study the impact of emotion awareness in group recommender systems. More concretely, we design and implement two different methods for emotion awareness: CoFeel and ACTI that visualize emotions using color wheels, and empatheticons that present emotions using dynamic animations of usersâ profile pictures. Our user studies show that emotion awareness tools can help users familiarize with other membersâ preferences, enhance their interpersonal relationships, increase the sense of connectedness in distributed social interactions, and result in higher consensus and satisfaction in group recommendations. We also examine social interactions for persuasive technologies. We design and develop a mobile social game called HealthyTogether that enables dyads to exercise together. With this platform, we study how different social interaction mechanisms, such as social accountability, competition, cooperation, and team spirits, can help usersmotivate and influence each other in physical exercises. We conducted three user studies lasting for up to ten weeks with a total of 80 users. Being accountable for each otherâs performance enhances interpersonal relationships. Supporting users to cooperate on health goals significantly improve their number of steps. When designing competition in the applications, it is crucial to help users to choose comparable buddies. Finally, teamwork in exercises not only helps users to increase their steps, but also help them sustain in exercise. Furthermore, we present an evaluation framework for social persuasive applications. The framework aims at modeling how social strategies and social influence affect user attitudes and behavioral intentions towards the system. Finally, we derive a set of guidelines for social interface and interaction design for group recommender systems. The guidelines can help researchers and practitioners effectively design social experiences for not only group recommenders but also other social software [...]

EPFL2015

Mining Conversational Social Video

Joan Isaac Biel Tres

EPFL2013

Mining Conversational Social Video

Joan Isaac Biel Tres

EPFL2013

Social Interface and Interaction Design for Group Recommender Systems

Yu Chen

EPFL2015