Ambiance in Social Media Venues: Visual Cue Interpretation by Machines and Crowds

We study the perception of ambiance of places captured in social media images by both machines and crowdworkers. This task is challenging due to the subjective nature of the ambiance construct as well as the large variety in layout, style, and visual characteristics of venues. For machine recognition of ambiance, we use state-of-the-art Residual Deep Convolutional Neural Networks (ResNets), followed by gradient-weighted class activation mapping (Grad-CAM) visualizations. This form of visual explanation obtained from the trained ResNet-50 models were assessed by crowdworkers based on a carefully designed crowdsourcing task, in which both visual ambiance cues of venues and subjective assessment of Grad-CAM results were collected and analyzed. The results show that paintings, photos, and decorative items are strong cues for artsy ambiance, whereas type of utensils, type of lamps and presence of flowers may indicate formal ambiance. Layout and design-related cues such as type of chairs, type of tables/tablecloth and type of windows are noted to have impact for both ambiances. Overall, the ambiance visual cue recognition results are promising, and the crowd-based assessment approach may motivate other studies on subjective perception of place attributes.

The YouTube Lens: Crowdsourced Personality Impressions and Audiovisual Analysis of Vlogs

Joan Isaac Biel Tres, Daniel Gatica-Perez

Despite an increasing interest in understanding human perception in social media through the automatic analysis of users' personality, existing attempts have explored user profiles and text blog data only. We approach the study of personality impressions in social media from the novel perspective of crowdsourced impressions, social attention, and audiovisual behavioral analysis on slices of conversational vlogs extracted from YouTube. Conversational vlogs are a unique case study to understand users in social media, as vloggers implicitly or explicitly share information about themselves that words, either written or spoken cannot convey. In addition, research in vlogs may become a fertile ground for the study of video interactions, as conversational video expands to innovative applications. In this work, we first investigate the feasibility of crowdsourcing personality impressions from vlogging as a way to obtain judgements from a variate audience that consumes social media video. Then, we explore how these personality impressions mediate the online video watching experience and relate to measures of attention in YouTube. Finally, we investigate on the use of automatic nonverbal cues as a suitable lens through which impressions are made, and we address the task of automatic prediction of vloggers' personality impressions using nonverbal cues and machine learning techniques. Our study, conducted on a dataset of 442 YouTube vlogs and 2210 annotations collected in Amazon's Mechanical Turk, provides new findings regarding the suitability of collecting personality impressions from crowdsourcing, the types of personality impressions that emerge through vlogging, their association with social attention, and the level of utilization of nonverbal cues in this particular setting. In addition, it constitutes a first attempt to address the task of automatic vlogger personality impression prediction using nonverbal cues, with promising results.

Ieee-Inst Electrical Electronics Engineers Inc2013

Loud and Trendy: Crowdsourcing Impressions of Social Ambiance in Popular Indoor Urban Places

Daniel Gatica-Perez, Darshan Santani

New research cutting across architecture, urban studies, and psychology is contextualizing the understanding of urban spaces according to the perceptions of their inhabitants. One fundamental construct that relates place and experience is ambiance, which is defined as "the mood or feeling associated with a particular place". We posit that the systematic study of ambiance dimensions in cities is a new domain for which multimedia research can make pivotal contributions. We present a study to examine how images collected from social media can be used for the crowdsourced characterization of indoor ambiance impressions in popular urban places. We design a crowdsourcing framework to understand suitability of social images as data source to convey place ambiance, to examine what type of images are most suitable to describe ambiance, and to assess how people perceive places socially from the perspective of ambiance along 13 dimensions. Our study is based on 50,000 Foursquare images collected from 300 popular places across six cities worldwide. The results show that reliable estimates of ambiance can be obtained for several of the dimensions. Furthermore, we found that most aggregate impressions of ambiance are similar across popular places in all studied cities. We conclude by presenting a multidisciplinary research agenda for future research in this domain. © 2015 ACM.

ACM2015

Computational Analysis of Urban Places Using Mobile Crowdsensing

Darshan Santani

In cities, urban places provide a socio-cultural habitat for people to counterbalance the daily grind of urban life, an environment away from home and work. Places provide an environment for people to communicate, share perspectives, and in the process form new social connections. Due to the active role of places to the social fabric of city life, it is important to understand how people perceive and experience places. One fundamental construct that relates place and experience is ambiance, i.e., the impressions we ubiquitously form when we go out. Young people are key actors of urban life, specially at night, and as such play an equal role in co-creating and appropriating the urban space. Understanding how places and their youth inhabitants interact at night is a relevant urban issue. Until recently, our ability to assess the visual and perceptual qualities of urban spaces and to study the dynamics surrounding youth experiences in those spaces have been limited partly due to the lack of quantitative data. However, the growth of computational methods and tools including sensor-rich mobile devices, social multimedia platforms, and crowdsourcing tools have opened ways to measure urban perception at scale, and to deepen our understanding of nightlife as experienced by young people. In this thesis, as a first contribution, we present the design, implementation and computational analysis of four mobile crowdsensing studies involving youth populations from various countries to understand and infer phenomena related to urban places and people. We gathered a variety of explicit and implicit crowdsourced data including mobile sensor data and logs, survey responses, and multimedia content (images and videos) from hundreds of crowdworkers and thousands of users of mobile social networks. Second, we showed how crowdsensed images can be used for the computational characterization and analysis of urban perception in indoor and outdoor places. For both place types, urban perception impressions were elicited for several physical and psychological constructs using online crowdsourcing. Using low-level and deep learning features extracted from images, we automatically inferred crowdsourced judgments of indoor ambiance with a maximum R2 of 0.53 and outdoor perception with a maximum R2 of 0.49. Third, we demonstrated the feasibility to collect rich contextual data to study the physical mobility, activities, ambiance context, and social patterns of youth nightlife behavior. Fourth, using supervised machine learning techniques, we automatically classified drinking behavior of young people in an urban, real nightlife setting. Using features extracted from mobile sensor data and application logs, we obtained an overall accuracy of 76.7%. While this thesis contributes towards understanding urban perception and youth nightlife patterns in specific contexts, our research also contributes towards the computational understanding of urban places at scale with high spatial and temporal resolution, using a combination of mobile crowdsensing, social media, machine learning, multimedia analysis, and online crowdsourcing.