Generalization and Personalization of Machine Learning for Multimodal Mobile Sensing in Everyday Life

A range of behavioral and contextual factors, including eating and drinking behavior, mood, social context, and other daily activities, can significantly impact an individual's quality of life and overall well-being. Therefore, inferring everyday life aspects with the use of smartphone and wearable sensors, also broadly known as mobile sensing, is gaining traction across both clinical and non-clinical populations due to the widespread use of smartphones around the world. Such inferences are of use in mobile health apps, mobile food diaries, and generic mobile apps. However, despite the long-standing promise in the domain, realizing the full potential of models, in the wild, is still far from reality due to two primary deployment challenges: the generalization and personalization of models. In addition, there are understudied domains, such as eating and drinking behavior modeling with multimodal mobile sensing and machine learning. Hence, this thesis delves into the realm of multimodal mobile sensing with an eye for the generalization and personalization of models, exploring a range of novel inferences at the intersection of eating and drinking behavior, mood, daily activities, and context.After introducing the topic in the first chapter and discussing data collection in the second, we expand on passive sensing for drink behavior modeling using multimodal sensor data in the third chapter. The fourth chapter demonstrates how smartphone sensors can infer self-perceived food consumption levels with personalized models. The fifth chapter showcases how phone sensors could be used to infer eating events with personalized models. The sixth chapter highlights the challenge of generic mood inference models struggling to adapt to specific contexts like eating. To tackle this, we propose a personalization technique to enhance model performance even with limited data. In the next three chapters, we delve further into the realm of model generalization within the context of multimodal mobile sensing. We also investigate the impact of personalization on generalization performance. Specifically, we investigate model generalization across countries---a problem that has been scarcely addressed in prior research. To this end, in the seventh chapter, we examine the generalization capabilities of mood inference models, while the eighth chapter focuses on the generalization of models for complex daily activity recognition. Upon highlighting the limitations of model generalization in the aforementioned chapters, we introduce a novel technique to enhance model generalization in the context of multimodal sensor data in the ninth chapter.In summary, this thesis offers an extensive exploration of novel inferences and deployment challenges in multimodal mobile sensing. First, the thesis explores eating and drinking behavior and its interplay with mood, social context, and daily activities, viewed through the lens of both model personalization and generalization. Additionally, the thesis delves into the challenge of cross-country generalization for mobile sensing-based models and presents a novel deep learning architecture for unsupervised domain adaptation, yielding enhanced performance in unfamiliar domains. As a result, this thesis contributes both empirically and methodologically to the fields of ubiquitous and mobile computing and digital health.