Predicting emotional response to visual stimuli, a machine learning approach

We created an emotion predicting model capable of predicting emotions in images using OpenAI CLIP as a backbone. Using the ArtEmis dataset which contains 80K paintings annotated on the base of perceived emotions (amusement, fear, etc..). We show that this method of predicting emotion is effective, outperforming previous methods in predicting dominant emotion (70% vs 60%) or positive/negative images (80% vs 77.7%). We leverage our method on text emotion prediction which allows to quickly identify the reason why an image has a certain affect. With this method we unveil CLIP’s political preferences, and discovered a lean towards the democrats. We also created an affect based query tool that allows the users to search a set of pictures with a prompt as well as an emotion.