Improving Facial Expression Recognition Through Data Preparation and Merging

Abstract

Human emotions present a major challenge for artificial intelligence. Automated emotion recognition based on facial expressions is important to robotics, medicine, psychology, education, security, arts, entertainment and more. Deep learning is promising for capturing complex emotional features. However, there is no training dataset that is large and representative of the full diversity of emotional expressions in all populations and contexts. Current facial datasets are incomplete, biased, unbalanced, error-prone and have different properties. Models learn these limitations and become dependent on specific datasets, hindering their ability to generalize to new data or real-world scenarios. Our work addresses these difficulties and provides the following contributions to improve emotion recognition: 1) a methodology for merging disparate in-the-wild datasets that increases the number of images and enriches the diversity of people, gestures, and attributes of resolution, color, background, lighting and image format; 2) a balanced, unbiased, and well-labeled evaluator dataset, built with a gender, age, and ethnicity predictor and the successful Stable Diffusion model. Single- and cross-dataset experimentation show that our method increases the generalization of the FER2013, NHFI and AffectNet datasets by 13.93%, 24.17% and 7.45%, respectively; and 3) we propose the first and largest artificial emotion dataset, which can complement real datasets in tasks related to facial expression.