A Development Framework for Automated Facial Expression Recognition Systems

Automated facial expression recognition (AFER) has become an important research area with several computer vision (CV) applications. A robust AFER system requires sufficient good quality training and testing data for development and evaluation of a robust AFER model. There exist a number of AFER datasets and an increasing number of research works in AFER. However, research works in AFER have not matured to a stage that there are openly available platforms or toolsets to implement the pipeline of AFER system development. New comers to the field are faced with various challenges such as 1) images in the datasets are messy or with low resolutions; 2) the data are not organized into separate training and testing data for fair evaluation; 3) majority of the datasets are very small leading to insufficiency for training a model; 4) some datasets do not provide important facial features, 5) it is unclear which dataset to start with, and 6) no development framework and methodologies to systematically implement and test new models. In this paper, we present a framework with complete source code and algorithms to: 1) detect faces and crop face images in a given dataset for AFER; 2) extract facial landmark features from the face images and store as landmark images; 3) split the dataset into training and testing sets and stored into two CSV files consisting filename, emotion, landmarks, and features vectors for each image in its respective set; 4) train and evaluate the features vectors of the dataset using a deep neural network (DNN) model as the baseline; 5) train and evaluate a baseline convolutional neural network (CNN) on the face cropped images; 6) demonstrate the trained model on live videos and images. This study also outlines the necessary steps involved in developing an AFER system. The framework can help researchers to use a dataset to develop AFER systems and further improve the framework and benchmark the results.