Image-based Nutrient Estimation for Chinese Dishes Using Deep Learning

Abstract Food image recognition systems facilitate dietary assessment and in turn track users’ dietary behaviors. However, due to the diversity of Chinese food, a quick and accurate food image recognizing is a particularly challenging task. The success of deep learning in computer vision inspired us to investigate its potential in this task. To satisfy its requirement on large-scale data, we established the first open-access image database for Chinese dishes, named ChinaFood-100, with quantitative nutrient annotations. We collected 10,074 images covering 100 food categories, including staple, meat, seafood, and vegetables. Based on this dataset, we trained four state-of-art deep learning neural network architectures for image recognition and showed that deep learning model Inception V3 resulted in the most advantageous recognition performance 78.26% in top-1 accuracy and 96.62% in top-5 accuracy. Based on this image recognition posterior, we further compared three nutrition estimation algorithms for food nutrient estimation. The results showed that the top-5 Arithmetic Mean (AM) algorithm achieved the highest regression coefficient (R2) up to 0.73 for protein estimation, which validated its applicability in practice. In addition, we analyzed our algorithm in terms of precision-recall and Grad-CAM. The results achieved by deep learning for food nutrient estimation may encourage artificial intelligence to be applied to the field of food, which shed the light on improvement in the future.