Supplementing Omitted Named Entities in Cooking Procedural Text with Attached Images

In this research, we aim at supplementing named entities, such as food, omitted in the procedural text of recipe data. It helps users understand the recipe and is also necessary for the machine to understand the recipe data automatically. The contribution of this research is as follows. (1) We construct a dataset of Chinese recipes consisting of 12,548 recipes. To detect sentences in which food entities are omitted, we label named entities such as food, tool, and cooking actions in the procedural text by using the automatic recipe named entity recognition method. (2) We propose a method of recognizing food from the attached images. A procedural text of recipe data is often associated with an image, and the attached image often contains the food even when it is omitted in the procedural text. Tool entities in images in recipe data can be identified with high accuracy by conventional general object recognition techniques. On the other hand, the general object recognition methods in the literature, which assume that the properties of an object are constant, perform not well for food in recipe image data because food states change during cooking procedures. To solve this problem, we propose a method of obtaining food entity candidates from other steps that are similar to the target step, both in sentence similarity and image feature similarity. Among all the 246,195 procedural steps in our dataset, there are 16,593 steps in which the food entity is omitted in the procedural text. Our method is applied to supplement the food entities in these steps and achieves the accuracy of 67.55%.