How CanWe Evaluate Object Recognition Algorithms Using a Public Object Image Database

In this paper, to evaluate the performance of object recognition algorithms, we propose a new evaluation framework by synthesizing natural scenes based on the Amsterdam Library of Object Images [1]. Here, the evaluation of an object recognition algorithm has the basis on searching an object in a synthetic scene. More specifically, an object is selected, and then the synthetic scene under a specific condition is generated by using images, affected by that condition, of that object and other objects in the database. When generating the synthetic scene, the other objects are randomly selected and all objects are naturally distributed in the synthetic scene. Let us refer to this synthetic scene as a Virtual Scene. Then the performance of object recognition algorithms for the specific condition can be analyzed by using a group of Virtual Scenes in that condition. As an example of utilizing the proposed framework, an object recognition algorithm using the scale-invariant feature transform [2] has been evaluated and analyzed in the case of changing the viewing direction, illumination color, and illumination direction.