Widget detection on screenshots using computer vision and machine learning algorithms

In this paper we consider the problem of detecting and recognizing widgets in screenshots of computer programs’ graphical user interface (GUI). This problem is fundamental in business process automation. The solution we propose here is based on detecting GUI elements with Canny edge operator, and recognizing already detected GUI elements with classifiers: neural networks, random forests, XGBoost, and others.