Using machine learning for cognitive Robotic Process Automation (RPA)

There are many business routine tasks and processes which are performed by qualified resources which can be reallocated, allowing qualified workers to dedicate their effort to other fields/tasks with more relevance. With Robotic Process Automation (RPA), repetitive tasks, even complex computer routine processes, can be automated and improved, independently of the interface or menu location.This paper proposes an RPA application, which in real-time, dynamically detects objects in software applications interface, allowing flexibility, performance, and accuracy regardless of the OS, location of interface tools, and necessary menus to reach it. For this purpose, a Convolution Neural Network (CNN) is trained with several interfaces and menus and used to classify software interfaces in real-time. Furthermore, a developed software takes automated actions, moving the mouse pointer, clicking, editing text, and performing any necessary action. Results show that the proposed RPA technique based on deep learning is capable of detecting objects in real-time, classify them, with outstanding accuracy, and take dynamic actions. This way, it is possible to automate any computer task.