Online incremental learning from scratch with application to handwritten gesture recognition

In recent years we have witnessed a significant growth of data to be processed by machines in order to extract all the knowledge from those data. For agile decision making, the arrival of new data is not bounded by time, and their characteristics may change over time. The knowledge stored in machines needs to be kept up to date. In traditional machine learning, this can be done by re-learning on new data. However, with the constant occurrence of high data volume, the process becomes overloaded, and this kind of task becomes unfeasible. Online learning methods have been proposed not only to help with the processing of high data volume, but also to become more agile and adaptive towards the changing nature of data. Online learning brings simplicity to the updates of the model by decoupling a single update from the whole updating process. Such a learning process is often referred to as incremental learning, where the machine learns by small increments to build the whole knowledge. The aim of this research is to contribute to online incremental learning for pattern recognition and more specifically, by handwritten symbol recognition. In this work, we focus on specific problems of online learning related to the stability-plasticity dilemma and fast processing. Driven by the application and the focus of this research, we apply our solutions to Neuro-Fuzzy models. The concept of Neuro-Fuzzy modeling is based on dividing the space of inseparable and overlying classes into sub-problems governed by fuzzy rules, where the classification is handled by a simple linear model and then used for a combined result of the model, i.e., only a partially linear one. Each handwritten symbol can be represented by a number of sets of symbols of one class that resemble each other but vary internally. Thus, each class appears difficult to be described by a simple model. By dividing it into a number of simpler problems, the task of describing the class is more feasible, which is our main motivation for choosing Neuro-Fuzzy models. To contribute to real-time processing, while at the same time maintaining a high recognition rate, we developed Incremental Similarity, a similarity measure using incremental learning and, most importantly, simple updates. Our solution has been applied to a number of models and has shown superior results. Often, the distribution of the classes is not uniform, i.e., there are blocks of occurrences and non-occurrences of some classes. As a result, if any given class is not used for a period of time, it will be forgotten. Since the models used in our research use Recursive Least Squares, we proposed Elastic Memory Learning, a method for this kind of optimization and, we have achieved significantly better results. The use of hyper-parameters tends to be a necessity for many models. However, the fixing of these parameters is performed by a cross-validation. In online learning, cross-validation is not possible, especially for real-time learning. In our work we have developed a new model that instead of fixing its hyper-parameters uses them as parameters that are learned in an automatic way according to the changing trends in the data. From there, the whole structure of the model needs to be self-adapted each time, which yields our proposal of a self-organized Neuro-Fuzzy model (SO-ARTIST). Since online incremental learning learns from one data point at a time, at the beginning there is only one. This is referred to as starting from scratch and leads to low generalization of the model at the beginning of the learning process, i.e., high variance. In our work we integrated a model based on kinematic theory to generate synthetic data into the online learning pipeline, and this has led to significantly lower variance at the initial stage of the learning process. Altogether, this work has contributed a number of novel methods in the area of online learning that have been published in international journals and presented at international conferences. The main goal of this thesis has been fulfilled and all the objectives have been tackled. Our results have shown significant impact in this area.