Resilient Combination of Complementary CNN and RNN Features for Text Classification through Attention and Ensembling

Natural language processing (NLP) pipelines are usually complex, including several ways of extracting features and processing the inputs and results. The difficulty of the task directly affects the complexity of the system: multiple modules work together, extracting complementary information that is needed for a good performance. In this work we focus on text classification and show that the same intuition applies to end-to-end neural NLP architectures and that the best results are obtained reliably by combining the information from different neural modules. Concretely, we combine convolution, recurrent and attention modules with ensembles and show that they are complementary. We empirically prove that the combination is robust to various and complex text classification jobs and show that it attains or surpasses the state-of-the-art on a wide variety of datasets with no changes to the architecture. In addition, we show that ensembling CNN-RNN stacks with attention improve the performance with respect to only using a subset of the aforementioned modules. These observations hold in both low and high data availability, as well as for multi-class problems.