Core Technologies: Machine Learning and Natural Language Processing

This chapter focuses on the topic of machine learning and introduces the basics of natural language processing, which increasingly uses machine-learning techniques. The link between traditional statistics and machine learning is discussed, and a vocabulary of fundamental machine-learning concepts is introduced. I differentiate between supervised and unsupervised learning methods, and introduce fundamental techniques that are widely used, I introduce Bayes’ theorem, which forms the basis of several methods. I discuss methods that apply to sequential data. I discuss the basics of natural language processing (NLP), and consider how it has influenced information retrieval: the challenges in scaling NLP up are also discussed. Some of the problems are intrinsic to the goals of NLP, which tries to do much more with text than information-retrieval methods.