Predictive demand models in the food and agriculture sectors : an analysis of the current models and results of a novel approach using machine learning techniques with retail scanner data

Agriculture commodities production and consumption are typically not aligned since the timing of commodity production with its pace of consumption is disjoint, once commodities are often produced periodically (with certain crops being harvested once a year) but with a continuous consumption throughout the year. The temporal mismatches in production and consumption require both commodities consumers (food industries) and producers (farmers) to predict future consumption based on limited unreliable information, about the future of demand and available historical data. Consequently, the lack of an appropriate understanding of what is the actual food consumption trend, lead's the producers in some cases to make wrong bets, which eventually causes food waste, price volatility and excess commodities stock. The commodities market has a good view of short-term supply fundamentals but still lacks powerful tools and frameworks to estimate long-term demand fundamentals, of which will drive the future supply. This thesis studies commodities demand forecasting using Nielsen's Retail Scanners data based on machine learning techniques to construct nonlinear parametric models of commodities consumption, using the U.S sugar cane as our use case. By combining Nielsen Retail Scanner data from January 2006 to December 2015 for a sample of 30% of U.S retail, wholesalers and small shops, considering a basket of products that has sugar as one of its main components, we were able to construct out-of-sample forecasts that significantly improve the prediction of sugar demand compared to classical base-line model approach of the historical moving average.