Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019

The objective of this research is to push the frontiers in Automated Machine Learning, specifically targeting Deep Learning. We analyse ChaLearn's Automated Deep Learning challenge whose design features include: (i) Code submissions entirely blind-tested, on five classification problems during development, then ten others during final testing. (ii) Raw data from various modalities (image, video, text, speech, tabular data), formatted as tensors. (iii) Emphasis on "any-time learning" strategies by imposing fixed time/memory resources and using the Area under Learning curve as metric. (iv) Baselines provided, including "Baseline 3", combining top-ranked solutions of past rounds (AutoCV, AutoNLP, AutoSpeech,and AutoSeries). (v) No Deep Learning imposed. Principal findings: (1) The top two winners passed all final tests without failure, a significant step towards true automation. Their solutions were open-sourced. (2) Despite our effort to format all datasets uniformly to encourage generic solutions, the participants adopted specific workflows for each modality. (3) Anytime learning was addressed successfully, without sacrificing final performance. (4) Although some solutions improved over Baseline 3, it strongly influenced many. (5) Deep Learning solutions dominated, but Neural Architecture Search was impractical within the time budget imposed. Most solutions relied on fixed-architecture pre-trained networks, with fine-tuning. Ablation studies revealed the importance of meta-learning, ensembling, and efficient data loading, while data-augmentation is not critical.