Governing the safety of artificial intelligence in healthcare

Artificial intelligence (AI) has enormous potential to improve the safety of healthcare, from increasing diagnostic accuracy,1 to optimising treatment planning,2 to forecasting outcomes of care.3 However, integrating AI technologies into the delivery of healthcare is likely to introduce a range of new risks and amplify existing ones. For instance, failures in widely used software have the potential to quickly affect large numbers of patients4; hidden assumptions in underlying data and models can lead to AI systems delivering dangerous recommendations that are insensitive to local care processes,5 6 and opaque AI techniques such as deep learning can make explaining and learning from failure extremely difficult.7 8 To maximise the benefits of AI in healthcare and to build trust among patients and practitioners, it will therefore be essential to robustly govern the risks that AI poses to patient safety. In a recent review in this journal, Challen and colleagues present an important and timely analysis of some of the key technological risks associated with the application of machine learning in clinical settings.9 Machine learning is a subfield of AI that focuses on the development of algorithms that are automatically derived and optimised through exposure to large quantities of exemplar ‘training’ data.10 The outputs of machine learning algorithms are essentially classifications of patterns that provide some sort of prediction—for instance, predicting whether an image shows a malignant melanoma or a benign mole.11 Some of the basic techniques of machine learning have existed for half a century or more, but progress in the field has accelerated rapidly due to advances in the development of ‘deep’ artificial neural networks12 combined with huge increases in computational power and the availability of enormous quantities of data. These techniques have underpinned recent public demonstrations of AI systems …