Central reading of ulcerative colitis clinical trial videos using neural networks.

Abstract Background and Aims Endoscopic disease activity scoring in ulcerative colitis (UC) is useful in clinical practice but infrequently done. It is required in clinical trials, where it is expensive and slow because human central readers (CR) are needed. A machine learning algorithm (MLA) automating the process could elevate clinical care and facilitate clinical research. Prior work using single-institution databases and endoscopic still images has been promising. Methods 795 full-length endoscopy videos (FLEV) were prospectively collected from a phase 2 trial of mirikizumab with 249 patients from 14 countries, totaling 19.5 million image frames. Expert CRs assigned each FLEV one endoscopic Mayo score (eMS) and one UC endoscopic index of severity (UCEIS) score. Initially, video data were cleaned, and abnormality features extracted using convolutional neural networks. Subsequently, a recurrent neural network (RNN) was trained on the features to predict eMS and UCEIS from individual full-length endoscopy videos. Results The primary metric to assess the performance of the RNN model was quadratic weighted kappa (QWK) comparing the agreement of the machine-read endoscopy score with the human central reader score. QWK progressively penalizes disagreements that exceed one level. The model’s agreement metric was excellent with QWK of 0.844 (95% CI, 0.787-0.901) for eMS and 0.855 (95% CI, 0.80-0.91) for UCEIS. Conclusion We found that a deep learning algorithm can be trained to predict levels of ulcerative colitis severity from full-length endoscopy videos. Our data set was prospectively collected in a multinational clinical trial, videos rather than still images were used, UCEIS and eMS were reported, and MLA performance metrics met or exceeded those previously published for UC severity scores.