Assessment of Expert-Level Automated Detection of Plasmodium falciparum in Digitized Thin Blood Smear Images

Importance Decades of effort have been devoted to establishing an automated microscopic diagnosis of malaria, but there are challenges in achieving expert-level performance in real-world clinical settings because publicly available annotated data for benchmark and validation are required. Objective To assess an expert-level malaria detection algorithm using a publicly available benchmark image data set. Design, Setting, and Participants In this diagnostic study, clinically validated malaria image data sets, the Taiwan Images for Malaria Eradication (TIME), were created by digitizing thin blood smears acquired from patients with malaria selected from the biobank of the Taiwan Centers for Disease Control from January 1, 2003, to December 31, 2018. These smear images were annotated by 4 clinical laboratory scientists who worked in medical centers in Taiwan and trained for malaria microscopic diagnosis at the national reference laboratory of the Taiwan Centers for Disease Control. With TIME, a convolutional neural network–based object detection algorithm was developed for identification of malaria-infected red blood cells. A diagnostic challenge using another independent data set within TIME was performed to compare the algorithm performance against that of human experts as clinical validation. Main Outcomes and Measures Performance on detectingPlasmodium falciparum–infected blood cells was measured by average precision, and performance on detectingP falciparuminfection at the image level was measured using sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). Results The TIME data sets contained 8145 images of 36 blood smears from patients with suspected malaria (30P falciparum–positive and 6P falciparum–negative smears) that had reliable annotations. For clinical validation, the average precision was 0.885 for detectingP falciparum–infected blood cells and 0.838 for ring form. For detectingP falciparuminfection on blood smear images, the algorithm had expert-level performance (sensitivity, 0.995; specificity, 0.900; AUC, 0.997 [95% CI, 0.993-0.999]), especially in detecting ring form (sensitivity, 0.968; specificity, 0.960; AUC, 0.995 [95% CI, 0.990-0.998]) compared with experienced microscopists (mean sensitivity, 0.995 [95% CI, 0.993-0.998]; mean specificity, 0.955 [95% CI, 0.885-1.000]). Conclusions and Relevance The findings suggest that a clinically validated expert-level malaria detection algorithm can be developed by using reliable data sets.