Performance of popular pulse oximeters compared with simultaneous arterial oxygen saturation or clinical-grade pulse oximetry: a cross-sectional validation study in intensive care patients

Background: The demand for pulse oximeters is high in the current COVID-19 pandemic. Despite their popularity, clinical studies to evaluate the reliability in obtaining information on a patient's oxygenation status, are lacking. Aim: To evaluate the performance of pulse oximeters under clinical conditions, with arterial blood gas measurement (SaO2) as reference standard. Methods: We studied the accuracy of ten top-selling pulse oximeters in Europe and the USA in an intensive care population at the Flevoziekenhuis in Almere, the Netherlands. The studied pulse oximeters were: AFAC FS10D, AGPTEK FS10C, ANAPULSE ANP 100, Cocobear, Contec CMS50D1, HYLOGY MD-H37, Mommed YM101, PRCMISEMED F4PRO, PULOX PO-200, and Zacurate Pro Series 500DL. Adult patients in whom an SaO2 measurement was obtained as part of routine care were asked for inclusion. Directly after obtaining the SaO2 blood sample we obtained the pulse oximeter readings (SpO2) of the investigational devices, in random order. Outcomes were bias (SpO2-SaO2) mean, root mean square difference (ARMS), mean absolute error (MAE), and accuracy in identifying hypoxemia (SaO2<=90%). As a clinical index test, we included a hospital-grade SpO2-monitor. Results: In 35 consecutive patients we obtained 2,258 SpO2-readings and 234 SaO2-samples. Mean bias ranged from -0.6 to -4.8. None of the pulse oximeters had an ARMS<=3%, the requirement set by ISO-standards and required for FDA 501(k) clearance. The MAE ranged from 2.3 to 5.1, and 5 out of 10 pulse oximeters met the requirements of [≤]3%. For hypoxemia, negative predictive values were 98-99%. Positive predictive values ranged from 11-30%. Highest accuracy (95%CI) were found for Contec CMS50D1; 91% (86-94) and Zacurate Pro Series 500DL; 90% (85-94). The hospital-grade SpO2-monitor had an ARMS of 3.0 and MAE of 1.9, and an accuracy of 95% (91-97%). Conclusion: Direct-to-consumer pulse oximeters do not meet ISO-standards required for FDA-clearance, but can accurately rule out hypoxemia.