Field-by-field evaluation of intraoperative transoesophageal echocardiography interpretative skills

A quality assurance system is essential for the credibility and structured growth of anaesthesiology-based transoesophageal echocardiography (TEE) programmes. We have developed software (Q/A Kappa), involving a 400-line source code, capable of directly reporting kappa correlation coefficient values, using external reviewer interpretations as the `gold standard', and thereby allowing systematic assessment of the validity of intraoperative echocardiographic interpretation. This paper presents assessment of the validity of 240 intraoperative anaesthesiologists' echocardiographic interpretations, and, in addition, the results of field testing of this prototypical software. Data, derived from consecutive cardiac surgery patients, consisted of standardized two-dimensional transoesophageal echocardiographic, colour flow and Doppler imaging sequences. Intraoperative and off-line `gold standard' TEE interpretations were compared for 19 fields or variables using the Q/A Kappa program. The kappa correlation coefficients were highly variable and dependent on the examination field, ranging from 0.08 for apical regional wall motion scores to 1.00 for tricuspid regurgitation grade, left atrial measurement, aortic valve anatomy and left ventricular long axis and short axis global function. The correlation coefficients were also operator dependent. These data (480 interpretations) were also manually integrated into the equation required for calculation of values of the variable kappa correlation coefficient. The relationship between Q/A Kappa-derived values and manually calculated values was highly significant (p < 0.001; r = 1.0). The implications and possible explanations of the results for particular examination fields are discussed. This study also demonstrates successful seamless functioning of this software program from data entry, segmentation into tables and valid statistical analysis. These findings suggest that it is practical to provide sophisticated continuous quality improvement TEE data on a routine basis.