Categorizing segmentation quality using a quantitative quality assurance algorithm

Introduction Obtaining high levels of contouring consistency is a major limiting step in optimizing the radiotherapeutic ratio. We describe a novel quantitative methodology for the quality assurance (QA) of contour compliance referenced against a community set of contouring experts. Methods Two clinical tumour site scenarios (10 lung cases and one prostate case) were used with QA algorithm. For each case, multiple physicians (lung: n = 6, prostate: n = 25) segmented various target/organ at risk (OAR) structures to define a set of community reference contours. For each set of community contours, a consensus contour (Simultaneous Truth and Performance Level Estimation (STAPLE)) was created. Differences between each individual community contour versus the group consensus contour were quantified by consensus-based contouring penalty metric (PM) scores. New observers segmented these same cases to calculate individual PM scores (for each unique target/OAR) for each new observer–STAPLE pair for comparison against the community and consensus contours. Results Four physicians contoured the 10 lung cases for a total of 72 contours for quality assurance evaluation against the previously derived community consensus contours. A total of 16 outlier contours were identified by the QA system of which 11 outliers were due to over-contouring discrepancies, three were due to over-/under-contouring discrepancies, and two were due to missing/incorrect nodal contours. In the prostate scenario involving six physicians, the QA system detected a missing penile bulb contour, systematic inner-bladder contouring, and under-contouring of the upper/anterior rectum. Conclusion A practical methodology for QA has been demonstrated with future clinical trial credentialing, medical education and auto-contouring assessment applications.