Inter- and Intraobserver Variation in the Assessment of Preoperative Colostograms in Male Anorectal Malformations: An ARM-Net Consortium Survey

Aim: Male patients with anorectal malformations (ARM) are classified according to presence and level of a recto-urinary fistula. This is traditionally established by a preoperative high pressure distal colostogram that may be variably interpreted by different surgeons. The aim of this study was to evaluate the inter- and intraobserver variation in the assessment by pediatric surgeons of preoperative colostograms with respect to the level of the recto-urinary fistula. Materials and Methods: Sixteen pediatric surgeons from 14 European centers, belonging to the ARM-Net Consortium, scored twice 130 images of distal colostograms taken in sagittal projection at a median age of 66 days of life (range: 4-1106 days). Surgeons were asked to classify the fistula in: bulbar, prostatic, bladder-neck, no fistula, and ”unclear anatomy”. Their assessments were compared with the intra-operative findings (kappa) for two scoring rounds with an interval of 6 months (intra-observer variation). Agreement among the scoring of surgeons (inter-observer variation) was also calculated, using the Krippendorff’s alpha. A kappa over 0.75 is considered excellent, between 0.40 and 0.75 fair to good, and below 0.40 poor. Surgeons were asked to score the images in “poor” and “good” quality and to provide their years of experience in ARM treatment. Results: Agreement between the image-based rating of surgeons and the intra-operative findings ranged from 0.06 to 0.45 (mean 0.31). Inter-observer variation was higher (Krippendorff’s alpha between 0.40 and 0.45). Years of experience in ARM treatment did not seem to influence the scoring. The mean intra-observer variation between the two rounds was 0.64. Overall the quality of the images was considered poor. Images categorized as having a good quality resulted in a significantly higher kappa (mean: 0.36 and 0.37 in the first and second round, respectively) than in the group of bad quality images (mean: 0.25 and 0.23, respectively). Conclusions: There is poor agreement among experienced pediatric colorectal surgeons on preoperative colostograms. Techniques and analyses of images need to be improved in order to generate a homogeneous series of patients and make comparison of outcomes reliable.