Accuracy and cross-calibration of video-quality metrics: new methods from ATIS/T1A1

Video quality metrics (VQM) have often been evaluated and compared using simple measures of correlation with observers. This approach does not fully take into account the variability implicit in the observers. We present techniques for determining the statistical resolving power of a VQM, defined as the minimum change in the value of the metric for which subjective test scores show a significant change. These techniques have been applied to the VQEG dataset, and incorporated into the recent ATIS/T1A1 series of technical reports (TR), which provide a comprehensive framework for characterizing and validating full-reference video quality metrics (VQM). These approved TR, while not standards, will enable the USA telecommunications industry to incorporate video quality metrics into contracts and tariffs for compressed video distribution. New methods for assessing VQM accuracy and cross-calibrating VQM are an integral part of the framework. These methods have been applied to two VQM at this point: PSNR and the version of Sarnoff's JNDmetrix tested by VQEG. The framework is readily extensible to additional VQM.