On the applicability of ANOVA models for CATA data

Abstract Check-All-That-Apply (CATA) questions return binary data from every subject on each product and attribute. To compare multiple samples, Cochran’s Q test is widely used to analyze such two-way data, with pairwise comparisons being conducted using McNemar’s test, the special case of Cochran’s Q for two samples. In many applications, this approach is viable, however, in some, it has limitations, most notably (i) incomplete and/or imbalanced data; (ii) experimental designs; (iii) more complex hypotheses. If an analysis of variance (ANOVA) would be valid in the sense of respecting the nominal type I error rate, all these issues would be resolved at once. CATA data is obviously not normally distributed. Yet, comparing analyses using ANOVA and Cochran’s Q, we noted F-tests for balanced designs to provide p values very close to those from Cochran’s Q test, and they have been shown earlier to be at least approximately valid for Rate-All-That-Apply (RATA) data. We explore whether ANOVA indeed offers a robust analysis for CATA data. To that end, we determined the randomization distribution for the respective F- and t-tests for various (balanced complete) CATA studies and compared it to the corresponding parametric distribution. The respective distributions overlap closely, suggesting that ANOVA-based tests (overall and for pairwise comparisons) are indeed valid. A limitation is found in case of very low elicitation rates across products, which leads to distinct discreteness of the randomization distribution for paired comparisons and hence the potential for slightly invalid results. Yet, even in those cases, ANOVA-based tests seem to provide a reasonable approximation for most purposes. We further discuss the properties for decreasing numbers of subjects as well as showcase applications for imbalanced data and modeling in a study using an experimental design for the products. We conclude with guidance on when the use of ANOVA for CATA data is reasonable.