Randomization of CATA attributes: Should attribute lists be allocated to assessors or to samples?

Abstract For the check-all-that-apply (CATA) question format, it is good practice to vary the attribute list order between evaluations to account for possible (and likely) position bias in the data. If attribute lists are to be randomized, the question is how to allocate these attribute lists orders. Some authors recommend a “to samples” allocation order, randomizing the attribute list order for each sample presented. Other authors recommend a “to assessors” allocation, randomizing the attribute list order for each assessor, such that the list order is stable across samples for each assessor (if replication is used, assessors are given a new attribute list order per replication). In this study, consumers ( n  = 93) performed CATA evaluations on 6 breads twice. The evaluation was done once using the “to assessors” CATA list order allocation scheme, and once using the “to samples” CATA list order allocation scheme, with the order of allocation schemes balanced by experimental design. Results suggest higher operational power when using the “to assessor” CATA attribute list order. This conclusion is supported by theoretical considerations.