A mixed-model approach for powerful testing of genetic associations with cancer risk incorporating tumor characteristics

Cancers are routinely classified into subtypes according to various features, including histo-pathological characteristics and molecular markers. Previous investigations of genetic loci have reported heterogeneous association between loci and cancer subtypes. However, it is not evident what is the optimal modeling strategy for handling correlated tumor features, missing data, and increased degrees-of-freedom in the underlying tests of associations. We propose a score test for genetic associations using a mixed-effect two-stage polytomous model (MTOP). In the first stage, a standard polytomous model is used to specify for all possible subtypes defined by the cross-classification of different markers. In the second stage, the subtype-specific case-control odds ratios are specified using a more parsimonious model based on the case-control odds ratio for a baseline subtype, and the case-case parameters associated with tumor markers. Further, to reduce the degrees-of-freedom, we specify case-case parameters for additional markers using a random-effect model. We use the EM algorithm to account for missing data on tumor markers. The score-test distribution theory is developed by borrowing analogous techniques from group-based association tests. Through analysis of simulations across a wide range of realistic scenarios and data from the Polish Breast Cancer Study (PBCS), we show MTOP substantially outperform several alternative methods for identifying heterogeneous associations between risk loci and tumor subtypes.