Generalized additive mixed modeling of longitudinal tumor growth reduces bias and improves decision making in translational oncology

Scientists working in translational oncology regularly conduct multi-group studies of mice with serially measured tumors. Longitudinal data collected can feature mid-study dropouts and complex nonlinear temporal response patterns. Parametric statistical models such as ones assuming exponential growth are useful for summarizing tumor volume over ranges for which the growth model holds, with the advantage that the model's parameter estimates can be used to summarize between-group differences in tumor volume growth with statistical measures of uncertainty. However, these same assumed growth models are too rigid to recapitulate patterns observed in many experiments, which in turn diminishes the effectiveness of their parameter estimates as summary statistics. To address this problem, we generalized such models by adopting a nonparametric approach in which group-level response trends for logarithmically scaled tumor volume are estimated as regression splines in a generalized additive mixed model. We also describe a novel summary statistic for group level splines over user-defined, experimentally relevant time ranges. This statistic reduces to the log-linear growth rate for data well described by exponential growth and also has a sampling distribution across groups that is well approximated by a multivariate Gaussian, thus facilitating downstream analysis. Real-data examples show that this nonparametric approach not only enhances fidelity in describing nonlinear growth scenarios but also improves statistical power to detect inter-regimen differences when compared to the simple exponential model so that it generalizes the linear mixed effects paradigm for analysis of log-linear growth to nonlinear scenarios in a useful way.