Nonlinear mixed effects dose repsonse modeling in high throughput drug screens: Application to melanoma cell line analysis

// Kuan-Fu Ding 1, 2 , Emanuel F. Petricoin 3 , Darren Finlay 5 , Hongwei Yin 4 , William P.D. Hendricks 4 , Chris Sereduk 4 , Jeffrey Kiefer 4 , Aleksandar Sekulic 4 , Patricia M. LoRusso 6 , Kristiina Vuori 5, 6 , Jeffrey M. Trent 4 and Nicholas J. Schork 1, 2, 4 1 J. Craig Venter Institute, La Jolla, CA, USA 2 University of California, San Diego, CA, USA 3 George Mason University, Fairfax, VA, USA 4 The Translational Genomics Research Institute, Phoenix, AZ, USA 5 Sanford Burnham Prebys Medical Discovery Institute, La Jolla, CA, USA 6 Yale University, New Haven, CT, USA Correspondence to: Nicholas J. Schork, email: nschork@jcvi.org Keywords: high throughput drug screen, nonlinear mixed effect models, bioinformatics, cancer, drug response Received: June 22, 2017      Accepted: September 30, 2017      Published: December 15, 2017 ABSTRACT Cancer cell lines are often used in high throughput drug screens (HTS) to explore the relationship between cell line characteristics and responsiveness to different therapies. Many current analysis methods infer relationships by focusing on one aspect of cell line drug-specific dose-response curves (DRCs), the concentration causing 50% inhibition of a phenotypic endpoint (IC 50 ). Such methods may overlook DRC features and do not simultaneously leverage information about drug response patterns across cell lines, potentially increasing false positive and negative rates in drug response associations. We consider the application of two methods, each rooted in nonlinear mixed effects (NLME) models, that test the relationship relationships between estimated cell line DRCs and factors that might mitigate response. Both methods leverage estimation and testing techniques that consider the simultaneous analysis of different cell lines to draw inferences about any one cell line. One of the methods is designed to provide an omnibus test of the differences between cell line DRCs that is not focused on any one aspect of the DRC (such as the IC 50 value). We simulated different settings and compared the different methods on the simulated data. We also compared the proposed methods against traditional IC 50 -based methods using 40 melanoma cell lines whose transcriptomes, proteomes, and, importantly, BRAF and related mutation profiles were available. Ultimately, we find that the NLME-based methods are more robust, powerful and, for the omnibus test, more flexible, than traditional methods. Their application to the melanoma cell lines reveals insights into factors that may be clinically useful.