Assessing fitness-for-purpose and comparing the suitability of COVID-19 multi-country models for local contexts and users

Background: Mathematical models have been used throughout the COVID-19 pandemic to inform policymaking decisions. The COVID-19 Multi-Model Comparison Collaboration (CMCC) was established to provide country governments, particularly low- and middle-income countries (LMICs), and other model users with an overview of the aims, capabilities and limits of the main multi-country COVID-19 models to optimise their usefulness in the COVID-19 response. Methods: Seven models were identified that satisfied the inclusion criteria for the model comparison and had creators that were willing to participate in this analysis. A questionnaire, extraction tables and interview structure were developed to be used for each model, these tools had the aim of capturing the model characteristics deemed of greatest importance based on discussions with the Policy Group. The questionnaires were first completed by the CMCC Technical group using publicly available information, before further clarification and verification was obtained during interviews with the model developers. The fitness-for-purpose flow chart for assessing the appropriateness for use of different COVID-19 models was developed jointly by the CMCC Technical Group and Policy Group. Results: A flow chart of key questions to assess the fitness-for-purpose of commonly used COVID-19 epidemiological models was developed, with focus placed on their use in LMICs. Furthermore, each model was summarised with a description of the main characteristics, as well as the level of engagement and expertise required to use or adapt these models to LMIC settings. Conclusions: This work formalises a process for engagement with models, which is often done on an ad-hoc basis, with recommendations for both policymakers and model developers and should improve modelling use in policy decision making.