On the reliability of predictions on Covid-19 dynamics: a systematic and critical review of modelling techniques

Since the beginning of the new coronavirus 2019-nCoV disease (Covid-19) in December 2019, there has been an exponential number of studies using diverse modelling techniques to assess the dynamics of transmission of the disease, predict its future course and determine the impact of different control measures. In this study, we conducted a global systematic literature review to summarize trends in the modelling techniques used for Covid-19 from January 1st 2020 to June 30th 2020. We further examined the reliability and correctness of predictions by comparing predicted and observed values for cumulative cases and deaths. From an initial 2170 peer-reviewed articles and preprints found with our defined keywords, 148 were fully analyzed. We found that most studies on the modelling of Covid-19 were from Asia (52.70%) and Europe (25%). Most of them used compartmental models (SIR and SEIR) (57%) and statistical models (growth models and time series) (28%) while few used artificial intelligence (5%) and Bayesian approach (3%). For cumulative cases, the ratio predicted/observed values and the ratio of the amplitude of confidence interval (CI) or credibility interval (CrI) of predictions and the central value were on average larger than 1 (4.49-9.98 and 1.10-1.94, respectively) indicating cases of incorrect predictions, large uncertainty on predictions, and large variation across studies. There was no clear difference among models used for these two ratios. However, the ratio predicted/observed values was relatively smaller for SIR models than for SEIR models, indicating that more complex models might not be more accurate for predictions. We further found that values of both ratios decreased with the number of days covered by studies, indicating that the wider the time covered by the data, the higher the correctness and accuracy of predictions. In 21.62% of studies, observed values fall within the CI or CrI of the cumulative cases predicted by studies. Only six of the 148 selected studies (4.05%) predicted the number of deaths. For 33.3% of these predictions, the ratio of predicted to actual number of deaths was close to 1. We also found that the Bayesian model made predictions closer to reality than the compartmental and the statistical models, although these differences are only suggestive due to the small size of the data. Our findings suggest that while predictions made by the different models are useful to understand the pandemic course and guide policy-making, there should be cautious in their usage.