A Value-Driven Framework For The Evaluation Of Biosurveillance Systems

Objective To describe the development of an evaluation framework that allows quantification of surveillance functions and subsequent aggregation towards an overall score for biosurveillance system performance. Introduction Evaluation and strengthening of biosurveillance systems is a complex process that involves sequential decision steps, numerous stakeholders, and requires accommodating multiple and conflicting objectives. Biosurveillance evaluation, the initiating step towards biosurveillance strengthening, is a multi-dimensional decision problem that can be properly addressed via multi-criteria-decision models. Existing evaluation frameworks tend to focus on “hard” technical attributes (e.g. sensitivity) while ignoring other “soft” criteria (e.g. transparency) of difficult measurement and aggregation. As a result, biosurveillance value, a multi-dimensional entity, is not properly defined or assessed. Not addressing the entire range of criteria leads to partial evaluations that may fail to convene sufficient support across the stakeholders’ base for biosurveillance improvements. We seek to develop a generic and flexible evaluation framework capable of integrating the multiple and conflicting criteria and values of different stakeholders, and which is sufficiently tractable to allow quantification of the value of specific biosurveillance projects towards the overall performance of biosurveillance systems. Methods We chose a Multi Attribute Value Theory model (MAVT) to support the development of the evaluation framework. Development of the model was done through online decision conferencing sessions with expert judgement, an indispensable part of MAVT modelling, provided by surveillance experts recruited from the member pool of the International Society for Disease Surveillance. The surveillance functions or quality criteria that were considered for the framework were initially gathered from a review of the literature with specific attention to a subset of public health quality criteria (1). Group discussions with the experts led to a final list of functions, finally reviewed to comply with the properties for good criteria in decision models. The eleven functions were: sensitivity; timeliness; positive predictive value (PPV); transparency; versatility; multiple utility; representativeness; sustainability; advancing the field and innovation; risk reduction; and actionable information. In addition, 24 different scenarios were developed for sensitivity, PPV, and timeliness since their values may differ with the level of infectiousness of the condition/event of interest, its severity and the availability of treatment and/or prevention measures. Four or five levels of performance were also developed for each criterion. Macbeth (Measuring Attractiveness by a Category-Based Evaluation Technique) tables were used to elicit values of different levels of performance from the experts using qualitative pairwise comparisons and then convert them into numerical values. Results To date, two criteria, sensitivity and transparency, have been assessed by more than one expert working on the same scenario. Value functions were generated for each criterion and scenario by calculating the median of the different values produced by the experts. For both sensitivity and transparency, value functions were mostly linear, indicating similar preferences between levels of performance. However, for some scenarios, experts allocated greater value to increases at the higher end of the performance level distribution. Conclusions At the time of writing new elicitation sessions are planned to conclude the model. Next, we will apply swing weights to support the trade-offs between the different criteria. We will present the baseline model elicitated from the experts and demonstrate how to apply portfolio decision analysis to assess overall performance of biosurveillance systems according to the specific needs of stakeholders and in conjunction with macro-epidemiological models.