Speaker
Description
In order for epidemiological forecasts to be useful for decision-makers the forecasts need to be properly validated and evaluated \cite{c19}. Although several metrics fore evaluation have been proposed and used none of them account for the potential costs and losses that the decision-maker faces. We have adapted a decision-theoretic framework to an epidemiological context which assigns a Value Score (VS) to each model by comparing the expected expense of the decision-maker when acting on the model forecast to the expected expense obtained from acting on historical event probabilities \cite{skill}. The VS depends on the cost-loss ratio and a positive VS implies added value for the decision-maker whereas a negative VS means that historical event probabilities outperform the model forecasts. We apply this framework to a subset of model forecasts of influenza peak intensity from the FluSight Challenge and show that most models exhibit a positive VS for some range of cost-loss ratios \cite{flu}. However, there is no clear relationship between the VS and the original ranking of the model forecasts obtained using a modified log score. This is in part explained by the fact that the VS is sensitive to over- vs. underprediction, which is not the case for standard evaluation metrics. We believe that this type of context-sensitive evaluation will lead to improved utilisation of epidemiological forecasts by decision-makers.
Bibliography
@article{skill,
title={A skill score based on economic value for probability forecasts},
author={Wilks, DS},
journal={Meteorological Applications},
volume={8},
number={2},
pages={209--219},
year={2001},
publisher={Cambridge University Press}
}
@article{c19,
title={Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States},
author={Cramer, Estee Y and Ray, Evan L and Lopez, Velma K and Bracher, Johannes and Brennen, Andrea and Castro Rivadeneira, Alvaro J and Gerding, Aaron and Gneiting, Tilmann and House, Katie H and Huang, Yuxin and others},
journal={Proceedings of the National Academy of Sciences},
volume={119},
number={15},
pages={e2113561119},
year={2022},
publisher={National Academy of Sciences}
}
@article{flu,
title={Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the US},
author={Reich, Nicholas G and McGowan, Craig J and Yamana, Teresa K and Tushar, Abhinav and Ray, Evan L and Osthus, Dave and Kandula, Sasikiran and Brooks, Logan C and Crawford-Crudell, Willow and Gibson, Graham Casey and others},
journal={PLoS computational biology},
volume={15},
number={11},
pages={e1007486},
year={2019},
publisher={Public Library of Science San Francisco, CA USA}
}
