Lessons from the COVID-19 pandemic
What have we learned from the COVID-19 pandemic?
The COVID-19 pandemic has been a global crisis that has touched every corner of the world, including Belgium. The crisis revealed how different layers are interconnected: data, science, communication, our behavior... Although the acute phase of the pandemic is behind us, various data are still continuously flowing in, and it will take years to connect everything. From long COVID to the seasonality of corona, there are still many questions without clear answers.
However, on a scientific level, we have learned a tremendous amount in recent years and are undoubtedly in a much better position than a few years ago. We have learned how to react faster and more effectively, from the data we need to how to use that information to formulate more reliable advice. We have gained new insights into what works, what doesn't, and how to distinguish between the two.
Unfortunately, the knowledge we have built about COVID-19 is, to some extent, relative because with every new virus, you start from scratch. You always have to get to know a new virus first, so we must be prepared to apply the insights gained to a new adversary.
Now we must translate the experience of the past years into sustainable structures and plans. It would be a real shame if we couldn't formalize and perpetuate the successes and ensure that we do not repeat the mistakes made.
How do we define ‘success’?
Before the COVID-19 crisis, we had many theories about pandemics and how they would influence our behavior or impact our economy, but now we also have hard data. That's why it's crucial to look back and try to figure out what worked and what didn't in curbing the virus.
What does a successful approach to a pandemic look like? Is it fewer deaths? A smaller impact on the gross national product? How do you weigh fewer deaths against other indirect health factors and economic losses? What does success mean in the short term versus the long term?
Current research typically focuses on a specific objective or outcome to evaluate the impact of measures, like the attack rate or the proportion of susceptible people who become ill within a specific time frame. In reality, it's always a trade-off between different, sometimes conflicting criteria. During the crisis, the primary goal of our government was to prevent our healthcare system from collapsing. We wanted to avoid situations where patients ended up in hospital hallways or even the parking lot without receiving the necessary care, and fortunately, we succeeded. The second priority was to keep schools open, which was heavily emphasized from the beginning because of all the evidence showing how essential it is to mitigate both the economic and mental impacts of the crisis. Overall, we did reasonably well in comparison to other European countries and beyond, as our schools remained closed for only a very limited period.
Of course, the measures taken to achieve these goals had several side effects, some of which were disproportionately large, according to some. As researchers, we want to use data to extract lessons on how to optimize the trade-offs between different priorities. We distinguish four categories of objectives: minimizing the disease burden from COVID-19, minimizing the disease burden from deferred care, minimizing the impact on mental well-being, and minimizing the economic impact of the crisis and the measures taken. We want to work with public health experts to find a meaningful balance between these four pillars that can express what good or poor indicators are for managing an epidemic or pandemic.
Along with colleagues at the Vrije Universiteit Brussel, we used reinforcement learning to evaluate the impact of the lockdown and exit strategy during the first wave on this interplay of objectives. Reinforcement learning is a form of machine learning that focuses on decision processes. The AI is built around what "optimal behavior" is through interactions in an environment where certain outcomes are rewarded and others are punished. The algorithm must discover by trial and error which decisions yield the highest reward, not only immediately but also in the long term. Thanks to reinforcement learning, self-driving cars can make decisions based on the environment, and the same principle helped the computer algorithm AlphaGo defeat the world champion in the Chinese board game Go.
In the context of the first wave, we applied the same strategy to our stochastic compartmental model, which we effectively used to inform policymakers during the pandemic. We let the computer simulate and calculate, using the knowledge we had about the virus's spread, which approach or measures would minimize both the disease burden and the societal consequences to the maximum extent. The algorithm succeeded in learning correctly that we can drastically relax social restrictions when hospitalizations are sufficiently low. It's an initial exercise demonstrating that this tactic is feasible to employ, even with complex epidemiological models.
Exploring the Pareto front of multi-objective COVID-19 mitigation policies using reinforcement learning
Abstract
Infectious disease outbreaks can have a disruptive impact on public health and societal processes. As decision making in the context of epidemic mitigation is hard, reinforcement learning provides a methodology to automatically learn prevention strategies in combination with complex epidemic models. Current research focuses on optimizing policies w.r.t. a single objective, such as the pathogen's attack rate. However, as the mitigation of epidemics involves distinct, and possibly conflicting criteria (i.a., prevalence, mortality, morbidity, cost), a multi-objective approach is warranted to learn balanced policies. To lift this decision-making process to real-world epidemic models, we apply deep multi-objective reinforcement learning and build upon a state-of-the-art algorithm, Pareto Conditioned Networks (PCN), to learn a set of solutions that approximates the Pareto front of the decision problem. We consider the first wave of the Belgian COVID-19 epidemic, which was mitigated by a lockdown, and study different deconfinement strategies, aiming to minimize both COVID-19 cases (i.e., infections and hospitalizations) and the societal burden that is induced by the applied mitigation measures. We contribute a multi-objective Markov decision process that encapsulates the stochastic compartment model that was used to inform policy makers during the COVID-19 epidemic. As these social mitigation measures are implemented in a continuous action space that modulates the contact matrix of the age-structured epidemic model, we extend PCN to this setting. We evaluate the solution returned by PCN, and observe that it correctly learns to reduce the social burden whenever the hospitalization rates are sufficiently low. In this work, we thus show that multi-objective reinforcement learning is attainable in complex epidemiological models and provides essential insights to balance complex mitigation policies.
We don’t believe AI should decide which measures we should or shouldn't implement. All outcomes of such a reinforcement learning approach will always need to be assessed and interpreted by experts. However, particularly for retrospective analyses, where we look back in time, they provide valuable assistance in objectively and fully weighing a vast number of scenarios and alternatives. We can also use this approach to measure the impact of the most extreme measures, which may never be applied in reality.
Quantity and quality
Because the impact of the crisis goes far beyond mortality figures, health economists and public health experts study quality-adjusted life years, or QALYs. One QALY is equal to a year of life in perfect health; in other words, this unit takes into account not only the quantity of life but also its quality.
While more detailed research on this topic, both in our country and abroad, is still ongoing, we can already discern some initial outlines. We looked at how certain aspects of the vaccination campaign in 2021 and 2022 influenced the quantity and quality of life for both the elderly and the young. Our analyses revealed that if more children had been vaccinated, there would have been less pressure on intensive care units, and fewer elderly people would have died. Conversely, if more adults had received booster shots, it would have primarily increased the QALYs for young adults. Both scenarios would have allowed for more and quicker physical interactions, which would benefit both our mental health and the economy.
Exploring the SARS-CoV-2 Burden of Disease and Age-Specific QALY Gains of Vaccination Strategies While Accounting for Emerging Variants of Concern
As the crisis continued, criticism increasingly arose that not enough consideration was given to the impact of the measures on our overall and mental well-being, especially among the most vulnerable. We are also looking for better indicators to reflect the complete picture as accurately as possible.