Models, Models Everywhere

Another post today that deals to a degree in arcanum. The latest iteration of the now famous models of the outbreak of SARS-CoV-2 and its travelling companion, COVID-19. The Institute for Health Metrics and Evaluation (IHME) have been producing models for some weeks now; models which are updated about every three days or so.

As noted, the model uses empirical data sourced from a number of locations to fit (by force) a sigmoidal function called the "Gaussian Error Function" (a sort of fancy way of looking at likelihoods that a variable drawn from a bell curve will fall within a given range).

Though most of the popular press focus on just one result - the projected total number of deaths - in reality the model offers mortality as well as resource utilisation. The latter focuses on items critical to health care - the number of people who will land in hospital, the number who will land in the ICU, and how many will require mechanical ventilation.

The results of the prior iteration for the US were that, by August, just over 65,000 Americans would lose their lives, with the peak on the 15th of April. That represented a reduction from just over 68,000 last week. The latest projection is now up, slightly, to 65.976.

The model must be re-fitted with new data as they become available.

People are to a degree, missing that models of this sort are not true epidemiological tools, as they have more often than not, been deployed to project resource needs. And in this case, what (and when) will the greatest demand be for the critical medical resources be. 

Of course, this approach (forced curve-fitting) is only one approach, and the IHME model has come under some criticism (I honestly suspect, some is motivated by politics, as this is the tool that Dr Deborah Birx and the US Coronavirus Task Force are using).

There are broadly three types of models that are commonly used in this field. These are the parametric "SEIR" models (making estimates across populations based upon the number of people who are susceptible - not yet infected but who could be, exposed - those who are exposed, but not yet positive, infected, and removed - those no longer at risk or ill because they either have recovered, or are dead), agent-based models (in a nut-shell, simulations not unlike the old game Sim City, where a population is created with an initial number of infected people, where the transmission can occur if an infected and uninfected person encounter one another, and then the populations are followed over time until resolution), and curve-fitting models (of the sort the IHME is).

In my career working as an epidemiologist and statistician, I have worked with all three types. (To be more precise, SEIR actually has as a subgroup, SIR, where "exposed" people are collapsed and distributed into the population). Each has its strengths and weaknesses. All require assumptions to be made. 

Most recently, I worked on a team modelling the impact of the introduction of pre-exposure prophylaxis in HIV. We used an agent-based model largely because we simply lacked the right amount of information about movement from compartments in an SEIR model, and thought that the agent-based approach allowed for better hedging against some of the assumptions, as well as being easier to test the assumptions through what in mathematics is called probabilistic sensitivity analysis, or PSA - re-running thousands of mini simulations where the assumptions, rather than being held fixed, themselves are allowed to be drawn from a random distribution.

While I agree with, e.g., Dr Marc Lipsitch (who is one of the world's leading epidemiologists at Yale) that curve-fitting models of this type are unorthodox in epidemiology, I think it's a useful tool.

A lot of hype surrounded the initial projections of the Imperial College in London (a group I, personally, have worked with in the past) that somewhere north of 2 million would die without mitigation. That projection was derived from an SEIR model. Of course, the US has deployed a series of increasingly strict shelter in place initiatives, and that projection almost surely is going to be "wrong" by an order of magnitude.

Statistician George Box once observed, decades ago, that all models are wrong, but some models are useful.

All of the competing models are going to be "wrong" in the end. But what use do they provide now?

First, they give us some parameters on where the pandemic is going. It's not certainty - pace Box - but the projections give us some space to work with. And most provide, in addition to the projections, confidence intervals (or credible intervals) that give a range of what is likely to happen. The less certainty, the wider the bands. 

When reading the projections, you ignore these bands at your peril.

I don't want to weigh in on which is "right," because frankly, they are all going to be wrong. Paraphrasing Tolstoy, while projections that are right are right in the same way, wrong models are wrong in their own way.

Here is a sampling of a few competing approaches.

Columbia University in New York have built an SEIR model to estimate mortality and ICU usage under differing scenarios of mitigation. Estimates ranged from 6800 (with extreme social distancing) up to over 400,000 under strong mitigation. 

Northeastern University have produced an agent-based model here that offers a few scenarios for 'stay at home' mitigation. Its most recent projections are that, by mid-May, approximately 70,000 people will lose their lives. The uncertainty range is 42,000 to as many as 127,000.

Unfortunately, they do not project beyond then, but given the rate of decline of the curves, barring a second flare up, the final mortality is going to be around that number.

A late entry is from a team at MIT, who have an unorthodox approach similar to an SEIR model, but that does not presume priors for initial transmission risk (the famous Ro), but "learns" from the data. That, and other key parameters.

This model is something of an outlier, in two ways. First, while other approaches cease projecting past the "first wave," which is estimated to be more or less over between late June and early July (even for the MIT model), the data scientists at MIT incorporate a second wave - one that will begin to grow in about middle July. 

This echo will add about 25 to 30,000 deaths. It could result in as many as 280,000 when the dust settles.

The first wave is projected, as of now, to kill 104,000 or so by the end of June. The range here is anywhere from 70,000 to 170,000. This is a large outlier from the others.

What then, are the take-aways?

My own preference is agent-based, for no particular reason other than familiarity. The lone agent-based simulation pegs mortality at around 70,000 in the first wave. The team did not project a second, which isn't to say that there won't be one.

The IHME model is now projecting about 66,000 (range or 45-125,000), which is, if not the same as the Northeastern model, in the same neighbourhood. 

One piece of good news is that the models, with the exception of MITs AI model, are converging around similar stories. The fact that the models, using different presumptions and different approaches, net out to a similar result at this stage (we are still three months from July), is a sort of empirical validation.

And it looks like, as of now, mortality is more than likely going to land somewhere between 60,000 and maybe 100,000 at the top end. 

It could be worse, of course. It could be much worse.

The second thing to glean is that social distancing and mitigation are working. All models are converging down, not up, with additional data. The sacrifices we are making are actually, as of now, writing a very different - and brighter - narrative than the story as it was unfolding a few weeks ago.

Keep staying home. Keep distancing yourself. Keep good hygiene.

It's working.

Finally, as the MIT model indicates, we need to be especially vigilant in the later summer. If there is a second wave - and there is every reason to believe that one could come, it is going to be absolutely essential that our public health professionals keep a damned close eye.

And it means our political leaders need to be ready to sound the alarm if there is even a whiff of an outbreak.

And it means we as citizens need to listen - and we need to obey - when we are asked to observe stay at home orders, and not go to the hair salon. It's that simple.

Treatments are coming. Vaccines are coming. It will not be tomorrow.

Again, it's worth saying, by our actions, we are choosing our own future. 

Keep on Keeping On - Day 24

The outbreak of SARS-CoV-2 (the coronavirus) continues. It's a pandemic, which means, of course, that it now is just about everywhere.

As I've said a few times now, I am an epidemiologist, which means that my work focuses on public health. I've spent most of my professional life creating and abstracting information from models.

The president last week warned Americans that the coming days were going to be very difficult, and that admonition has not been wrong.

The news, while terrible, is not completely bleak.

I am following a number of analytical sources each day, trying my best to understand the evolution of the crisis. Two in particular are, I find useful.

The first - the COVID Tracking Project - is provided by a group of journalists, initially led by Robinson Meyer and Alexis Madrigal at The Atlantic. It tracks state-by-state reported statistics on total tests done, how many are positive, how many are negative, how many tests remain to be resolved (results confirmed and reported), how many patients are hospitalized, and how many people have died.

The former are critical from an epidemiology perspective, as we get a feel for the cumulative impact, and importantly, the day over day change. We hear a lot about "bending the curve." For this, the rate of change is a key metric; at least as much as the cumulative impact. The cumulative impact details the past - something we cannot change. The rate of change is more immediate, and can be impacted by choices that we make today.

The latter is important because tracking hospitalization gives us a much deeper view into how the disease is actually manifesting. As is now well-known, about half or so of people who get infected are either totally without symptoms, or have mild to moderate symptoms. They don't require hospital, and overwhelmingly will recover. For these people, COVID-19 will be an unpleasant experience. It won't be a fatal one. The more severe cases - ones that require hospitalization, or worse, ICU admission - are the ones that really drive the magnitude of the fear. These are virtually all of the patients who will die. They are the ones who will compete for resources.

I recommend to bookmark and check their data each day.

The second is a project run by the Institute for Health Metrics and Evaluation (IHME), a research centre at the University of Washington in Seattle. As an aside, I know, personally, more than one person at the University of Washington Department of Biostatistics, and it is one of the best in the world.

This team uses reported data on infections, mortality (death), hospital admissions, ICU admission, and ventilator demand to model the trajectory of disease. Data at one point were updated nearly daily. Updates now are regular, but not quite at the level.

This is one of the models the US government looks at when making decisions on resource planning, and it is the one that Dr Deborah Birx often refers to in the daily press briefings.

Like the COVIDtracking group, the IHME estimate, on a state by state basis, current predicted hospital loads and mortality. But they go a level deeper - projections are made on admissions to ICU and itubations (need for mechanical ventilation). The latter two are critical to understand the ability for our health care systems to meet life-saving demands of the infected population.

Predictions on mortality are also offered.

You can go to the site and see how your state is "doing" with respect to mortality and demand vs. resource needs.

Initially, the model predicted that mortality in the US would grow, slowly at first, then rapidly (a classic exponential growth model) until mid to late April, plateauing, and then falling until about June. When the statistical dust settled, it forecast around 100,000 deaths in the country, with a range of 75,000 to 250,000.

Those are terrible numbers.

About 95% of the country is now under direction to observe "social distancing" - an awkward phrase that really just means "separate yourself, physically, by about six feet from other people." It means to stay home unless essential - going to the grocery store, for example. And it means being vigilant about hand washing.

The results are starting to come in.

Friday, the predicted mortality of the US from this forecast model was about 95,000.

Monday, when updates were available, that number had fallen to 81,000.

The latest news this morning puts the estimated number at just over 60,000.

That's a lot of people. But it's a third less than the initial estimates.

Actions taken by people are starting to show results. The mortality curve is flattening. Here is how it looks as of today:

The coloured range should get your attention - this is a band of uncertainty (remember; these are mathematical models). The current situation has a range of about 40,000 to as much as 130,000 dead. The total mortality could turn out to be 130,000, even given where we are now.

These are, it's worth noting, models. As Dr Fauci noted, models have utility, but data are better. Statistician George Box famously quipped that all models are wrong, but some are useful. For me, this model is quite useful.

You'll also note that the line is solid until today - this is the historical data. The dotted line are projections. If that solid line trends back up, projected mortality is going to go up even more.

I re-iterate the point I've made a dozen times.

HOW this plays out is up to you. It's your choice. It's your future.

The final narrative is ours - whether it's a scary and terrible chapter, or a catastrophe.

These data are a hopeful sign. But they also should be a warning.

Stay home.

A Few Questions (and Answers, I Hope) about Hydroxychloriquine

Apologies for what is likely to be a very, very dry post today. We are living in strange days; better ones are coming. The comments to follow are not political. They should not be seen as endorsing one political narrative versus another.

As Queen Elizabeth (channeling Vera Lynn circa 1939), we will meet again.

CAVEAT: I am not a medical doctor. I am an epidemiologist. It is (and has been, for 26 years) my day to day job to build, run, and abstract the results from disease models. I've run analytics of randomized clinical trials. I have helped prepare (and been a subject matter expert for questions from) the United States Food and Drug Administration (FDA).

But I am not a "scientist." I would not question the medical expertise of a doctor, whose job it is to make medical decisions.

Let me say, with no ambiguity or equivocation, that ANY decision (yes or no) people make about medical care absolutely should be made in consultation with a doctor.

In the past few days, there has been a lot of talk in the news and the popular press about the use of hydroxychloroquine (with or without azithromycin) to treat people suffering from infection with the SARS-CoV-2 virus. The US president has made many statements in support of it. People in the press have attacked his comments.

There is, I think, a lot of misinformation and misunderstanding about these medicines, why there is some belief that they help treat COVID-19 (the illness associated with SARS-CoV-2), the risks of the medications, and frankly, what is going on.

What is COVID-19? 

First, you hear "COVID-19," "coronavirus," and, to a lesser extent, "SARS-Cov-2" used interchangeably. They aren't.

COVID-19 is a terminology resulting from the abbreviation resulting from "coronavirus disease" (COVID), concatenated with 19, the year (2019) it was identified. This is a standard way that the World Health Organisation (whose job it is to come up with the nomenclature) assign.

Coronavirus is a heuristic to describe a family of viruses (of which this particular strain is included) that are so called due to their physical structure - a central structure that houses the RNA (essentially, viruses are primitive structures of genetic material) with spiky projections that make the virus look like a crown. The word "corona" means "crown," in the original Latin.

SARS-CoV-2 is an abbreviation for "SARS-CoV" (severe, acute respiratory syndrome, of the coronavirus type). It is a strongly believed to be a close genetic cousin of the virus that caused the outbreak of SARS in 2005, hence the "2." The name was implemented by the International Committee on the Taxonomy of Viruses.

So briefly, the virus is SARS-CoV-2; the illness it provokes is COVID-19. Think of it in parallel that HIV is the virus that causes people to get sick. AIDS was the disease that resulted. It's not a perfect metaphor (and thankfully, with highly effective treatment, these days, people with HIV infection can suppress infection and avoid AIDS).

What is hydroxychloroquine?

Hydroxychlorquine is a medication that was first approved for use by the FDA in 1955. It is in the family of antimalarial drugs called aminoquinolines. Its first use was for treating malaria; later, it was approved by FDA for use in treating Lupus (SLE), and rheumatoid arthritis. Its activity against infection was well-established in its early days against malaria. How, exactly, it works with RA is still not well understood.

Hydroxychloroquine is known under a brand name called "Plaquenil."

It is on the WHO List of Essential Medicines - a list of what the WHO describes as the safest and most effective medications that are essential to basic health for a health system. These represent well-established and basic for the key population health needs. There are currently approximately 500 medications on this list.

What is azithromycin?

Azithromycin is an antibiotic (medications that are used to kill bacterial infections) called a macrolide antibiotic. It has broad activity against a host of GRAM positive (and some, more serious GRAM negative) bacteria, and is widely used for inner ear infections, community acquired bacterial pneumonia, skin infections, and some respiratory and throat infections. It was first approved for use in the USA in 1988.

Azihtromycin is marketed under the brand name Zithromax; often, it is sold in seven or 10 days packages called "Z Pack."

Azithromycin is, like hydroxychoroquine, on the WHO list of essential medications.

Why has hydroxychloroquine been suggested for COVI-19?
There are several problems with Dr Raoult's study. The first is that the sample size is very small. Only 42 patients is too small a sample for any regulatory body to approve a medication for use.

The results are encouraging.

One of the controversies not said in the US media is that Dr Raoult is not without controversy. He is a highly respected infectious disease doctors. He was named one of the ten leading researchers in all of France by Nature magazine, and has published two thousand peer-reviewed manuscripts in his career. He was awarded, in 2010, with the Grand Prix de l'INSERM (Institut National de la Sante et de la Receherce Medicale), the French national institute for health research.

ALL medications have adverse events. When FDA approve drugs, it is implicitly understood that there are risks. When medicines are approved, FDA (and other regulators) are asked to balance the risk/benefit calculus - is the benefit that the medication greater than the risk associated?

Frequent complaints on the news are about how the use of hydroxychloroqine as an intervention represents bad, or at the least incomplete, science. Dr Anthony Fauci, who, with Dr Deborah Birx, are the medical experts informing our government's response to the outbreak of SARS-CoV-2. Dr Fauci is one of the leading experts in the country on infectious disease. He earned his bona fides over the years working in HIV disease, as did Dr Birx. His opinion absolutely must be respected.

Medications are "approved" for use by regulators - in the US, by the FDA. When they are approved, there is a very detailed "label," and more in what is called a package insert - the little paper in six point Courier type in the bottle - that describe the indication (the disease its use is "approved" for), side effects observed by more than 1% of patients in the trial, the clinical trial data, dosing, and other details.

COVID-19 is a very serious problem. As of this writing, 400,000 Americans have been diagnosed. 13,000 have lost their lives.

A small, observational study was conducted by the French infectious disease expert called Didier Raoult at his hospital in Marseille, France (Institut Hospitalo-Universitaire, or IHU) in which he treated 42 patients who had been admitted to his hospital with COVID-19, at various severity of disease. 26 of the patients received hydroxychloroqine. 16 received 'standard of care,' which is to say, palliative care. Six of the 26 in the treatment arm had azihtromycin added to their treatment.

The results were published by Dr Raoult in the International Journal of Antimicrobial Agents in mid-March 2020.

Dr Raoult reported that 14/20 (70%) of the treatment arm were reported free of viral loads at six days following inclusion. In the control arm, just two had this outcome (12.5%).

This is a stark result, and is certainly, from the perspective of statistics, an encouraging number.

So, what is the controversy?

Second, the study was not a 'randomised control trial' (RCT). This is the gold-standard for the research and approval of medications in the US (FDA), the EU (European Medicines Agency, or EMA), Canada (Health Canada), and most other industrialised nations on earth. It involves "double blinding" (enrolling patients into the trial into arms that neither the subject nor the investigator knows is which) that are broken only at the end to control for biases. Dr Raoult's study was "open label" (subjects and doctors knew who was in the treatment arm, and who was in the control arm), and it was observational (meaning that there was no attempt, through randomisation, to allocate patients into the two arms to balance things like underlying health, age, and other factors that can bias results).

Third (and in my view an under-reported problem) is that the study was not an intent to treat (or modified intent to tread) analysis. Dr Raoult reported that 14/20 patients (and 6/6 of those on both hydroxychloroquine AND azithromycin) recovered. In fact, there were six patients lost to follow up, and not included in the calculations. Three of these in fact died (of their infection, not side effects). Three others did not complete treatment. So the results reported (70% efficacy) represent a sort of survivor bias. In fact, 14/26 (54%) recovered. That's much less exciting than the numbers broadcast.

The study in effect was very small, uncontrolled, and had design issues. People are right to be sceptical of these results.

Who is Dr Raoult

A significant part of the backstory that frequently remains unsaid in the US media is that Dr Raoult is not without controversy. He is a highly respected infectious disease doctors. He was named one of the ten leading researchers in all of France by Nature magazine, and has published two thousand peer-reviewed manuscripts in his career. He was awarded, in 2010, with the Grand Prix de l'INSERM (Institut National de la Sante et de la Receherce Medicale), the French national institute for health research.

He is, in short, an acclaimed doctor and researcher.

But he is idiosyncratic. In 2013, he ventured outside his lane and questioned climate change, which drew widespread ire and scorn.

What role is there in the arguments about safety?

Both hydroxychloroquine and azithromycin are among the most prescribed medications in the world. If you are over the age of five, it is almost certain that you have taken azithromycin at some point. Of course it has risks - the most significant, perhaps, is that it causes QT prolongation - a heart condition that can result in irregular heartbeat. This is a real, but rare risk.

It is not zero.

The reality is, most of the "side effects" of the treatment are things like nausea, rash, and headache. These are the types of common 'adverse events' reported in most clinical trials. Anecdotally, of the trials I have run analyses on, nausea, headache, diarrhoea, and rash have in every case been the most common events reported.

What about the 'science?'

But people, I think, are either misunderstanding what Dr Fauci is saying, or they are distorting it.

He is a scientist as well as a doctor. His comments reflect that the only way, really, to establish the efficacy of a treatment right now is via RCTs. Randomisation is simply the best way to mitigate sampling and other biases. Using control arms are the best way to establish placebo effects.

In a perfect world, of course we would conduct multiple RCTs. And I suspect that we will. Many are right now going on, and the results will be in.

The problem is, we simply do not have the luxury of time.

I am old enough to remember life before the advent of effective HIV therapies (in fact, not that long ago). 25 years ago, when antiretroviral therapies (ARV) had not been discovered, the criticisms of FDA among others were that they were too slow to react and were obstructing treatments.

HIV activists were, at the time, highly critical of Dr Fauci for just this reason.

Dr Fauci's is right to be conservative. It is part of his job to question whether this approach will work.

But people need to listen to what he is actually saying rather than what they think he is.

Basically, the treatments have side effects, but they have been in use for decades, and are on the whole, pretty safe. They don't have no risk, but the risks are well-known. What we don't know is the actual efficacy, which the RCTs will establish.

Thus, in deciding whether or not to treat, patients should consult their doctors; medical expertise should decide.

In short, talk with your doctor.

Both hydroxychloroqine and azithromycin require a doctor's prescription. You cannot walk into Safeway and pick it up as if it were aspirin.

What does it mean for a drug to be "approved?" What is off-label use?

Medications are not "approved." They are approved for specific diseases. Adalimumab (Humira), for example, is "approved" for rheumatoid arthritis, psoriasis, ulcerative colitis, and a few other autoimmune disorders.

Doctors can - and frequently do - prescribe medications for disease not specifically indicated in the label. This is called "off label use."

The laws of the US presume that the best person to decide care for a patient is his or her doctor - a medical expert who knows the specifics of the patient before him or her - and is best positioned to understand the nuances, risks, and benefits to make the best medical decision.

This is, again, why it is critical for patients to talk to - and listen to - the opinions of their doctors.

Right now, you cannot get access to hydroxychloroquine - whether for an approved use (Lupus) or off-label (COVID-19) without your doctor.

Bottom Line

The reality is, those numbers are going to end up being much higher.

I don't know that hydroxychloroquine (with or without azithromycin) is effective. The initial evidence is encouraging, but it remains to be seen if this will be borne out in larger studies.

I do know that both have some risks, but those risks are pretty small. The risk of death from SARS-CoV-2 is many orders of magnitude higher.

We hear, every day, about ventilators. And it's true that they are essential. Without them, virtually all of those who require invasive ventilation would die.

But they are not a cure. Data right now indicate that, of those going in ventilators, mortality is 30-50%.

I worked, 10 years ago, in Phase II (dose-ranging) studies for a novel treatment of Lupus. One of the comparators we used was Plaquenil. Safety for Plaquenil was something we evaluated versus our treatment. But it was not the driving factor. Not close.

If I were to test positive for the virus, I personally would not immediately turn to this unproven therapy. But if I were to be put in the hospital, I would seriously talk to my doctor about it. And I would absolutely listen to what the doctor said.

The claims by the president that "you have nothing to lose" by trying hydroxychoroquibe are false.

But given that we have nothing else in the armamentarium right now means that I would think long and hard about the novel treatments before I required a ventilator.