How sure are we about the effectiveness of COVID-19 vaccines?

Hypothetical case replacement can be used to quantify the robustness of trial results

Once a vaccine is approved for distribution to the full population and no longer just to a test population, scientists expect the efficacy rates to change slightly. A new statistical tool can help assess the strength of results from clinical trials for vaccines and other studies. The tool has just been published in the Journal of Clinical Epidemiology, and the tool was developed by a group led by researchers at the University of Chicago, Michigan State University, the University of Indiana, and Arizona State University. 

The Moderna and Pfizer vaccines for COVID-19 have demonstrated effectiveness in large scale clinical trials. For example, Moderna’s vaccine has been estimated as 94% effective because 185 of the people who received the placebo got COVID, compared to only eleven of the people who received the vaccine, for a total of 196 people in the study who became ill.  

One way to describe the effectiveness is to compare the results with a threshold for taking action. In August 2020, Dr. Anthony Fauci, Director of the National Institute for Allergy and Infectious Diseases, said “Scientists are hoping for a coronavirus vaccine that is at least 75% effective, but 50% or 60% effective would be acceptable, too.” 

For example, if researchers work with a threshold of 70% as a cut-off for acceptable effectiveness for a vaccine, how much would have to change in the study to see that decline of efficacy?

In the Moderna study, 46 people in the vaccinated group would have to have gotten COVID for the vaccine to be rendered only 70% effective. That is 35 more than the eleven who actually did. The difference (46-11=35) is related to what epidemiologists describe as the fragility of the evidence. The more the outcomes would have to shift to change the current course of action, the more confidence we can have that the conclusion is robust.

While an uptick of 35 infections in the clinical trial may not seem to be much, “[t]he new research puts fragility numbers like 35 in perspective by calculating the robustness of inference to replacement which considers how rare an event is in the placebo group,” said Dr. Qinyun Lin, Postdoctoral Researcher at Center for Spatial Data Science, University of Chicago, a lead co-author on the study.

Calculating the robustness of inference to replacement reveals that in order to generate the 35 extra COVID victims in the vaccinated group, more than 2,600 (out of about 14,000) of the vaccinated patients with would had had to instead received the placebo. The size of the replacement number reflects the infection rate of about 1% in the placebo group.

“It’s not like everyone in the placebo group got COVID. It’s sort of like finding double yoke eggs,” said Kenneth Frank, Michigan State Foundation professor of Sociometrics and another lead co-author on the paper.

“If you discovered three in a month, and I discovered one I might have to crack open hundreds more eggs to discover two more to ‘catch up.’  Just so, it would take replacing a lot of patients in the vaccinated group to ‘catch up’ to the placebo group in the rate of infections.”

“Since it would take changing more than 2600 patients’ outcomes to change our conclusions about the effectiveness of the Moderna vaccine, we can be quite confident that the vaccine has the effectiveness desired to help curb the pandemic,” said Anna Mueller, Luther Dana Waterman Associate Professor, Indiana University Bloomington, and also a lead co-author. 

Similar calculations based on findings for the Pfizer vaccine reveal a comparably robust inference; one would have to replace more than 3600 of 18,198 vaccinated cases to reduce the effectiveness of 95% below the threshold of 70%.

“Indeed, this type of calculation, also recently discussed in the New York Times, suggests the inferences of the effectiveness of the Moderna and Pfizer vaccines are some of the most robust we have at this point for any response to COVID-19 (see the konfound-it blog). And this new study gives us a way of articulating the strength of the evidence in terms of people’s experiences” said Frank.

+++

Note: The article is “Hypothetical case replacement can be used to quantify the robustness of trial results.” by Kenneth A. Frank*, Qinyun Lin*, Spiro Maroulis*, Anna S. Mueller*, Ran Xu, Joshua M. Rosenberg, Christopher S. Hayter, Ramy A. Mahmoud, Marynia Kolak, Thomas Dietz, Lixin Zhang. (https://doi.org/10.1016/j.jclinepi.2021.01.025) Published in the Journal of Clinical Epidemiology published by Elsevier.