There are various ways to evaluate whether a vaccine works or not. If it is a vaccine for a communicable disease that hasn’t yet circulated in the community, like bird flu, the best you can do is rely on laboratory values as surrogates. The ability to raise antibodies in human volunteers is an example. When new vaccines are tested for safety and efficacy in the licensing process, this is the end point. But the real bottom line is whether it works to prevent disease in people living in the real world. Epidemiology and biostatistics are the public health specialties that try to figure this out.

As an epidemiologist I know many non-epidemiologist colleagues think that all we do is count things. We do count things, it’s true (although we also do things that don’t depend on counting), but the trick is not in counting them but in figuring out who and what to count, where to find the things to count once you figured what to count, when to count them, how to count them, how to organize what you’ve counted in a way to reveal the patterns you are interested in, and finally trying to figure out what the counts mean. Let’s take a look at vaccine efficacy as an example.

First question: what does it mean for a vaccine to work? That seems pretty straightforward, although you might have to ponder a bit about the question of “works for whom?” Children, the elderly, pregnant women? You get the idea. But it turns out that’s just for starters. Because even for flu, there is more than one vaccine. In the US, for example, there is an attenuated live virus vaccine that is sprayed in the nose and causes a very mild infection that in turn causes an immune response. Most people are familiar with another kind of vaccine that is injected into a muscle. This is an inactivated (no live virus) vaccine that is composed of only the parts of the flu virus that are good for provoking an immune response. We also know that vaccines work differently in different age groups. Now we are talking about additives called adjuvants that boost the immune response and allows less viral product and hence makes the vaccine go further. There’s more than one kind of adjuvant and each may be used at several doses. Right away we are starting to see an impressively large number of different combinations.

Then some people are already immune to flu (maybe they were vaccinated or had a case of the flu recently), so you should check to see if any of your subjects have signs of immunity before they even get the vaccine. Once you’ve got all this sorted out (I didn’t give you the full list of things you need to think about; just enough to give you the general idea), then you are ready to see if the vaccine “works” in the particular population you’ve selected. Usually you want to restrict your test subjects to some extent because otherwise there are too many variables to keep track of.

So let’s ask again, what does it mean for a vaccine to work? If you have a bunch of people you give the vaccine to and another group you don’t, you might think you are done. You simply observe them to see if there is a difference. Observe them for what? When? How? A paper last year by Basta, Halloran et al. reviewing and summarizing results from many different kinds of flu vaccine studies will give you some idea of the problems involved. Elizabeth Halloran is a biostatistician whose academic specialty is evaluating vaccine efficacy. There aren’t many who know more about it than she does. She points out at least four different outcomes you could use to evaluate a vaccine, and they correspond roughly to the terms we talked about in yesterday’s post: transmissibility, pathogenicity and virulence. Not exactly, but close enough that what we said there is applicable to what Halloran did.

Here are the four ways Halloran notes that a vaccine could be beneficial. It could make people less susceptible to the flu virus in the sense that they’d be exposed to it but wouldn’t get infected. It could make people who do get infected not get sick (decrease pathogenicity). We know that many people are infected but asymptomatic and a vaccine could increase that proportion. The vaccine could increase the chance that if you get sick it’s not so bad (decrease virulence). You are able to fight off the virus and get back on your feet more quickly. And independently of all that, the vaccine might make you less infectious to others, regardless of how sick you yourself might be (decrease transmissibility). When people talk about vaccine efficacy they should also tell you: effective for what end point. Usually you have to examine the study to see, and most often it’s the second category, prevention of people being simultaneously infected and sick with flu symptoms or sometimes serious flu complications, like hospitalization for pneumonia. You also want to know what the comparison is. It could be comparing live virus vaccine to inactivated vaccines (this is called relative efficacy) or either kind of vaccine to no vaccine at all (absolute efficacy).

How would you express efficacy, whatever outcome you used? The most natural way is through some kind of relative risk measure. In this case a relative risk is the risk of getting flu in the vaccinated group compared to the unvaccinated group (NB: in epidemiology a relative risk is a kind of effect measure; other effect measures are risk differences and odds ratios. This blog is named Effect Measure after this term of art in epidemiology). If the vaccine were completely effective, i.e., no who was vaccinated got the flu, then the risk in the vaccinated group would be zero and some other, positive risk in the unvaccinated group (assuming both groups were exposed to flu virus, either deliberately or naturally). The relative risk (RR) would be zero (zero divided by some other number). Vaccine efficacy is (1 – RR) times 100%, so if the RR was zero (a completely effective vaccine), the vaccine efficacy would be 100%. If there was no effect, the risk in each group would be the same, the RR=1, and the vaccine efficacy would be 0%. Of course there is always some random variation, and being able to come up with a good estimate with some notion of how much uncertainty there is is how biostatisticians like Halloran make a living.

More important, though, vaccines are hardly ever 100% effective, and this is especially true of flu vaccine. You can get vaccinated and still get the flu, or get infected, or be contagious, or whatever the endpoint is. It is not a guarantee for an individual. But what we know from the public health standpoint, where the measure is the population, the flu vaccine has a major effect on influenza in the community. We know this from numerous studies that have been done demonstrating it. I’ve hardly scratched the surface of this business. If you want to pursue it you could start with the Basta, Halloran et al. paper in the American Journal of Epidemiology (Am J Epidemiol. 2008 Dec 15;168(12):1343-52) and move on to look at individual studies. There are all kinds of them: double blind randomized trials where people are deliberately infected (“challenged” is the euphemism) with flu virus, randomized trials of vaccines in the community during flu season, sometimes where it turns out one or more vaccine component is mismatched to the circulating virus, etc. There are all sorts of pitfalls and problems in doing these studies, including how accurately endpoints are determined (which may depend on illness definitions or laboratory methods that vary from study to study), people initially enrolled in the study dropping out and lost to follow-up and much more. So there are holes in what we know, but to give you a flavor of the kinds of vaccine efficacies being seen, here is the abstract of the Basta, Halloran et al. paper:

In this paper, the authors provide estimates of 4 measures of vaccine efficacy for live, attenuated and inactivated influenza vaccine based on secondary analysis of 5 experimental influenza challenge studies in seronegative adults and community-based vaccine trials. The 4 vaccine efficacy measures are for susceptibility (VE(S)), symptomatic illness given infection (VE(P)), infection and illness (VE(SP)), and infectiousness (VE(I)). The authors also propose a combined (VE(C)) measure of the reduction in transmission in the entire population based on all of the above efficacy measures. Live influenza vaccine and inactivated vaccine provided similar protection against laboratory-confirmed infection (for live vaccine: VE(S) = 41%, 95% confidence interval (CI): 15, 66; for inactivated vaccine: VE(S) = 43%, 95% CI: 8, 79). Live vaccine had a higher efficacy for illness given infection (VE(P) = 67%, 95% CI: 24, 100) than inactivated vaccine (VE(P) = 29%, 95% CI: -19, 76), although the difference was not statistically significant. VE(SP) for the live vaccine was higher than for the inactivated vaccine. VE(I) estimates were particularly low for these influenza vaccines. VE(SP) and VE(C) can remain high for both vaccines, even when VE(I) is relatively low, as long as the other 2 measures of vaccine efficacy are relatively high. (Basta et al., AJE [2008])

Let’s see what some of this means. If you pick your way through this abstract (even better, read the paper if you have access to a medical library), you will see that using laboratory-confirmed infection, either live or inactivated flu virus has an efficacy of 40% – 45%. This means that the vaccinated group had less than half the amount of lab confirmed flu than the unvaccinated group. This is clearly a Big Win from the public health perspective — half as much flu — but for an individual, it means your chance of a win is about 50-50. That’s like flipping a fair coin with heads, a win, tails, a loss. That doesn’t sound so wonderful, but remember what the comparison is. The unvaccinated group’s coin has tails on both sides. It’s pretty clear to me which coin I’d rather play with, so I get a flu shot every year. But also remember what the end point is: infection with flu virus, not getting sick from the infection. Once you start parsing the outcomes (you have to dig into the paper for this) you get studies where the vaccine is well matched or mismatched and the outcome is infected plus sick or something else. Things get more complicated. You start to see differences in age categories and differences related to how the outcome was determined. For example, for children, the live virus vaccine seemed to be much more effective, but the reverse is true for adults. For adults the live vaccine had efficacy of about 30%, the inactivated (injected) one, 67%. Moreover this figure was for a study done in 2004-2005 where the vaccine was mismatched, so even in this case vaccination was doing pretty well. For some combinations the efficacy is 90% (for example, well matched live virus vaccine in a challenge study with illness as an endpoint). And some of the biggest uncertainties has to do with people in my age category, over 65, where the data are sparse and there is some evidence we don’t respond to vaccines as well as younger folks. Still, I get a flu shot every year. I’m playing the odds. If you are under 60, there’s no question. You should get vaccinated against influenza. You’ll come out ahead even if the vaccine is mismatched, which it sometimes is, although not often.

The Basta, Halloran et al. paper has lots of other summaries for different combinations which we won’t give you because it would get too confusing and isn’t the main point of this post — which is: estimating whether a vaccine works or not is a surprisingly difficult business, even during a pandemic.