Very recently we came across several studies in which small differences in design had a major impact on outcomes. A blog is not a good place to dig deeply into the minutiae of study design as the discussion becomes quickly very dry and quite technical. However, readers and clinicians are often unaware how a seemingly small change in design can affect results, thinking that reported outcomes are robust enough to be affected little by somewhat different ‘real world’ conditions. As clinicians we are used to extrapolating from idealized study populations to other groups of patients, a practice so common that we accept B-III category recommendations in Guidelines, which are no longer data-driven but rather opinion-based.
This is an area rarely discussed but consequential for drug developers and regulators; it has not attracted much attention in the literature to the best of my knowledge. Hence we will provide a mini-series and start today with an example from the world of vaccine trials.
Case #1: LAIV Influenza Vaccine
The news releases from CDC and ACIP this week give us the following information on vaccine effectiveness (VE):
- 2015 / 2016:
Seasonal Flu Vaccine VE Network, preliminary study data 
LAIV: 3% (CI: -49%, 37%) age 2-17
IIV: 63% (CI: 52%, 72%) age 2-17
non-CDC studies: similar lower VE with LAIV compared to IIV
- 2014 / 2015:
Seasonal Flu Vaccine VE Network
activity against H3N2
LAIV: -23% (CI: -90%, 21%) age 2-8
IIV: 15% (CI: -20%, 40%) age 2-8
- 2013/ 1 4:
ACIP data review:
activity against H1N1
LAIV: no measurable effectiveness for LAIV against influenza A (H1N1)
IIV: 60% (mid-point estimate) age 2-8
Over the last 12 years, the weight-adjusted average VE for flu vaccination was a respectable 45%. While there were big swings in VE from year to year, please let us be mindful of how difficult the forecasting process is for next season’s influenza strains. Selection of vaccine strains is based on many assumptions, which may or may not come true; it is a time-critical process in which decisions have to be made early each year to allow for sufficient vaccine production time.
It would be unfair to blame poor protection in any single year on the vaccine without considering how well the vaccine strains matched the flu strains actually circulating in the following respiratory season. Comparisons between different vaccines deployed during the same influenza season are more meaningful than point estimates; only parallel (i.e., contemporaneous) comparisons can truly give reliable insights into performance characteristics and the VE of different vaccines.
With that in mind and the data shown above, it is not difficult to understand why the CDC / ACIP decided to withdraw its recommendation for LAIV, the only available live attenuated non-injectable vaccine for influenza prophylaxis. LAIV clearly had a disappointingly poor track record for several respiratory seasons in a row compared to IIV, the injectable inactivated flu virus vaccine.
Why then is AstraZeneca claiming the exact opposite: In a press release, they quote excellent VE during the last years of LAIV deployment stating that “studies by AstraZeneca as well as preliminary independent findings by public health authorities in other countries … demonstrate FluMist Quadrivalent was 46-58% effective overall against the circulating influenza strains during the 2015-2016 season”.
What is going on? Are we missing something here?
Let’s have a closer look at the 2 references quoted by AstraZeneca in support of LAIV effectiveness:
The first reference
- is not a publication at all but a slide set from THL, the Finnish Department of Health Protection . For full disclosure, THL is collaborating with vaccine manufacturers in a “public-private partnership”.
- LAIV was only recently introduced in Finland. It seems that only a small fraction of flu vaccine recipients in the Finnish 2015/16 flu season received LAIV: compared to 1.3 million doses of the trivalent inactivated vaccine (TIV), but only a measly 22,500 doses of LAIV were delivered to Finland. Therefore, very few Fins were vaccinated with LAIV compared to TIV.
- The slides mention that VE was 51-60% and 51%-54% for children and elderly, respectively. As these percentages were based on a “Number vaccinated” far greater than all the doses of LAIV delivered, we might have to assume that these VE rates are averages across all vaccines, and thus predominantly rather reflect TIV effectiveness. Sadly, the authors do not report VE of the individual vaccines administered in Finland.
- Note: there was a good match between vaccine strains and circulating flu virus.
In conclusion, these data do not support AZ’s claim of LAIV effectiveness. The data sets reviewed by CDC are not only larger but also comparative in nature. AZ or THL may have more detailed information on file but this is what could be found in the public domain.
The second reference
- provided by AZ comes from pre-Brexit Great Britain . In a cohort of 729 children studied with LAIV, the preliminary VE was 57.6% (CI: 25%, 76%). These results are very different from the CDC data listed above and in favor of LAIV effectiveness.
- Like in the CDC trial, the British study used the TND approach to determine VE, a commonly used methodology. It is based on a retrospective analysis of patients coming to medical attention for a respiratory illness at certain centers that are proficient in influenza lab testing.
- In the British study the size of the LAIV study population was quite small (N=729). We are not provided data from an age-matched comparison group.
- By contrast, the CDC study cohort is 10x larger (N=7563); in addition, a direct comparison was made between LAIV and the injectable inactivated flu vaccine in the same population and the same season.
So, there is no argument that the CDC data is better in quality and quantity. In light of the huge discrepancy of results between 2 studies using very similar methodology, we need to be concerned about the inherent shortcomings of TND (Test-Negative Design).
TND does not match RCT in quality and scientific rigor, not even closely. For that reason, TND results are not referred to as ‘efficacy’ but ‘effectiveness’. TND is neither randomized nor prospectively controlled. It is basically a case control design with its major appeal being expediency. While some believe that TND results compare favorably with those from RCTs, this view can be challenged  .
AstraZeneca is probably not amused with this recent CDC assessment of the LAIV vaccine – its MedImmune subsidiary is already in a tight spot. MedImmune’s infectious disease franchise had been exceptionally unlucky / unsuccessful / unproductive in recent years and kept on being a drag on profitability. Its infectious disease portfolio has shrunk and now only lists an RSV compound and antibodies against S. aureus and P. aeruginosa, all problematic projects for their own individual and very specific reasons.
PS: We just learned (10/1/16) that the H1N1 component of the LAIV vaccine may be the culprit; according to MedImmune, it is not replicating as well as it should. 
ACIP Advisory Committee on Immunization Practices
IIV injectable influenza vaccine
LAIV life-attenuated influenza vaccine
RCT randomized controlled trial
RSV respiratory syncytial virus
TND Test-Negative Design
VE vaccine effectiveness
 H Nohynek for the THL Influenza Research Team. Nordic Vaccine Meeting April 2016 Reykjavik Iceland. Nohynek_NorVac_influenzaLessonsNohynek_short
 Public Health England. Influenza Vaccine Effectiveness in adults and children in primary care in the UK: provisional end-of-season results 2015-16. Influenza_vaccine_effectiveness_in_primary_care_in_children.pdf
 G De Serres. Eurosurveillance, 18: 2013
 S McNeill. Overview of Vaccine Efficacy and Vaccine Effectiveness. Canadian Center for Vaccinology. https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwit4Lml4-DNAhVJXh4KHcZHCVoQFgghMAA&url=http%3A%2F%2Fwww.who.int%2Finfluenza_vaccines_plan%2Fresources%2FSession4_VEfficacy_VEffectiveness.PDF&usg=AFQjCNGoJd1GV_b-ivAFuqv0JA86sTHatA&sig2=RXBFfS3c70GhfgBs-psJPw&bvm=bv.126130881,d.dmo