A few more notes on that vaccine study

Figures from the which-arm-for-the-second-shot study I wrote about yesterday reminded me of the delightfully titled “An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability” published in PNAS. ↬Dean Eckles To be clear, this is not the error study authors made, but see for yourself the chart that got them a USA Today head nod:

So much variability! And even with 143 samples, still so much uncertainty. But increasing the sample size would help the variability much, as noted in PNAS:

You can squish the error bars, but you can’t squish your study population. Anyway, I thought it was interesting.

But now that we have that figure from the vaccine article up, let’s note two more things:

• The percentages in question are absolutely tiny. Median is 0.06% in the ipsilateral and <0.03% in the contralateral group, if we just eyeball it. At that point you are looking at what, 2–3 dots on the flow cytometry scatterplot? Interoperator variability must be huge.
• Their cutoff for saying that someone “has” spike-directed CD8 T cells was, I presume from the dotted horizontal line labeled “DL”, 0.03%. They based this off of historic values of unvaccinated healthy controls from this paper from the same group. I may be getting into the weeds too much here, but the text of that article says that “0.03% of reactive T cells was set as the detection limit based on the distribution of T cell frequencies after control stimulations”. This is so vague that it’s perfectly useless, but looking at Figure 1 gives it away: they picked one standard deviation away from the mean for the CD4-positive cells as the detection limit for both CD8-positive and CD4-positive spike-specific T cells. And even in that experiment, the variability of spike-specific CD8 T cells is so much greater than that of the CD4+ ones (0.02% versus >1% maximum in the healthy, unvaccinated and unexposed-to-SARS-CoV-2 controls). So they are not only extrapolating old experimental data into the new one, they are extrapolating one cell type into another when no such extrapolation should be made. Put hey, it came out in Nature Medicine, so it must be OK. Sigh.