Posts in: science

March lectures of note

Available to general public!

ChatGPT and Potential Healthcare Implications of Large Language Models

Tests for Early Cancer: Facts vs. Opinions Can We Detect Early Cancer?

It’s a Bacterial World

Bias of the day: immortal time

This is when you do a retrospective study, select cohorts according to exposure, but measure outcomes — usually death, or hospitalization, or something else bad — in a way that guarantees one or more of the cohorts have a period of time when that outcome couldn’t have happened. That’s how you get “immortal”, or “guaranteed” time.

Three classic examples: Courtesy of Bing.

  • A study that evaluated the hypothesis that menopause occurring before age 40 years is associated with the development of cardiovascular disease (CVD). The study counted women who had premature menopause as exposed to CVD risk from their date of birth, ignoring the fact that they could not have developed CVD before menopause.
  • A study that assessed the effect of statins on diabetes progression and insulin use. The study classified patients as statin users based on their first prescription after hospital discharge, but included the time between discharge and prescription as exposed time, even though they were not taking statins during that period.
  • A study that examined the association between beta-blocker use and mortality in patients with heart failure. The study defined beta-blocker exposure based on pharmacy claims after hospitalization, but did not account for the time lag between hospital discharge and drug initiation, which could have biased the results in favor of beta-blockers.

To these three classics we can now add two more, one highly publicized, the other less so, both surprising considering the journals and the supposed peer review they must have gone through:

  • A study that compared hospitalization rates of those who had a positive SARS-CoV-2 test and did not take the Covid-19 drug Paxlovid to those who took Paxlovid regardless of test status. The study counted the day of the positive test as “Day 0” for the untreated cohort; however, for patients who started taking Paxlovid and did not have records of a positive test “Day 0” was one day before treatment start. See this Twitter thread for relevant excerpts and a more detailed explanation. As the study excluded patients who were already hospitalized when they started Paxlovid, but included untreated patients who had a positive test and were hospitalized on the same day, this guaranteed 1 day of “immortal time” for the Paxlovid cohort.
  • A study that used “real world data” to compare standard blood thinners to a procedure called left atrial appendage occlusion (LAAO) for prevention of death and stroke in patients with atrial fibrillation. However, patients who underwent LAAO had to live long enough with atrial fibrillation without getting a stroke to “graduate into” the procedure. Not only is immortal time here so glaring, the study is eerily similar to the original heart transplantation study in which this bias was first identified.

Cardiologist John Mandrola explains in depth why the LAAO paper, and the way it was spun, is particularly egregious.

Note that this is only a problem in retrospective — or, how they now like to be rebranded, “real-world” — studies. As the most recent cases show, these are not only worthless for informing anyone’s real-world decision, but also contribute to the noise, the chaos, and the general fear-uncertainty-doubt of medicine. A voluntary moratorium would not be out of line.

Finished reading: Fundamentals of Clinical Trials by Lawrence M. Friedman 📚

It is assigned reading for a course I’m helping prepare, so I thought I’d better read the book we’ll ask our students to use. Like many textbooks, it suffers from MANE — many authors no editors — and like many academic texts, it can get way too pedantic. Still, it is hard to argue with its overarching themes: that randomized controlled trials are the pinnacle of medical evidence generation, and that much of the trial paperwork done in the name of quality is unnecessary. I have more comments on that last point, but that is for another time.

For Valentine’s Day, The Washington Post decided to write about a promising new male contraceptive drug being studied in mice. That’s fine. What’s unusual, bizarre, and a bit of a troll is hanging a red banner over the front page presenting it as breaking news. Seriously, WaPo?

When Russ Roberts, my favorite interviewer, speaks with Adam Mastroianni, the author of my favorite newsletter, of course I have to share it. No excerpts, just listen to the whole thing, please.

The academic prisoners' dilemma

As of this year, eLife no longer has “accept/reject” decisions after peer review: Which I learned via Andrew Gelman.

All papers that have been peer-reviewed will be published on the eLife website as Reviewed Preprints, accompanied by:

Authors will then receive a paper with a full DOI that can be used on funding applications. They will be able to include a response to the assessment and reviews, and decide what to do next:

  • Revise and resubmit
  • Declare the Reviewed Preprint as the final Version of Record

This is as it should be in the age of unlimited digital space.

The quality of public peer review on eLife seems above average: I have once, as the sole peer review of this paper from a double-digit impact factor journal, Impact factor of eLife, per Wikipedia, is 8.7 received a single sentence which amounted to “sample size too small”, but with more spelling errors and the same lack of punctuation. If your goal when reading a paper is both critical appraisal and learning, you could do worse than reading this exchange.

But! Eleven reviewed preprints total in the last 5 months seems… low? Am I missing other public reviews? I would, for example, very much like to learn what the reviewers said about this.

More generally, I am worried that this will make eLife become the default publication of last resort — trouble for the Infection and Immunity and Leukemia and Lymphomas of the world, but not exactly the killing blow to Science or Nature or most of its million offshoots.

The current, bizarre, inefficient, unsustainable — Byzantine, if you will, thought that is too disrespectful of Byzantium — keeps itself alive through force of reputation. Critical thinking is hard, so unless I am in the opposing team and my goal is to tear down your data I will save many a mental cycle by “trusting the process” and taking the conclusion, abstract, that one piece of information I need to cite in my own work… at face value. And evidence to the contrary be damned, say published in NEJM to a clinician and their ears will perk up.

So we are in a prisoner’s dilemma of sorts. Take a group of one hundred researchers: the average benefit to all of them, and to science in general, would be greater if all published in eLife. But, if 90 of the 100 submit to CNS journals or NEJM first then go down the impact factor list and only 10 shmucks go straight to eLife, there will be only a handful of “winners”, the state of science remains what it is, and everyone ends up wasting so… much… time.

It doesn’t have to be this way – and Covid did expedite some reputational decay – so this is a good a time as any to place a chisel in the crack. What’s needed now is some forceful movement of the hammer and, well, I guess people who publish People who review are equally important, but maybe just maybe we will at one point be able to leave that to an algorithm. It would certainly do a better job than most! are the hammer in this strained analogy.

Should I start with myself? I do have a handful of side projects which are neither industry nor strictly academic — myself having no academic affiliation. Stay tuned.

February lectures of note

NIH has always made most of its lectures available to public. With the pandemic, the production values have gotten better, and more people have gotten used to viewing lectures online. Here are a few interesting ones scheduled for this month. Some of them are part of NIH’s Demystifying Medicine series which is open to public and tries to target the curious layperson Alas, not always successfully. to the best of presenters' abilities.

Fact Stranger than Fiction: Adventures in the Genomics of Inflammation

Somatic Mutations in “Benign” Diseases

  • Speaker: Neal Young, MD
  • Date: Tuesday, February 14, 2023, 3:00 - 4:00 p.m. ET
  • Register here.

The Use of JAK Inhibitors in Autoimmune Disease

Clinical Center Grand Rounds: How Nucleic Acid Structure and Chromatin Environment Influence Gene Transcription

Nitpick of the day: clinical trial versus clinical study

At the very start of the textbook Fundamentals of Clinical Trials the authors make a distinction between clinical trials — comparing two or more different interventions — and clinical studies, which merely describe an intervention without comparing it to anything. So, there can be no such thing as a “Phase 1 trial”, since they typically involve a single drug at different doses and schedules. The only true trials, according to the authors, would fall under Phase 3, or Phase 2b at the earliest.

This is stupid, misleading, and not at all how the words “trial” and “study” are used by anyone else, including the biggest and most important drug regulatory agency in the world. There are many such pointless exercises of professorial power in medicine, including my favorite: whether the correct pronunciation of “+” in “7+3” is “plus” or “and”. They amount to nothing more than purity tests that award the wielders of the right language a false sense of precision. As Nassim Taleb wrote, nitpicking is the enemy of thought.

The rest of the book is good enough, but more on that later.

National Cancer Advisory Board meeting on February 9

The National Cancer Advisory Board is as big of a deal as it sounds, but have a look at their Charter — Description of Duties in particular — and try not to yawn. I suspect whoever wrote this also writes grant application instructions. May this in part be why science is bogged down?

In any case, the Board meetings are open to public. The next one is on February 9, 2023 at 1:15pm EST and can be viewed at videocast.nih.gov. That website is itself a trove of excellent recordings unrelated to federal body deliberations: just last week there was a lecture on bacterial immune systems, here is my old boss talking about interleukin-15, and hey what’s this?

Science and medicine blogs on FeedLand

After a few months of intermittently kicking the tires on Dave Winer’s FeedLand, I’ve finally had the time to port over a few feeds from my preferred RSS reader. The wonderful thing about FeedLand is that you can easily follow my feed categories and read posts without having an account (which is fortunate, since new signups on Winer’s own server are on hold). The full list of feeds is here. There is even a feed of posts I liked! It’s feeds all the way down.

The Science category has your usual suspects but I had to dig deep for Medicine since many of the blogs I follow haven’t been updated in years and others have turned into HuffPost-level text mills. Fortunately, Substack enabled a resurgence of medical writing, with feeds enabled by default.

Did I mention NetNewsWire is a free, open source RSS reader available on MacOS and iOS, and can sync via iCloud? For the anti-Apple readers, Feedly is there, I guess?