We like to do things in medicine, and medicine’s big contribution to science was figuring out how best to answer the question of whether the things we do actually work. But of course things aren’t so simple, because “Does it work?” is actually two questions: “Can it work?”, i.e. will an intervention do more harm than good under ideal circumstances, and “Does it work in practice?”, i.e. will an intervention do more good than harm in usual practice.

We also like to complicate things in medicine, so the person to first delinate this distinction, Archie Cochrane of the eponymous collaboration named them efficacy and effectiveness respectively — just similar enough to cause confusion. He also added efficiency for good measure (“Is it worth it?) Fifty years later, people are still grappling with these concepts and talking over each other’s heads when discussing value in health care. Which is to say, it’s best not to use the same prefix for overlapping terms, but if you had to, “eff” is most appropriate.

The most recent example is masks. Cochrane Colaboration’s review said they didn’t “work” ⊕ The paper caused an uproar and language has since been toned down, but that was the gist. for preventing respiratory infections. Now, knowing what Cochrane was all about the first question to ask is: what sense of “work” did the authors intend, and this particular group is all about effectiveness (working in “the real world”), not about efficacy (working under ideal conditions). This caused some major cognitive dissonance among the covid-19 commenters. Vox had the typical sentiment:

Furthermore, neither of those studies [included in the meta-analysis] looked directly at whether people wear masks, but instead at whether people were encouraged or told to wear masks by researchers. If telling people to wear masks doesn’t lead to reduced infections, it may be because masks just don’t work, or it could be because people don’t wear masks when they’re told, or aren’t wearing them correctly.

There’s no clear way to distinguish between those possibilities without more original research — which is not what a meta-analysis of existing work can do.

But this is the difference between ideal (you force a person to wear a mask and monitor their compliance) and typical conditions (you tell the person to wear a mask and keep your fingers crossed), and Cochrane is interested in the latter, ⊕ Though of course, the chasm between ideal and typical circumstances varies by country, and some can do more than others to bring the circumstances closer to ideal, by more or les savory means. which is the one more important to policy-makers.

This is an important point: policy makers make broad choices at a population level, and thus (do? should?) care more about effectiveness. Clinicians, on the other hand, make individual recommendations for which they generally need to know both things: how would this work under ideal conditions, how does it work typically, and — if there is a large discrepancy — what should I do to make the conditions for this particular person closer to the ideal? We could discuss bringing circumstances closer to ideal at the population level as well, but you an ask the people of Australia how well that went.

The great colonoscopy debate is another good example of efficacy versus effectivness. There is no doubt that a perfectly performed colonoscopy at regular intervals will bring the possibility of having colon cancer very close to zero, i.e. the efficacy is as good as you can hope for a medical intervention. But: perfection is contingent on anatomy, behavior, and technique; “regular intervals” can be anything from every 3 months to every 10 years; and there are risks of both the endoscopy and the sedation involved, or major discomfort without the sedation. And thus you get large randomized controlled trials with “negative” results ⊕ Though they do provide plenty of fodder for podcasts and blogs, so, thanks? that don’t end up changing practice.

So with all that in mind, it was… amusing? to see some top-notch mathematicians — including Nassim Taleb! — trying to extrapolate efficacy data out of a data set created to analyze effectivness. The link is to the preprint. Yaneer Bar-Yam, the paper’s first author, has a good X thread as an overivew. To be clear, this is a worthwhile contribution and I’ll read the paper in depth to see whether its methods can be applied to cases where effectiveness data is easier to come by than efficacy (i.e. most of actual clinical practice.) But it is also an example of term confusion, where efficacy and effectiveness are for the most part used interchangeably, except in the legend for Table 1 which say, and I quote:

The two by two table provides the incidence rates of interest in a study of the efficacy (trial) or effectiveness (observational study) of an intervention to reduce risk of infection from an airborne pathogen.

Which seems to imply that you measure efficacy exclusively in trials and effectiveness in observational studies, but that is just not the case (the colonoscopy RCT being the perfect example of an effectiveness trial). And of course it is a spectrum, where efficacy can only be perfectly measured in impossible-to-achieve conditions of 100% adherence and a sample which is completely representative of the population in question so any clinical trial is “tainted” with effectiveness, though of course the further down you are on the Phase 1 to Phase 4 rollercoaster the closer you are to 100% effectivness.

I wonder how much less ill will there would be if the authors on either side realized they were talking about different things. The same amount, most likely, but one could hope…

Update: Not two seconds after I posted this, a JAMA Network Open article titled “Masks During Pandemics Caused by Respiratory Pathogens—Evidence and Implications for Action” popped into my timeline and wouldn’t you know it, it also uses efficacy and effectiveness interchangeably, as a matter of style. This is in a peer-reviewed publication, mind you. They shouldn’t have bothered.

If you thought the state of American media was bad — and justifiably so — I can assure you that most of the world has it much worse. Every so often I get sent a link to a Serbian news site writing about cancer research, and it is always a disaster. Here is the most recent one, short enough to be quotted fully here (translation courtesy of Google):

A German company presented an anti-cancer drug: The tumor stopped growing in all patients

The German company Biontek (BioNTech) is currently raising hopes with its cancer vaccine CARVac.

The first research results show that tumors can be stopped from growing, and sometimes even reduced. The first successes occurred after two out of four vaccination doses.

Most study participants (59 percent) had their tumors shrink by at least 30 percent. In addition, the tumor stopped growing in almost all patients (95 percent) after vaccination. Like the covid 19 vaccine, the vaccine is based on mRNA technology.

This means that a certain protein is taken into the cell, allowing the body to repair it itself.

The new vaccine was developed by a team led by Biontek founder Ugur Sahin (58) and founder Ozlem Turecci (56).

So far, 44 patients have received it in four doses. Success was particularly high after two doses, after four doses the tumors were reduced by at least 30 percent in just under half (45 percent), and the cancer was stabilized in 74 percent of all patients.

Let me list the ways in which this is a terrible new story:

No source

Where did the data come from? Was it a paper, an abstract, a press release, or a leak? A 2-second journey to DuckDuckGo shows that they were, in fact, presented at the 2023 ESMO Congress, which is the annual gathering of the European Society of Medical Oncology. The Serbian website does mention a Bosnian article as a “source” for there copy/paste job, but that article also doesn’t list where the data came from.

Wrong data

“The first research results…”, the article begins. Being the first is big news. But this aren’t the first results. Some were presented last year at the same congress, and even that was a follow-up of data presented earlier.

Incomplete data

Vaccines make the news, so that’s what they highlight, but the trial is actually of a cell therapy with and without the vaccine. The 44 patients they mention are the ones who got the cell therapy with and without the vaccine, and there is no breakdown of how many of them got the actual vaccine. With cancer vaccine’s abysmal past record ⊕ No, they are not now being “tried in cancer” after the success in Covid-19. They were, in fact, developed for cancer treatment, experienced failure after failure, and pivoted back to infectious diseases because of Covid-19; and what a good thing for all of us that they did! I highly doubt that the effect we saw was wholly due to the cells, not the vaccine (then again, I work at a cell therapy company). The paper which came out concomitantly with the abstract shows that about the same number of participants who got the vaccine progressed and responded (see Figure 2 for that).

No context

“The tumor stopped growing in all patients”, the headline says. Well, loog at Figure 2 again, it’s what we call a waterfall plot, which is an aspirational name: if the bar goes up from baseline it means that the tumor grew, if it goes down it means that it shrank, so you want it to look like a waterfall. But in 8 of the 21 participants presented in the paper it grew! And in 5 more it barely came down — those count as “stable disease” because measuring tumors is not a precise science and a pixel here or there on the digital ruler can make all the difference. In only 8 of the participants did the tumor shrink, and in only one of those did it go away completely.

This is, I’m sad to say, about what you would expect for a Phase 1 trial of a cancer drug. Most patients who make it to such a trial have slow-growing tumors, and having a “stable disease” in that context — where you are allowed to have the tumor grow by 20% before calling it “progression” — is perfectly meaningless. Note that you will find terms like “disease control rate” or “clinical benefit rate” which combine participants whose tumors shrunk with those who had this “stable disease”. Those two metrics are also meaningless without a control group.

This became longer than I intended so I’ll stop here, but yes, it’s a sad state. It reminds me of dostarlimab, only much worse since in that case there was at least clear evidence that the drug was good, the only thing missing was context. Caveat lector!

Here is an obvious analogy for you: the physical world — meatspace, if you will — as “meat” of an actual body, both skeletal (muscles, ligaments, tendons and such), and visceral (entrails, the liver, vital organs); the internet as nerve impulses connecting the various parts both sensorially (how are the navels of the world doing these days?) and in effect (from Facebook groups to GoFundMe pages bringing actual change).

You know how X and other social networks made everything feel connected to everything else? Well, there is an organic counterpart to this phenomenon, and it’s called a generalized tonic-clonic — or grand mal — seizure, manifesting, in the clonic phase, in widespread convulsions of the body.

The reason why our bodies are usually not convulsing is that the nerve impulse pathways are tightly controlled in space: there are separate nerves, differentiated brain areas for different roles, and let’s not forget the biggest separation of them all: two semi-independent brain hemispheres connected only by the corpus callosum which, imagine this, is sometimes cut completely for treatment of refractory seizures. There is also chemical separation: many of the pathways are inhibitory, and the most abundant neurotransmitter in the body is not dopamine, serotonin, acetylcholine or others you’ve heard of because they go haywire, but glycin, a modest amino acid which people don’t hear about because it is so good at its job of tamping down bad impulses.

The world’s ongoing convulsions started — after an initial tonic phase — right after we have all become interconnected: Hezbollah, Hamas, and your neighborhood association all hooked up to the same firehose. There is a feeling at the edge of my consciousness that the answer to solving them is in ourselves, and not in a new age self-fulfilment way but in pragmatic steps we can take to extrapolate from this most obvious analogy.

My friend and fellow oncologist Timothée Olivier has just started a YouTube channel called Primum Non Nocere — yay for Latin — and the first video, about reading clinical papers, is well worth 40 minutes of your time.

The recent conversation between Peter Attia and Russ Roberts on cancer screening and longevity has left a good impression, so in case you rushed out to buy his new book, Outlive, here is some thoughtful criticism. Biennial colonoscopies and whole-body MRIs at any frequency are indeed unreasonable.

The first one is tomorrow, and it’s a good one!

• The Ethics of Using Large Language Models by Nick Asendorf, PhD. Wednesday, October 4, 2023 at 12pm EDT.
• Clinical Center Grand Rounds: Pericles and the Plague of Athens by Philip A. Mackowiak, MD, MBA. Wednesday, October 11, 2023 at 12pm EDT.
• Understanding and Addressing Housing Instability for Cancer Survivors by Angela E. Usher, PhD, LCSW, OSW-C and Brenda Adjei, MPA, EdD (and no, I don’t know what most of those acronyms mean). Tuesday, October 17, 2023 at 2pm EDT.
• WALS lecture: The Lives of Bacteria Inside Insects by Nancy A. Mora, PhD. Wednesday, October 25, 2023 at 2pm EDT.
• Reading Remedy Books: Manuscripts and the Making of a National Medical Tradition by Melissa B. Reynolds, PhD. Thursday, November 2, 2023 at 2pm EDT. And yes, I know it’s in November, but I likely would have missed it for the next post since it is so early in the month.

The 2023 Nobel Prize in Medicine went to Katalin Karikó and Drew Weissman, and deservedly so. I do not look forward to the re-writing of history that will inevitably come about the role that the NIH, University of Pennsylvania, and academia in general had in their work. As a reminder:

“Every night I was working: grant, grant, grant,” Karikó remembered, referring to her efforts to obtain funding. “And it came back always no, no, no.”

By 1995, after six years on the faculty at the University of Pennsylvania, Karikó got demoted. She had been on the path to full professorship, but with no money coming in to support her work on mRNA, her bosses saw no point in pressing on.

She was back to the lower rungs of the scientific academy.

“Usually, at that point, people just say goodbye and leave because it’s so horrible,” Karikó said.

She didn’t quit. But even when the breakthrough came, the leading journal saw it as “incremental”:

“The breakthrough, as you put it, we first sent to a Nature journal, and within 24 h, they rejected it as an incremental contribution. I started learning English only at university, so I had to look up the meaning of the word incremental! Anyway, we then sent it to Immunity, and they accepted it (3). We literally did all the work ourselves, Drew and I. Even at the age of 58, I didn’t have much help or funding to perform the experiments, so I did them with my own hands. It took us a while to publish the follow-up paper in Molecular Therapy in 2008, where we presented data on the superior translation of the pseudouridine-containing mRNA and the lack of immune activation in mice.”

The story gets more tangled from there: Karikó and Weissman co-founded a company that failed, then joined BioNTech, and in parallel Moderna started working on their own modified RNA platform, and none of it would have mattered an iota if SARS-CoV-2 hadn’t provided the unfortunate opportunity for mRNA vaccines to shine. For all of our (deserved!) ex post glorification of everyone involved, no Covid-19 — no glory.

Which reminds me very much of The Tartare Steppe’s lonely soldier Drogo who wastes away his life guarding a fortress from the barbarian hordes that don’t arrive until it is too late for him to shine in battle. How lucky for us all that humanity has enough Drogos, and how lucky for this particular pair of soldiers that their Tartars showed up on time.

Emily Fridenmaker, who is a pulmonary disease and critical care physician, writes on X:

Everything is so complex.

Logging into things is complex, placing orders is complex, figuring out who to page is complex, getting notes sent to other doctors is complex, insurance is complex, etc etc. But we just keep doing it.

At what point is all this just too much to ask?

There are a few more posts in that thread, and I encourage you to read all of it to get a sampling of why doctors feel burnt out. Whether you are in medicine, science, or education, your professional interactions have slowly — They Live-style — been replaced by a series of fragile Rube Goldberg machines that worked great in the minds of their technocratic developers, but break, stutter, stammer, and grind to a halt as soon as they encounter another one of their brethren. Which is all the time!

Too much of our professional lives has been spent playing around with a series of Rube Goldberg nesting dolls, ⊕ Before reading I Am a Strange Loop I would have apologized for mixing metaphors, but this is how our brains think and it doesn’t have to make sense in the physical world to be useful, so apology rescinded. 2FA inside a 2FA, and if Apple is wondering why people are taking more and more time to replace their aging iPhones, I bet a good chunk of them dread doing it because they don’t even know how many different authenticating services, email clients, education portals, virtual machines — and all other needless detritus sold to management by professional salespeople — they would need to log back into.

Don’t get me wrong: Rube Goldberg machines are fun to play with — The Incredible Machine was one of my first gaming memories — and they can even be useful for individual workflows. But mandating that others use your string-and-pulley concoction that will break at first unexpected interaction is sadistic. Just this Monday we had yet another AV failure at a weekly lecture held at a high-tech newly-opened campus. I knew there would be trouble the moment I saw that the only way to interact with any AV equipment was via a touchscreen that had no physical buttons and no way to remove the power cord, which was welded to the screen on one end, and went into a closed cabinet on the other. Lo and behold the trouble came not two weeks later: we couldn’t get past the screensaver logo. We ended up asking students to look at their own screens while guest lecturers were speaking — and nowadays everyone carries at least two screens with them to school — which was too bad, because I was looking forward to using the whiteboard which is as far from Rube Goldberg as it gets.

Me from 20 years ago would have salivated for that much technology in my everyday life, but I’m hoping it was a function of the time, not of my age, and that kids-these-days know better. My own kids' experience with the great remote un-learning of 2020–2021 makes me hopeful that they will be more cautious about introducing technological complexity into their lives.

There was a major update today in the Maintenance of Certification saga: the president of ASH (American Society of Hematology, which, oh what a coincidence, I mentioned just yesterday) wrote an open letter to the CEO of ABIM requesting to end MOC as we know it. In what is I am sure a completely unrelated announcement, the CEO of ABIM said he would step down in September 2024. He may want to reconsider that timeline.

I don’t hide my disdain for Eric Topol, and of course one has to wonder whether professional jealousy plays a role; he is, after all, a high-profile doctor with thoughts about technology. But this morning I found an excellent counterfactual in Peter Attia who is slightly closer to me in age, moves in high-profile circles, and spends time “creating content” about what I think is a bit of a time-waster for rich people: prolonging lifespan healthspan. In other words, he carries the perfect confluence of properties to create even more disdain on my part; and yet, I think that overall he is an upstanding guy who is smart, no-nonsense, and great at communicating complex ideas.

This was a long-winded intro to my recommendation for today’s episode of EconTalk, which has confirmed my priors and reminded me that it’s never too early in the week to call Topol a hack. Him and Attia are so similar on paper, so different in reality.