Next Wednesday looks busy, but there is a Thanksgiving-sized gap in the calendar.

• Development of a CMV Vaccine: From Bench to Congress by Sallie Permar, MD PhD; Wednesday November 8 at 12pm EST
• Unraveling Regulation and Unexpected Role of Interleukin 17 by Huyn Park; Wednesday November 8 at 4pm EST
• Nanomedicine: Which Cancerst to Treat by Gregory Lanza, MD PhD is a 3-hour workshop that is free to public requires pre-registration; held today at 2pm EST, and if you can’t make it there is one on Wednesday, November 8 and Tuesday, November 14, also at 2pm EST
• Exploratory and Inferential Spatial Statistical Methods: Tools to Understand the Geography of Health Across the U.S. by Loni Philip Tabb, PhD; Tuesday, November 18 at 1pm EST, open to public by requires registration

As a long-time fan of Goodhart’s law — see here, here, and of course here — I can only nod my head in appreciation of Adam Mastroianni’s concept of self-Goodharting. My first encounter with the concept, and still an excellent introduction, was an article on Ribbonfarm.

We like to do things in medicine, and medicine’s big contribution to science was figuring out how best to answer the question of whether the things we do actually work. But of course things aren’t so simple, because “Does it work?” is actually two questions: “Can it work?”, i.e. will an intervention do more harm than good under ideal circumstances, and “Does it work in practice?”, i.e. will an intervention do more good than harm in usual practice.

We also like to complicate things in medicine, so the person to first delinate this distinction, Archie Cochrane of the eponymous collaboration named them efficacy and effectiveness respectively — just similar enough to cause confusion. He also added efficiency for good measure (“Is it worth it?) Fifty years later, people are still grappling with these concepts and talking over each other’s heads when discussing value in health care. Which is to say, it’s best not to use the same prefix for overlapping terms, but if you had to, “eff” is most appropriate.

The most recent example is masks. Cochrane Colaboration’s review said they didn’t “work” ⊕ The paper caused an uproar and language has since been toned down, but that was the gist. for preventing respiratory infections. Now, knowing what Cochrane was all about the first question to ask is: what sense of “work” did the authors intend, and this particular group is all about effectiveness (working in “the real world”), not about efficacy (working under ideal conditions). This caused some major cognitive dissonance among the covid-19 commenters. Vox had the typical sentiment:

Furthermore, neither of those studies [included in the meta-analysis] looked directly at whether people wear masks, but instead at whether people were encouraged or told to wear masks by researchers. If telling people to wear masks doesn’t lead to reduced infections, it may be because masks just don’t work, or it could be because people don’t wear masks when they’re told, or aren’t wearing them correctly.

There’s no clear way to distinguish between those possibilities without more original research — which is not what a meta-analysis of existing work can do.

But this is the difference between ideal (you force a person to wear a mask and monitor their compliance) and typical conditions (you tell the person to wear a mask and keep your fingers crossed), and Cochrane is interested in the latter, ⊕ Though of course, the chasm between ideal and typical circumstances varies by country, and some can do more than others to bring the circumstances closer to ideal, by more or les savory means. which is the one more important to policy-makers.

This is an important point: policy makers make broad choices at a population level, and thus (do? should?) care more about effectiveness. Clinicians, on the other hand, make individual recommendations for which they generally need to know both things: how would this work under ideal conditions, how does it work typically, and — if there is a large discrepancy — what should I do to make the conditions for this particular person closer to the ideal? We could discuss bringing circumstances closer to ideal at the population level as well, but you an ask the people of Australia how well that went.

The great colonoscopy debate is another good example of efficacy versus effectivness. There is no doubt that a perfectly performed colonoscopy at regular intervals will bring the possibility of having colon cancer very close to zero, i.e. the efficacy is as good as you can hope for a medical intervention. But: perfection is contingent on anatomy, behavior, and technique; “regular intervals” can be anything from every 3 months to every 10 years; and there are risks of both the endoscopy and the sedation involved, or major discomfort without the sedation. And thus you get large randomized controlled trials with “negative” results ⊕ Though they do provide plenty of fodder for podcasts and blogs, so, thanks? that don’t end up changing practice.

So with all that in mind, it was… amusing? to see some top-notch mathematicians — including Nassim Taleb! — trying to extrapolate efficacy data out of a data set created to analyze effectivness. The link is to the preprint. Yaneer Bar-Yam, the paper’s first author, has a good X thread as an overivew. To be clear, this is a worthwhile contribution and I’ll read the paper in depth to see whether its methods can be applied to cases where effectiveness data is easier to come by than efficacy (i.e. most of actual clinical practice.) But it is also an example of term confusion, where efficacy and effectiveness are for the most part used interchangeably, except in the legend for Table 1 which say, and I quote:

The two by two table provides the incidence rates of interest in a study of the efficacy (trial) or effectiveness (observational study) of an intervention to reduce risk of infection from an airborne pathogen.

Which seems to imply that you measure efficacy exclusively in trials and effectiveness in observational studies, but that is just not the case (the colonoscopy RCT being the perfect example of an effectiveness trial). And of course it is a spectrum, where efficacy can only be perfectly measured in impossible-to-achieve conditions of 100% adherence and a sample which is completely representative of the population in question so any clinical trial is “tainted” with effectiveness, though of course the further down you are on the Phase 1 to Phase 4 rollercoaster the closer you are to 100% effectivness.

I wonder how much less ill will there would be if the authors on either side realized they were talking about different things. The same amount, most likely, but one could hope…

Update: Not two seconds after I posted this, a JAMA Network Open article titled “Masks During Pandemics Caused by Respiratory Pathogens—Evidence and Implications for Action” popped into my timeline and wouldn’t you know it, it also uses efficacy and effectiveness interchangeably, as a matter of style. This is in a peer-reviewed publication, mind you. They shouldn’t have bothered.

This list of NCI’s Lasker Clinical Research Scholars has some familiar faces, and I couldn’t be more proud. These are all MDs and MD/PhDs who are forgoing lucrative careers in industry and private practice and exposing themselves to metric tonnes of federal red tape, all to find cures for rare and neglected cancers (looking at you, T-cell lymphomas and AIDS-related malignancies). May their Tartars show up.

I’ve just spent 45 minutes teaching a dozen and a half first-graders how to use a microscope — with mixed success — and it was the best time I’ve had all week. We looked at frog’s blood, the leg of a housefly, paramecia, and some pollen, all of which sound like something a witch would have on hand. Perfect for Halloween!

Serbia in the 1990s had a peculiar mass media landscape in that movies were rarely officially released, yet were shown on TV days after premiering through the magic of pirating, practiced by both broadcast and cable networks. The most successful of these was TV Pink, now a horror show of reality TV, and one of the many peculiar features of TV Pink was that its daytime content would rely heavily on those E! making-of fillers and patter interviews with exhausted celebrities on their movie-promoting circuit. While home-bound sick kids of America filled their days with Bob Barker and Jerry Springer, in Serbia it was all Hollywood all the time — unless you were the weirdo who liked to watch reruns of 1960s kids shows and their poorly made hyperinflation-era remakes, which was the only thing state TV was capable of producing.

But that wasn’t me! So when I got bacterial pneumonia back in 6th grade and was stuck at home for two whole weeks while receiving twice-daily intramuscular right-into-the-gluteus aminoglycoside antibiotics — the lackadaisical attitude of Serbian pediatricians towards dosing and toxicities is a different story — Pink took up more of my time than I care to admit, and making-of videos from that period got engrained in my memory more so than the movies themselves. Topping the list was the 1995 comedy Get Shorty starring John Travolta, which back then I thought must have been the biggest blockbuster ever if they were talking about it so much.

All this is a preamble to what I heard said by Travolta, or his co-star Danny DeVito, or maybe it was the director Barry Sonnenfeld, and it was this: the movie was based on a book, and the book was outstanding and written by Elmor Leonard who had a way with writing dialogue, and whenever they had an urge to improvise their lines they would hold back because Leonard must have already thought about the things that came to the actors minds first and decided that, no, this thing on the page was better.

And I have heard that line so many times — my pre-frontal cortex still developing — that I have now completely internalized it and act on it unconsciously. When evaluating someone’s work — outside of grading papers, for that is a different matter entirely — I start with the assumption that they have thought long and hard about the paper they’ve submitted for my peer review, certainly longer than the few hours I can dedicate to reviewing it, and I give them the benefit of the doubt. My first impression is probably something they thought about and dismissed, to come up with what they are submitting. Now, some papers are so egregiously wrong that they will still be red all over after I’m done; but if there is a small difference of opinion, or a nit to pick with style, or something I would maybe have done slightly differently, I just let it be, in deference to the authors' work and respect to their, the editor’s, and — why hide it? — my own, time.

After being on the receiving end of quite a few paper and grant reviews myself ⊕ Oh, and meetings. So many meetings., I am beginning to suspect that not everyone is following the Get Shorty ethos.

Now, the worst peer review I have ever received was also the shortest. It was for a paper about a clincial study in rare disese that had a one-sentence rejection from the first journal where it was submitted: “Only 11 patients, they need more”. But most other reviews are not “bad” in that sense, but rather overly verbose and nit-picky about the tiniest of details with dozens of comments per review, the purpose of which is not to improve the article, but rather to show to the editor of the prestigious journal — the higher the impact factor, the more nits to pick — that the reviewer was worthy of the invitation to provide his or her services free of charge to the academic publishing machine. Look at me, ma', I’m paying attention!

Which is fine for papers, I guess, since the reviewers will be in the ballpark of your field (those that aren’t won’t accept the review), and you may at least get a chance to respond. Grants are worse: not only are the reviewers forced into it for the prestige of being on a study section, there is little chance if any that they will have the knowledge of your field ⊕ This is why, I suspect, the best predictor of receiving an NIH grant is already having received an NIH grant. Not only have you been stamped as a success for the “educated lay-people” on the study section, but if you reapply to the study section their knowledge of your field will have been what you told them, and the Program Officer, in prior grant applications. The problem is trebled if you apply with a clinical trial, because all the people with clinical trial expertise across all of the NIH study sections could probably all fit in a Mini. But how much more difficult could designing a clinical trial be from running a lab, eh?

The examples are many and I’m not at liberty to discuss most of them, but back when I was opening trials in T-cell lymphoma a grant was not funded mostly because they didn’t think we could enroll patients for this “rarest of the rare” disease in time (we were well over half-way done with enrollment by the time we heard of this decision). In case you were wondering why all the money goes to breast, colon, and lung cancer research, well, no one ever had a problem recruiting for those!

There is a role for peer review: to weed out the impossible and the truly un-fundable. But after that it may as well be a lottery: why would rolling the dice be worse than adding up laundry lists of small, irrelevant issues that could sink an application which gets assigned to two or three particularly detail-oriented reviewers. It would also save a hell of a lot of time for everyone involved.

Tim Harford:

The proportion of genetics papers with autocorrect errors was estimated in 2020 to have reached 30 per cent. The Human Gene Name Consortium decided to rename the genes in question, wisely accepting that this would be easier than weaning researchers away from Excel.

At the intersection of science and technology lies the festering boil that is Microsoft Office.

Adam Mastroianni had another interview with Russ Roberts, and this one didn’t sit with me as well as his previous appearance. They talked about things we learn in school — higher education, for the most part — and what the point of it all was when most of what we cram in is forgotten.

They gave several examples of this, all of which I have quickly forgotten (ha!), but listing things we were made to learn in medical school never to use again was a popular past time on #medtwitter so I will list a few topics that come to mind first:

• Bernoulli’s equation
• the Krebs cycle
• names of all 12 branches of the maxillary artery
• formula for the independent two-sample t-test
• recommended step height and depth for elementary school stairwells

Though that last one was clearly a relic of Serbia’s socialist past, the first four weren’t, and are still being taught in pre-med courses and medical schools around the world. If the goal was to have every doctor know all of these throughout their careers, well, mission failed. But why would we even want that to happen?

Well, I have come around a bit since that 6-year-old tweet and came to appreciate the exposure to different concepts as the scaffolding to whatever career we end up in. No one cries out, after a skyscraper is complete, about all the money and time wasted putting up a scaffolding, setting up cranes, temporary elevators, and such. It is not a perfect analogy since most people in higher education don’t have a blueprint — not even medical students since a doctors' job can be anything from an artist (plastic surgery) to woodworking (orthopedics) to glorified administrative asssistant (general practitioners in most countries) — so it is like building a scaffolding to nowhere, parts of which ossify into the building proper, parts of which decay with time, and parts of which you dismantle as soon as it seems safe to do so, since you hate them from the bottom of your being.

“Why ever did I bother learning about the Krebs cycle five different times!?” ⊕ Twice in high school — biology and chemistry separately, and three times in medical school — chemistry proper, biochemistry, and physiology. I cry out now, as a hematologist/oncologist without a regular practice; but things could have taken a turn towards a career in organic chemistry, or genetics, or one of those specialties where the cycle is more relevant (though really oncology may very well be one of them!)

The poor Krebs cycle is notorious because it is repeated so often without practical use for most of medicine, but there are many more such concepts throughout life that went in one ear and out the other (Are viruses causing hemorrhagic fevers made of DNA or RNA? Well, I knew it for my USMLE Step 1 exam!) ⊕ RNA, says Wikipedia.

The scaffolding analogy puts a slightly different spin on grades as well, which could be a rather useful signal of where your construction should go and what kind of a building you want to make, and not your worth as a human being that most teachers and some students want it to be. But let’s not bring up grades again.

So I was surprised by Mastroianni’s and Roberts' surprise about us forgetting — because of course we do! And if the intent of the teachers was to instill knowledge that will last forever and ever, well, most of it is a miserable failure, except for that one sliver of insight that each of us carry for life. But the slivers are different for each of us, and to appreciate your unique sliver you may still need background knowledge that you will eventually forget — the more specialized the area, the more background knowledge needed, so good luck trying to untangle that web.

When the “Nobel Prize for Economics” gets announced, and people cry out that well-actually it’s the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel and it’s not part of Alfred Nobel’s original endowment, it is because this joker has won it, so how valuable could it possibly be? On the other hand, the most recent award went to the genuinely brilliant Claudia Goldin — here is a good pre-award interview — and the other Nobel prizes also went to some real ding-dongs. Things are never so clear-cut.

The last time I attended a large annual meeting of a professional society was in 2019, and either things have taken a strange turn in the last 4 years, or hematology and oncology are so different that attending any other society’s opening session feels uncanny.

I say this because I attended one yesterday — see the other post from today — and if my eyes could have rolled all the way back into my skull, they would have. This is what I am used to: speakers standing behind the podium — usually elevated, sometimes not — looking straight ahead, reading off the teleprompter more or less skillfully — since after all they are doctors and PhDs, not professional salespeople — occasionally averting their gaze in an attempt to connect with the crowd, which is of course impossible because of the glaring stage lights, but all is forgiven because, after all, we are there to learn, not to be entertained.

Well, someone must have come in and told the medicine men they were doing it all wrong, because this session looked like an Apple keynote — the boring parts, where Tim Cook talks about how many stores they opened — crossed with a TED Talk from a dubious but super-enthusiastic liberal arts professor. There is no podium to anchor yourself, so you and your hands are all over the place gesticulating wildly — power-posing, perhaps? — while you gaze into the teleprompter positioned at 45° above the horizon to give you that contemplative look.

This style of presentation is cringy even when most tech companies do it: Apple is Apple; the other big ones don’t even try to compete, and it’s not until you come to the mid to lower-tier This is a link to the Procreate Dreams reveal video. It is an excellent product which I will definitely get for my kidds to play with, and the creators seem rightfully proud of what they accomplished, but copying Apple did it a disservice. that the “person emoting in front of a large professionally-made slide” style re-emerges.

I wish that was all there was to it. Alas, there was a keynote speaker for this openning session of a professional medical society, and the keynote speaker was — it was at this point that I realized the universe was making me pay for coming to Hawaii — a YouTuber! Which is fine, only they decided to set their talk to background music and flashy videos, and pepper it with excerpts from their own content, letting a good personal story of their family and education get drowned in dumb glitz.

Therefore, to be possess’d with double pompe,
To guard a Title, that was rich before;
To gilde refined Gold, to paint the Lilly;
To throw a perfume on the Violet,
To smooth the yce, or adde another hew
Vnto the Raine-bow; or with Taper-light
To seeke the beauteous eye of heauen to garnish,
Is wastefull, and ridiculous excesse.

Four centuries later, we are in full-on gild mode.