Posts in: science

The Get Shorty style of peer review

Serbia in the 1990s had a peculiar mass media landscape in that movies were rarely officially released, yet were shown on TV days after premiering through the magic of pirating, practiced by both broadcast and cable networks. The most successful of these was TV Pink, now a horror show of reality TV, and one of the many peculiar features of TV Pink was that its daytime content would rely heavily on those E! making-of fillers and patter interviews with exhausted celebrities on their movie-promoting circuit. While home-bound sick kids of America filled their days with Bob Barker and Jerry Springer, in Serbia it was all Hollywood all the time — unless you were the weirdo who liked to watch reruns of 1960s kids shows and their poorly made hyperinflation-era remakes, which was the only thing state TV was capable of producing.

But that wasn’t me! So when I got bacterial pneumonia back in 6th grade and was stuck at home for two whole weeks while receiving twice-daily intramuscular right-into-the-gluteus aminoglycoside antibiotics — the lackadaisical attitude of Serbian pediatricians towards dosing and toxicities is a different story — Pink took up more of my time than I care to admit, and making-of videos from that period got engrained in my memory more so than the movies themselves. Topping the list was the 1995 comedy Get Shorty starring John Travolta, which back then I thought must have been the biggest blockbuster ever if they were talking about it so much.

All this is a preamble to what I heard said by Travolta, or his co-star Danny DeVito, or maybe it was the director Barry Sonnenfeld, and it was this: the movie was based on a book, and the book was outstanding and written by Elmor Leonard who had a way with writing dialogue, and whenever they had an urge to improvise their lines they would hold back because Leonard must have already thought about the things that came to the actors minds first and decided that, no, this thing on the page was better.

And I have heard that line so many times — my pre-frontal cortex still developing — that I have now completely internalized it and act on it unconsciously. When evaluating someone’s work — outside of grading papers, for that is a different matter entirely — I start with the assumption that they have thought long and hard about the paper they’ve submitted for my peer review, certainly longer than the few hours I can dedicate to reviewing it, and I give them the benefit of the doubt. My first impression is probably something they thought about and dismissed, to come up with what they are submitting. Now, some papers are so egregiously wrong that they will still be red all over after I’m done; but if there is a small difference of opinion, or a nit to pick with style, or something I would maybe have done slightly differently, I just let it be, in deference to the authors' work and respect to their, the editor’s, and — why hide it? — my own, time.

After being on the receiving end of quite a few paper and grant reviews myself Oh, and meetings. So many meetings., I am beginning to suspect that not everyone is following the Get Shorty ethos.

Now, the worst peer review I have ever received was also the shortest. It was for a paper about a clincial study in rare disese that had a one-sentence rejection from the first journal where it was submitted: “Only 11 patients, they need more”. But most other reviews are not “bad” in that sense, but rather overly verbose and nit-picky about the tiniest of details with dozens of comments per review, the purpose of which is not to improve the article, but rather to show to the editor of the prestigious journal — the higher the impact factor, the more nits to pick — that the reviewer was worthy of the invitation to provide his or her services free of charge to the academic publishing machine. Look at me, ma', I’m paying attention!

Which is fine for papers, I guess, since the reviewers will be in the ballpark of your field (those that aren’t won’t accept the review), and you may at least get a chance to respond. Grants are worse: not only are the reviewers forced into it for the prestige of being on a study section, there is little chance if any that they will have the knowledge of your field This is why, I suspect, the best predictor of receiving an NIH grant is already having received an NIH grant. Not only have you been stamped as a success for the “educated lay-people” on the study section, but if you reapply to the study section their knowledge of your field will have been what you told them, and the Program Officer, in prior grant applications. The problem is trebled if you apply with a clinical trial, because all the people with clinical trial expertise across all of the NIH study sections could probably all fit in a Mini. But how much more difficult could designing a clinical trial be from running a lab, eh?

The examples are many and I’m not at liberty to discuss most of them, but back when I was opening trials in T-cell lymphoma a grant was not funded mostly because they didn’t think we could enroll patients for this “rarest of the rare” disease in time (we were well over half-way done with enrollment by the time we heard of this decision). In case you were wondering why all the money goes to breast, colon, and lung cancer research, well, no one ever had a problem recruiting for those!

There is a role for peer review: to weed out the impossible and the truly un-fundable. But after that it may as well be a lottery: why would rolling the dice be worse than adding up laundry lists of small, irrelevant issues that could sink an application which gets assigned to two or three particularly detail-oriented reviewers. It would also save a hell of a lot of time for everyone involved.


Tim Harford:

The proportion of genetics papers with autocorrect errors was estimated in 2020 to have reached 30 per cent. The Human Gene Name Consortium decided to rename the genes in question, wisely accepting that this would be easier than weaning researchers away from Excel.

At the intersection of science and technology lies the festering boil that is Microsoft Office.


Education as scaffolding

Adam Mastroianni had another interview with Russ Roberts, and this one didn’t sit with me as well as his previous appearance. They talked about things we learn in school — higher education, for the most part — and what the point of it all was when most of what we cram in is forgotten.

They gave several examples of this, all of which I have quickly forgotten (ha!), but listing things we were made to learn in medical school never to use again was a popular past time on #medtwitter so I will list a few topics that come to mind first:

  • Bernoulli’s equation
  • the Krebs cycle
  • names of all 12 branches of the maxillary artery
  • formula for the independent two-sample t-test
  • recommended step height and depth for elementary school stairwells

Though that last one was clearly a relic of Serbia’s socialist past, the first four weren’t, and are still being taught in pre-med courses and medical schools around the world. If the goal was to have every doctor know all of these throughout their careers, well, mission failed. But why would we even want that to happen?

Well, I have come around a bit since that 6-year-old tweet and came to appreciate the exposure to different concepts as the scaffolding to whatever career we end up in. No one cries out, after a skyscraper is complete, about all the money and time wasted putting up a scaffolding, setting up cranes, temporary elevators, and such. It is not a perfect analogy since most people in higher education don’t have a blueprint — not even medical students since a doctors' job can be anything from an artist (plastic surgery) to woodworking (orthopedics) to glorified administrative asssistant (general practitioners in most countries) — so it is like building a scaffolding to nowhere, parts of which ossify into the building proper, parts of which decay with time, and parts of which you dismantle as soon as it seems safe to do so, since you hate them from the bottom of your being.

“Why ever did I bother learning about the Krebs cycle five different times!?” Twice in high school — biology and chemistry separately, and three times in medical school — chemistry proper, biochemistry, and physiology. I cry out now, as a hematologist/oncologist without a regular practice; but things could have taken a turn towards a career in organic chemistry, or genetics, or one of those specialties where the cycle is more relevant (though really oncology may very well be one of them!)

The poor Krebs cycle is notorious because it is repeated so often without practical use for most of medicine, but there are many more such concepts throughout life that went in one ear and out the other (Are viruses causing hemorrhagic fevers made of DNA or RNA? Well, I knew it for my USMLE Step 1 exam!) RNA, says Wikipedia.

The scaffolding analogy puts a slightly different spin on grades as well, which could be a rather useful signal of where your construction should go and what kind of a building you want to make, and not your worth as a human being that most teachers and some students want it to be. But let’s not bring up grades again.

So I was surprised by Mastroianni’s and Roberts' surprise about us forgetting — because of course we do! And if the intent of the teachers was to instill knowledge that will last forever and ever, well, most of it is a miserable failure, except for that one sliver of insight that each of us carry for life. But the slivers are different for each of us, and to appreciate your unique sliver you may still need background knowledge that you will eventually forget — the more specialized the area, the more background knowledge needed, so good luck trying to untangle that web.


When the “Nobel Prize for Economics” gets announced, and people cry out that well-actually it’s the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel and it’s not part of Alfred Nobel’s original endowment, it is because this joker has won it, so how valuable could it possibly be? On the other hand, the most recent award went to the genuinely brilliant Claudia Goldin — here is a good pre-award interview — and the other Nobel prizes also went to some real ding-dongs. Things are never so clear-cut.


When did medical conference sessions become Apple keynotes?

The last time I attended a large annual meeting of a professional society was in 2019, and either things have taken a strange turn in the last 4 years, or hematology and oncology are so different that attending any other society’s opening session feels uncanny.

I say this because I attended one yesterday — see the other post from today — and if my eyes could have rolled all the way back into my skull, they would have. This is what I am used to: speakers standing behind the podium — usually elevated, sometimes not — looking straight ahead, reading off the teleprompter more or less skillfully — since after all they are doctors and PhDs, not professional salespeople — occasionally averting their gaze in an attempt to connect with the crowd, which is of course impossible because of the glaring stage lights, but all is forgiven because, after all, we are there to learn, not to be entertained.

Well, someone must have come in and told the medicine men they were doing it all wrong, because this session looked like an Apple keynote — the boring parts, where Tim Cook talks about how many stores they opened — crossed with a TED Talk from a dubious but super-enthusiastic liberal arts professor. There is no podium to anchor yourself, so you and your hands are all over the place gesticulating wildly — power-posing, perhaps? — while you gaze into the teleprompter positioned at 45° above the horizon to give you that contemplative look.

This style of presentation is cringy even when most tech companies do it: Apple is Apple; the other big ones don’t even try to compete, and it’s not until you come to the mid to lower-tier This is a link to the Procreate Dreams reveal video. It is an excellent product which I will definitely get for my kidds to play with, and the creators seem rightfully proud of what they accomplished, but copying Apple did it a disservice. that the “person emoting in front of a large professionally-made slide” style re-emerges.

I wish that was all there was to it. Alas, there was a keynote speaker for this openning session of a professional medical society, and the keynote speaker was — it was at this point that I realized the universe was making me pay for coming to Hawaii — a YouTuber! Which is fine, only they decided to set their talk to background music and flashy videos, and pepper it with excerpts from their own content, letting a good personal story of their family and education get drowned in dumb glitz.

Therefore, to be possess’d with double pompe,
To guard a Title, that was rich before;
To gilde refined Gold, to paint the Lilly;
To throw a perfume on the Violet,
To smooth the yce, or adde another hew
Vnto the Raine-bow; or with Taper-light
To seeke the beauteous eye of heauen to garnish,
Is wastefull, and ridiculous excesse.
William Shakespeare, King John, 1623

Four centuries later, we are in full-on gild mode.


Last week I went to Bethesda for the NIH hematology/oncology fellowship’s “career panel”. They didn’t have those back in my day so I can’t attest to their value for the heme/onc fellows, but I learned a lot! One interesting tidbit noted by a panelist: academia emphasizes owning the idea, industry emphasizes owning the execution.


Some wise words from Thomas Basbøll:

It is not whether what you are saying is true, but how you respond when someone tells you that you are wrong, that determines whether you’re an academic (or at least what kind of academic you are.)

He blogs at Inframethodology, which is a wonderful resource for academic writers.


By the way, these were just the lectures that interested me. The entire NIH calendar of lectures and courses is freely available to everyone, and the ones that are videocast and/or provide CME are marked as such, so feel free to make your own autodidact list.


October lectures of note

The first one is tomorrow, and it’s a good one!


Sometimes, the Tartars do show up

The 2023 Nobel Prize in Medicine went to Katalin Karikó and Drew Weissman, and deservedly so. I do not look forward to the re-writing of history that will inevitably come about the role that the NIH, University of Pennsylvania, and academia in general had in their work. As a reminder:

“Every night I was working: grant, grant, grant,” Karikó remembered, referring to her efforts to obtain funding. “And it came back always no, no, no.”

By 1995, after six years on the faculty at the University of Pennsylvania, Karikó got demoted. She had been on the path to full professorship, but with no money coming in to support her work on mRNA, her bosses saw no point in pressing on.

She was back to the lower rungs of the scientific academy.

“Usually, at that point, people just say goodbye and leave because it’s so horrible,” Karikó said.

She didn’t quit. But even when the breakthrough came, the leading journal saw it as “incremental”:

“The breakthrough, as you put it, we first sent to a Nature journal, and within 24 h, they rejected it as an incremental contribution. I started learning English only at university, so I had to look up the meaning of the word incremental! Anyway, we then sent it to Immunity, and they accepted it (3). We literally did all the work ourselves, Drew and I. Even at the age of 58, I didn’t have much help or funding to perform the experiments, so I did them with my own hands. It took us a while to publish the follow-up paper in Molecular Therapy in 2008, where we presented data on the superior translation of the pseudouridine-containing mRNA and the lack of immune activation in mice.”

The story gets more tangled from there: Karikó and Weissman co-founded a company that failed, then joined BioNTech, and in parallel Moderna started working on their own modified RNA platform, and none of it would have mattered an iota if SARS-CoV-2 hadn’t provided the unfortunate opportunity for mRNA vaccines to shine. For all of our (deserved!) ex post glorification of everyone involved, no Covid-19 — no glory.

Which reminds me very much of The Tartare Steppe’s lonely soldier Drogo who wastes away his life guarding a fortress from the barbarian hordes that don’t arrive until it is too late for him to shine in battle. How lucky for us all that humanity has enough Drogos, and how lucky for this particular pair of soldiers that their Tartars showed up on time.