Serbia in the 1990s had a peculiar mass media landscape in that movies were rarely officially released, yet were shown on TV days after premiering through the magic of pirating, practiced by both broadcast and cable networks. The most successful of these was TV Pink, now a horror show of reality TV, and one of the many peculiar features of TV Pink was that its daytime content would rely heavily on those E! making-of fillers and patter interviews with exhausted celebrities on their movie-promoting circuit. While home-bound sick kids of America filled their days with Bob Barker and Jerry Springer, in Serbia it was all Hollywood all the time — unless you were the weirdo who liked to watch reruns of 1960s kids shows and their poorly made hyperinflation-era remakes, which was the only thing state TV was capable of producing.
But that wasn’t me! So when I got bacterial pneumonia back in 6th grade and was stuck at home for two whole weeks while receiving twice-daily intramuscular right-into-the-gluteus aminoglycoside antibiotics — the lackadaisical attitude of Serbian pediatricians towards dosing and toxicities is a different story — Pink took up more of my time than I care to admit, and making-of videos from that period got engrained in my memory more so than the movies themselves. Topping the list was the 1995 comedy Get Shorty starring John Travolta, which back then I thought must have been the biggest blockbuster ever if they were talking about it so much.
All this is a preamble to what I heard said by Travolta, or his co-star Danny DeVito, or maybe it was the director Barry Sonnenfeld, and it was this: the movie was based on a book, and the book was outstanding and written by Elmor Leonard who had a way with writing dialogue, and whenever they had an urge to improvise their lines they would hold back because Leonard must have already thought about the things that came to the actors minds first and decided that, no, this thing on the page was better.
And I have heard that line so many times — my pre-frontal cortex still developing — that I have now completely internalized it and act on it unconsciously. When evaluating someone’s work — outside of grading papers, for that is a different matter entirely — I start with the assumption that they have thought long and hard about the paper they’ve submitted for my peer review, certainly longer than the few hours I can dedicate to reviewing it, and I give them the benefit of the doubt. My first impression is probably something they thought about and dismissed, to come up with what they are submitting. Now, some papers are so egregiously wrong that they will still be red all over after I’m done; but if there is a small difference of opinion, or a nit to pick with style, or something I would maybe have done slightly differently, I just let it be, in deference to the authors' work and respect to their, the editor’s, and — why hide it? — my own, time.
After being on the receiving end of quite a few paper and grant reviews myself Oh, and meetings. So many meetings., I am beginning to suspect that not everyone is following the Get Shorty ethos.
Now, the worst peer review I have ever received was also the shortest. It was for a paper about a clincial study in rare disese that had a one-sentence rejection from the first journal where it was submitted: “Only 11 patients, they need more”. But most other reviews are not “bad” in that sense, but rather overly verbose and nit-picky about the tiniest of details with dozens of comments per review, the purpose of which is not to improve the article, but rather to show to the editor of the prestigious journal — the higher the impact factor, the more nits to pick — that the reviewer was worthy of the invitation to provide his or her services free of charge to the academic publishing machine. Look at me, ma', I’m paying attention!
Which is fine for papers, I guess, since the reviewers will be in the ballpark of your field (those that aren’t won’t accept the review), and you may at least get a chance to respond. Grants are worse: not only are the reviewers forced into it for the prestige of being on a study section, there is little chance if any that they will have the knowledge of your field This is why, I suspect, the best predictor of receiving an NIH grant is already having received an NIH grant. Not only have you been stamped as a success for the “educated lay-people” on the study section, but if you reapply to the study section their knowledge of your field will have been what you told them, and the Program Officer, in prior grant applications. The problem is trebled if you apply with a clinical trial, because all the people with clinical trial expertise across all of the NIH study sections could probably all fit in a Mini. But how much more difficult could designing a clinical trial be from running a lab, eh?
The examples are many and I’m not at liberty to discuss most of them, but back when I was opening trials in T-cell lymphoma a grant was not funded mostly because they didn’t think we could enroll patients for this “rarest of the rare” disease in time (we were well over half-way done with enrollment by the time we heard of this decision). In case you were wondering why all the money goes to breast, colon, and lung cancer research, well, no one ever had a problem recruiting for those!
There is a role for peer review: to weed out the impossible and the truly un-fundable. But after that it may as well be a lottery: why would rolling the dice be worse than adding up laundry lists of small, irrelevant issues that could sink an application which gets assigned to two or three particularly detail-oriented reviewers. It would also save a hell of a lot of time for everyone involved.