Doc in a Box from Alex Tabarrok links to an official state government document, from the Utah Department of Commerce. The document is titled “Key Statistics on the Doctronic Pilot Program” but reads more like a bulleted press release, full of percentages without a denominator, begging for a flow chart. Press releases are like that because you typically won’t add images — although this one randomly selected from today does indeed include it along with the full abstract submitted to the ASCO annual meeting, and good for them — but more importantly because you want to pick the best possible picture-perfect view of your shiny spotless data elephant without also acknowledging that it has a rear end, a bunch of flies buzzing around, smells a bit rank. Does your elephant not have an ass, Utah? Or did you just copy/paste what Doctronic — a startup whose wonky web page doesn’t even work — sent you?
Doctronic.
We have hit a technical snag. Go to Homepage to hit it again.
So how many patients could they have evaluated? This article in JAMA Forum says that “[p]hysicians hired by Doctronic will review the AI’s output for the first 250 patients before the system takes any action and will review the next 1000 patients retrospectively after the AI agent begins acting autonomously.” Are the key statistics from the first 250? The very first bullet point in the press release summary document says that the program is still in Phase One and that “the number of patients so far is limited”, so I guess not. Is it 100 at least? Surely they wouldn’t use a percentage as high as 97 if there were fewer than that involved. Except that as low as 30 will give you a percent roundable to 97. So, 30 to 249?
Why am I being so pedantic? Well, these techniques are par for the course in biotech world but coming from a state agency make me think there is a bit too much enthusiasm for it, coming from a government source. Compare and contrast to the shellacking LLMs got in this report from the Office of the Auditor General of Ontario, which reviewed AI Scribe functionality from 20 vendors. Their report even has absolute numbers in it! These state government officials should realize that they are prime targets for flim flam merchants and should behave accordingly.
Note that I am not against the idea in general. The project’s goal is in fact quite noble: there is no reason why plain ol’ machine learning shouldn’t be able to suss out majority of refill requests for chronic medications and flag patients who haven’t had their bloodwork or diabetic foot assessments done, or who’ve had abnormal office blood pressure readings at prior visits. Having that easy refill option available would mean a patient coming in for an in-person visit for what should be “only” prescription refills is even more of a signal that something else may be amiss, even if the patient can’t or won’t verbalize it. So yes, LLM refills, bring ’em on. Doctronic’s end-goal of actual autonomous Shoggoths putting on white coats and replacing MDs, PAs, NPs and other credentialed humans… not so much.