Somehow I can’t see myself ever intentionally using a naked ChatGPT interface for much of anything, especially, however, not for medicine. I’ll have to look at the Claude version, as I’ve been using Claude for some other things successfully, but my go to LLM for medical work remains Open Evidence. Thanks for the summary!
The problem with AI, for me anyway, is that it takes twice as long to use it. First I read the AI answer, then I read other sources to verify. It's getting more accurate, but there are still errors. Might be because im a pediatrician. Peds is usually the last subject they tackle in primary care. I was a beta tester for Up To Date. It was crap in peds at that stage. But they've improved a lot and now I use it
My experience, also. However, with Open Evidence, in most cases, I have been able to recognize temporizing or equivocal responses to real clinical questions, in which case I do review the sources, vs the times where the response immediately reinforces my clinical gestalt.
So I hadn't used Chat GPT for quite a while. I used OE and DoxGPT somewhat interchangeably for my medical work when I ran across ChatGPT Health. Trying to figure out WTH it was seemed like a Herculean effort. I kept getting info re a waitlist. Anyhow, I used it as a good reason to check out where and when one source was better than the other. I used the wholly independent CoPilot 😂😜 to find my answers.
This was the quick answer I got:
1) Many physicians still use ChatGPT for broad reasoning and explanation, then OpenEvidence to verify the evidence base.
2) DoxGPT is increasingly used for drug safety, quick lookups, and specialty‑specific prompts.
3) OpenEvidence and DoxGPT are emerging as complements, not replacements, for general LLMs.
The whole endeavor was useful in that I've started using ChatGPT again and am looking forward to what their healthcare version offers that is different. However, without EPIC, it will be hard to arrange a BAA that will allow it to be used with HIPAA compliance. Both DoxGPT and OE are compliant and you can store notes with PHI in OE. OE is slowing down, though.
Thanks so much for the kind introduction, Brian! As you are one of the medical blogosphere's OG's, and a great voice here on Substack some twenty years later, I say the same about you!
I find OE and DoxGPT to be fairly reliable, as I think they are a bit more constrained in their responses drawing from a defined set of journals, guidelines, and validated medical resources rather than the whole mess out there. Nonetheless it is all pretty opaque. I read an OpEd in which the physician author believes doctors not consulting AI at the point of care will soon expose themselves to malpractice risk, as we cannot otherwise keep on top of the 1,000's of new bits of evidence published every day. But consensus, expert guidelines are still the bedrock of decision making.
I believe that there is a real place in the medical sphere for AI. Patients can't use it the same way they used Doctor Google (thank God). They will get lost in it without appropriate prompts. Having the physician work with the patient to get to the bottom of a difficult diagnosis is the key. And, yea, I see the errors but I don't see them as much when I give robust prompts.
Somehow I can’t see myself ever intentionally using a naked ChatGPT interface for much of anything, especially, however, not for medicine. I’ll have to look at the Claude version, as I’ve been using Claude for some other things successfully, but my go to LLM for medical work remains Open Evidence. Thanks for the summary!
The problem with AI, for me anyway, is that it takes twice as long to use it. First I read the AI answer, then I read other sources to verify. It's getting more accurate, but there are still errors. Might be because im a pediatrician. Peds is usually the last subject they tackle in primary care. I was a beta tester for Up To Date. It was crap in peds at that stage. But they've improved a lot and now I use it
For me the issue with using GPT for any issue is how much you need to spoon feed the context.
And then do more than 10 questions with answers on it, and it will forget the original data you gave it.
Not only are there errors, there are SIGNIFICANT errors in my experience.
My experience, also. However, with Open Evidence, in most cases, I have been able to recognize temporizing or equivocal responses to real clinical questions, in which case I do review the sources, vs the times where the response immediately reinforces my clinical gestalt.
Can any of these AI healthcare products ever enter the EMR space directly?
So I hadn't used Chat GPT for quite a while. I used OE and DoxGPT somewhat interchangeably for my medical work when I ran across ChatGPT Health. Trying to figure out WTH it was seemed like a Herculean effort. I kept getting info re a waitlist. Anyhow, I used it as a good reason to check out where and when one source was better than the other. I used the wholly independent CoPilot 😂😜 to find my answers.
This was the quick answer I got:
1) Many physicians still use ChatGPT for broad reasoning and explanation, then OpenEvidence to verify the evidence base.
2) DoxGPT is increasingly used for drug safety, quick lookups, and specialty‑specific prompts.
3) OpenEvidence and DoxGPT are emerging as complements, not replacements, for general LLMs.
The whole endeavor was useful in that I've started using ChatGPT again and am looking forward to what their healthcare version offers that is different. However, without EPIC, it will be hard to arrange a BAA that will allow it to be used with HIPAA compliance. Both DoxGPT and OE are compliant and you can store notes with PHI in OE. OE is slowing down, though.
Thanks so much for the kind introduction, Brian! As you are one of the medical blogosphere's OG's, and a great voice here on Substack some twenty years later, I say the same about you!
I find OE and DoxGPT to be fairly reliable, as I think they are a bit more constrained in their responses drawing from a defined set of journals, guidelines, and validated medical resources rather than the whole mess out there. Nonetheless it is all pretty opaque. I read an OpEd in which the physician author believes doctors not consulting AI at the point of care will soon expose themselves to malpractice risk, as we cannot otherwise keep on top of the 1,000's of new bits of evidence published every day. But consensus, expert guidelines are still the bedrock of decision making.
I am tempted to start a blog to accumulate examples of the most egregious medical AI errors. Just this week I saw a couple of real whoppers.
I believe that there is a real place in the medical sphere for AI. Patients can't use it the same way they used Doctor Google (thank God). They will get lost in it without appropriate prompts. Having the physician work with the patient to get to the bottom of a difficult diagnosis is the key. And, yea, I see the errors but I don't see them as much when I give robust prompts.