ChatGPT plays doctor with 72% success
(www.axios.com)
from stopthatgirl7@kbin.social to technology@lemmy.world on 10 Sep 2023 10:42
https://kbin.social/m/technology@lemmy.world/t/441871
from stopthatgirl7@kbin.social to technology@lemmy.world on 10 Sep 2023 10:42
https://kbin.social/m/technology@lemmy.world/t/441871
As AI capabilities advance in complex medical scenarios that doctors face on a daily basis, the technology remains controversial in medical communities.
threaded - newest
Sucks if your one of the 10-20% who don’t get proper treatment (maybe die?) because some doctor doesn’t have time to double check. But hey … efficiency!
Ya that’s a fundamental misunderstanding of percentages. For an analogous situation with which we’re all more intuitively familiar, a self driving car that is 99.9% accurate in detecting obstacles crashes into one in one thousand people and/or things. That sucks.
Also, most importantly, LLMs are incapable of collaboration, something very important in any complex human endeavor but difficult to measures, and therefore undervalued by our inane, metrics-driven business culture. Chatgpt won’t develop meaningful, mutually beneficial relationships with its colleagues, who can ask each other for their thoughts when they don’t understand something. It’ll just spout bullshit when it’s wrong, not because it doesn’t know, but because it has no concept of knowing at all.
It really needs to be pinned to the top of every single discussion around chatgbt:
It does not give answers because it knows. It gives answers because it thinks it looks right.
Remember back in school when you didn’t study for a test and went through picking answers that “looked right” because you vaguely remember hearing the words in Answer B during class at some point?
It will never have wisdom and intuition from experience, and that’s critically important for doctors.
How do you think medicine works?
Tests are expensive, invasive, and often outright destructive. Medicine is very much about treating symptoms and trying to determine what diagnosis “looks right”. If we had perfect knowledge then this would actually be trivial for AI because it would literally just be looking up symptoms in a textbook.
I was going to make a joke about not watching so much House but… actually watch more House. Ignore all the medical malpractice, complete lack of ethics, etc. Instead, focus on just how often they get down to two or three probable diagnoses based on the known data (… and ignore that it is always the obscure one). And then they get a new symptom and a new set of probable diagnoses.
“Looks right” in a human context means the one that matches a person’s actual experience and intuition. “Looks right” in an LLM context means the series of words have been seen together often in the training data (as I understand it, anyway - I am not an expert).
Doctors are most certainly not choosing treatment based on what words they’ve seen together.
What is experience and intuition other than past data and knowledge of likely outcomes?
if my doctor is “using their gut” to decide what treatment I need and not basing it on data? I am gonna ask for another doctor. Also, if/when I die, they are completely screwed if there is any form of an investigation.
They literally (in the truest sense of the word) are. Words have meaning and are how we convey information related to collected data. Collected data is compared against past data and experience and slotted in to likely diagnoses. This is likely an incredibly bad example but: Stomach ache, fever, constipation, and nose bleeds? You might have stomach cancer. Let’s do a biopsy.
Doctors read charts. That is how they know what collected data there is. There is a LOT of value in the doctor also speaking to the patient, but that more speaks to patients not communicating valuable information because “they are just a nurse” or whatever other stupidity.
But, at the end of the day: it is data aggregation followed by pattern matching. Which… AI are actually incredibly good at. It is just the data collection that is a problem and that is a problem in medicine as a whole. Just ask any woman or person of color who was ignored by their doctor until they got a white dude to talk for them.
This is very much a situation where a bit of a philosophy background helps a lot. Because the idea of “what is consciousness” and “what is thought” long predate even the idea of artificial intelligence. And as you start breaking down cognitive functions and considering what they actually mean…
Just to be clear, I am not at all saying LLMs will take over for doctors. But as a preliminary diagnosis and a tool used throughout patient care? We have a ways to go, but this will be incredibly valuable. Likely in the same way that good doctors learn to listen to and even consult the nurses who spend the most time with those patients.
Which is what we already see in computer programming and (allegedly) other fields like legal writing and the like. You still need a trained professional to check the output. But reading through a dozen unit tests and utility functions or some boilerplate text is a lot faster than writing it. And then they can spend the majority of their “brain juice” on the more interesting/difficult problems.
Same here. Analyzing blood tests after a physical or the vast majority of stuff at an urgent care? Let the AI do the first pass and the doctors can check that that makes sense. Which lets them focus on the emergencies and the “chronic pain” situations.
Or one of the ninty nine percent of people who don’t give the AI their symptoms in medical terminology.
Medicine is going to take awhile for anything except small-scope tools to handle one specific thing, due to the massive variation in the presentation of different problems that doctors can face.
What won’t take as long, because there isn’t the same inherent variation in presentation, is law.
About 70 percentage points better than my doctor. Nice!
Yeah, medicine is one of the areas where I really feel like AI could make serious strides. Most people don't have a doctor they see regularly anyway so any input would be welcome. Anecdotally I've known several people who were misdiagnosed or just had doctors not believe them.
Of course I'd want to be able to escalate and have different treatment options but I could probably be ok with AI-assisted medicine.
When you read some bs on internet but thankfully its US only
First in medical spending. 40th or 50th in positive medical outcomes.
Among Western nations.
Those odds are shit. Meanwhile most of us have zero doctor.
A few things, that’s an abysmal rate when it comes to people’s health. A doctor with that success rate would be sued into next century. The rate dropped further when it came to differential diagnosis, implying chat gpt was leaving out important rarer possibilities. Often doctors work by starting with the most common and narrow down from there after repeated rounds of testing if it ends up being something uncommon, but one of their primary jobs is also thinking about rarer dangerous stuff that can mimic more common things and must be ruled out immediately.
Most importantly, the information fed into this was optimized with accurate descriptive medical terminology. This is a language that, in general, patients do not speak. People can also describe things very differently, for instance a patient saying something is weak when a doctor may say no that’s numb not weak or visa versa. And dizzy could mean just about anything. Someone typing their own story directly into chat gpt is going to get much worse results than this without someone to interpret the word choices and ask the right questions that people may not even realize are important.
Anyways, the possibilities of AI use in Healthcare is interesting, but disappointing it does worse the less common things get and is bad at a differential diagnosis, the areas that would be really helpful as an aid to diagnosis. Some other areas to think about though could be maybe as a front end to find clinical trials with the us gov database, which can be hard to browse, or maybe streamlining the endless insurance paperwork. I’d be surprised insurance companies don’t use something similar already.
Don’t forget the inherent biases that are introduced with AI training! Women especially have a history of having their symptoms dismissed out of hand - if the LLM training data includes these biases, in combination with the bad diagnosis women could be really screwed.
similarly to people from different races/countries … it’s not only that their conditions might vary and require more data, it is also that some communities don’t visit/trust hospitals to even have their data collected to be in the training set. Or they can’t afford to visit.
Sometimes, people from more vulnerable communities (eg LGBT) might prefer not to have such data collected in the first place, making data sparser.
Anecdotally my doctor is prolly only right 75% of the time too. But I’ll go back to him later with more information about the ailment (or more tests) and he’ll eventually get it right. Medical diagnoses in general are not terribly accurate.
I don’t think your comment really disputes anything the previous poster said.
But maybe you read the first lines and skimmed the rest (I certainly do that more often than I would like).
Every decade, new diagnoses and discoveries are made. Many manmade or climate change related. We never had microplastic poison before. Or random chemicals added into our foods because it gives a company that +1% profit.
In other words, we are finding new ways to destroy our bodies!
And with AI is always working with historical data, it’ll be a long time before we only use a AI Doctor.
IME my doctor would not outperform ChatGPT
So it’s about as good as people going to WebMD and diagnosing themselves? hoo boy
That’s a whole 22% better than a coin toss!!!
Having made many models that were only slightly better than a coin toss, that’s really not bad. Especially since that’s not even their primary design goal.
Not really comparable since a medical diagnosis has hundreds of possible results
I mean if the AI takes women seriously then that's honestly already better than most of the doctors I've had
I was about to say…
Wonder what the success rate of doctors is. I’d be surprised if it is above 70% lol
Unfortunately, if the data is biased, the model is biased.
Yes, that’s something that’s constantly emphasized in scientific research. You might have the most infallible algorithm, but… garbage in, garbage out. You’ll still get garbage data if what you enter into the algorithm is garbage
Or if it’s better at diagnosing minorities, too.
It almost certainly won’t, but it’s nice to hope.
It might remove the face to face human bias of a GP but it doesn’t make up for the decades of preconceived or absent research about women or minorities.
Sadly, that percentage is probably better than a good number of doctors.
What was the result in the human’s group?
I think the result is good if you consider that ChatGPT wasn’t made to be a doctor.
I don’t use ChatGPT and ain’t planning to, but someone should try asking it something like…
“How often should a male change their tampon?”
See what, if any nonsense it regurgitates.
Okay then, well go figure. I was guessing it would puke up some nonsense, but apparently not.
You know it’s free to use right? You can play around with it yourself and see what it can do
Been there, tried that. Unless something done changed, they said I had 10 free trial uses, then I’d have to pay.
You sure you’re using the real thing at chat.openai.com? What you’re describing does not sound right
I’m saving this comment for later, I’m not exactly sure which site I was using, but the engine was supposedly ChatGPT.
You might be right. Thank you for the link. 👍
Men that menstruate exist.
Now I’m no expert, but if you’re bleeding from your penis or your anus, you should probably go see a doctor about that.
It shows.
I used the biological word ‘male’, not the opinionated word ‘man’. IDGAF what you identitfy as.
This isn’t up for debate, either you have XX or XY chromosomes. You were either born with a dick or you weren’t.
So unless you’re a hermaphrodite, you should probably get yourself checked if you are somehow bleeding from your penis.
Or not, I don’t care if your dick falls off.
There are so many biological exceptions to this that it’s not even funny. For example, it is very possible to have XY chromosomes yet not be born with a dick. Look up Androgen Insensitivity Syndrome for a start.
There are no absolutes when talking about sex and gender. None.
What? No documentation to back this up?
I mean seriously, are there any documented cases of anyone born with a penis yet also having monthly periods?
I can see already you’re not arguing in good faith, you’re just trying to raise a stink. Have a mediocre day.
Details matter here. Here's a few from the study: