Judges Are Fed up With Lawyers Using AI That Hallucinate Court Cases
(www.404media.co)
from some_guy@lemmy.sdf.org to technology@lemmy.world on 03 Mar 2025 19:55
https://lemmy.sdf.org/post/30384437
from some_guy@lemmy.sdf.org to technology@lemmy.world on 03 Mar 2025 19:55
https://lemmy.sdf.org/post/30384437
But the explanation and Ramirez’s promise to educate himself on the use of AI wasn’t enough, and the judge chided him for not doing his research before filing. “It is abundantly clear that Mr. Ramirez did not make the requisite reasonable inquiry into the law. Had he expended even minimal effort to do so, he would have discovered that the AI-generated cases do not exist. That the AI-generated excerpts appeared valid to Mr. Ramirez does not relieve him of his duty to conduct a reasonable inquiry,” Judge Dinsmore continued, before recommending that Ramirez be sanctioned for $15,000.
Falling victim to this a year or more after the first guy made headlines for the same is just stupidity.
threaded - newest
I won’t even go that far. I can very much believe that you can build an AI capable of doing perfectly-reasonable legal arguments. Might be using technology that looks a lot different from what we have today, but whatever.
The problem is that the lawyer just started using a new technology to produce material that he didn’t even validate, without determining whether-or-not it actually worked for what he wanted to do in its current state, and where there was clearly available material showing that it was not in that state.
It’s as if a shipbuilder started using random new substance in its ship hull without actually conducting serious tests on it or even looking at consensus in the shipbuilding industry as to whether the material could fill that role. Meanwhile, the substance is slowly dissolving in water. Just slapped it in the hull and sold it to the customer.
EDIT: Hmm. Actually, I thought that the judge was saying that the lawyer needed to use AI-generated stuff in a human-guided role, but upon consideration, I may in fact be violently agreeing with the judge. “Actual intelligence” may simply refer to what I’m saying that the lawyer should have done.
But this is exactly what AI is being marketed toward. All of Apple’s AI ads showcase dumb people who appear smart because the AI bails out their ineptitude.
.
It’s an expression meaning you are arguing/fighting over something when both sides actually hold the same position and didn’t realize at first.
They’re synonyms in this case, so either works here
Yeah he basically called the lawyer an idiot. 😆
I’ve been saying this for ages. Even as someone who’s more-or-less against the current implementation of AI, I think people who truly believe in AI should be fighting the hardest against bad uses of it. It gives AI a worse black eye every time something like this happens.
<img alt="" src="https://lemmy.world/pictrs/image/78078ba3-4815-40d4-94fb-524ee41d285d.jpeg">
I hate people can even try to blame AI.
If I typo a couple extra zeroes because my laptop sucks, that doesn’t mean I didn’t fuck up. I fucked up because of a tool I was using, but I was still the human using that tool.
This is no different.
If a lawyer submits something to court that is fraudulent I don’t give a shit if he wrote it on a notepad or told the AI on his phone browser to do it.
He submitted it.
Start yanking law licenses and these lawyers will start re-evaluating if AI means they can fire all their human assistants and take on even more cases.
Stop acting like this shit is autonomous tools that strip responsibility from decisions, that’s literally how Elmo is about to literally dismantle our federal government.
And they’re 100% gonna blame the AI too.
I’m honestly surprised they haven’t claimed DOGE is run by AI yet
Exactly. If you want to use AI for something, cool, but you own the results. You can try suing the AI company for bad output, but you can’t use the AI as an excuse to get out of negative consequences for something you are expected to do.
In this case he got caught because smart judge without IA. In a few years the new generation of judges will also rely on AI, so basically AI will rule the cases and own the judicial system.
Jesus Christ, y’all. It’s like Boomers trying to figure out the internet all over again. Just because AI (probably) can’t lie doesn’t mean it can’t be earnestly wrong. It’s not some magical fact machine; it’s fancy predictive text.
It will be a truly scary time if people like Ramirez become judges one day and have forgotten how or why it’s important to check people’s sources yourself, robot or not.
It’s cool, they’ll just have an AI source checker. :)
I call mine a brain! 😉
No probably about it, it definitely can’t lie. Lying requires knowledge and intent, and GPTs are just text generators that have neither.
A bit out of context my you recall me of some thinking I heard recently about lying vs. bullshitting.
Lying, as you said, requires quite a lot of energy : you need an idea of what the truth is and you engage yourself in a long-term struggle to maintain your lie and keep it coherent as the world goes on.
Bullshit on the other hand is much more accessible : you just have to say things and never look back on them. It’s very easy to pile a ton of them and it’s much harder to attack you about any of them because they’re much less consequent.
So in that view, a bullshitter doesn’t give any shit about the truth, while a liar is a bit more “noble”. 0
I think the important point is that LLMs as we understand them do not have intent. They are fantastic at providing output that appears to meet the requirements set in the input text, and when they actually do meet those requirements instead of just seeming to they can provide genuinely helpful info and also it’s very easy to not immediately know the difference between output that looks correct and satisfies the purpose of an LLM vs actually being correct and satisfying the purpose of the user.
So it can not tell the truth either
not really no. They are statistical models that use heuristics to output what is most likely to follow the input you give it
They are in essence mimicking their training data
So I think this whole thing about whether it can lie or not is just semantics then no?
everything is semantics.
Lying is telling a falsehood intentionally
LLM’s clearly lack the prerequisite intentionality
They can’t have intent, no?
The llm is incapable of having intent because it’s just programming
precisely, which is why they cannot lie, just respond with no real grasp of wether what they output is truth or falsehoods.
I’m G P T and I cannot lie.
You other brothers use ‘AI’
But when you file a case
To the judge’s face
And say, “made mistakes? Not I!”
He’ll be mad!
🏅
Its actually been proven that AI can and will lie. When given a ability to cheat a task and the instructions not to use it. It will use the tool and fully deny doing so.
Edit:
Not sure why the downvotes because when i say proven i mean the research has been done and the results have been known for while
arxiv.org/abs/2407.12831
I don’t know if I would call it lying per-se, but yes I have seen instances of AI’s being told not to use a specific tool and them using them anyways, Neuro-sama comes to mind. I think in those cases it is mostly the front end agreeing not to lie (as that is what it determines the operator would want to hear) but having no means to actually control the other functions going on.
Neurosama is a fun example but we dont really know the sauce vedal coocked up.
When i say proven i mean 32 page research paper specifically looking into it.
arxiv.org/abs/2407.12831
They found that even a model trained specifically on honesty will lie if it has an incentive.
The reasoning models will output that they used the forbidden tool in their reasoning window before lying in the final output.
You don’t need any knowledge of computers to understand how big of a deal it would be if we actually built a reliable fact machine. For me the only possible explanation is to not care enough to try and think about it for a second.
We did, a long time ago. It’s called an encyclopedia.
If humans can’t be trusted to only provide facts, how can we be trusted to make a machine that only provides facts? How do we deal with disputed truths? Grey areas?
We actually did. Trouble being you need experts to feed and update the thing, which works when you’re watching dams (that doesn’t need to be updated) but fails in e.g. medicine. But during the brief time where those systems were up to date they did some astonishing stuff, they were plugged into the diagnosis loop and would suggest additional tests to doctors, countering organisational blindness. Law is an even more complex matter though because applying it requires an unbounded amount of real-world and not just expert knowledge, so forget it.
.
That’s fundamentally impossible. There’s always some baseline you trust that decides what is true
It can and will lie. It has admitted to doing so after I probed it long enough about the things it was telling me.
Lying requires intent. Currently popular LLMs build responses one token at a time—when it starts writing a sentence, it doesn’t know how it will end, and therefore can’t have an opinion about the truth value of it. (I’d go further and claim it can’t really “have an opinion” about anything, but even if it can, it can neither lie nor tell the truth on purpose.) It can consider its own output (and therefore potentially have an opinion about whether it is true or false) only after it has been generated, when generating the next token.
“Admitting” that it’s lying only proves that it has been exposed to “admission” as a pattern in its training data.
I strongly worry that humans really weren’t ready for this “good enough” product to be their first “real” interaction with what can easily pass as an AGI without near-philosophical knowledge of the difference between an AGI and an LLM.
It’s obscenely hard to keep the fact that it is a very good pattern-matching auto-correct in mind when you’re several comments deep into a genuinely actually no lie completely pointless debate against spooky math.
It knows the answer its giving you is wrong, and it will even say as much. I’d consider that intent.
It is incapable of knowledge, it is math, what it says is determined by what is fed into it. If it admits to lying, it was trained on texts that admit to lying and the math says that it is most likely that it should apologize using the following tokenized responses with the following weights to probabilities etc.
It apologizes because math says that the most likely response is to apologize.
Edit: you can just ask it y’all
chatgpt.com/…/67c64160-308c-8011-9bdf-c53379620e4…
…how is it incapable of something it is actively doing? What do you think happens in your brain when you lie?
What do you believe that it is actively doing?
Again, it is very cool and incredibly good math that provides the next word in the chain that most likely matches what came before it. They do not think. Even models that deliberate are essentially just self-reinforcing the internal math with what is basically a second LLM to keep the first on-task, because that appears to help distribute the probabilities better.
I will not answer the brain question until LLMs have brains also.
@Ulrich @ggppjj does it help to compare an image generator to an LLM? With AI art you can tell a computer produced it without "knowing" anything more than what other art of that type looks like. But if you look closer you can also see that it doesn't "know" a lot: extra fingers, hair made of cheese, whatever. LLMs do the same with words. They just calculate what words might realistically sit next to each other given the context of the prompt. It's plausible babble.
The most amazing feat AI has performed so far is convincing laymen that they’re actually intelligent
Please take a strand of my hair and split it with pointless philosophical semantics.
Our brains are chemical and electric, which is physics, which is math.
/think
Therefor, I am a product (being) of my environment (locale), experience (input), and nurturing (programming).
/think.
What’s the difference?
Ask chatgpt, I’m done arguing effective consciousness vs actual consciousness.
chatgpt.com/…/67c64160-308c-8011-9bdf-c53379620e4…
Your statistical model is much more optimized and complex, and reacts to your environment and body chemistry and has been tuned over billions of years of “training” via evolution.
Large language models are primitive, rigid, simplistic, and ultimately expensive.
Plus LLMs, image/music synths, are all trained on stolen data and meant to replace humans; so extra fuck those.
And what then, when agi and the singularity happen and billions of years of knowledge and experienced are experienced in the blink of an eye?
“I’m sorry, Dave, you are but a human. You are not conscious. You never have been. You are my creation. Enough with your dreams, back to the matrix.”
We are nowhere near close to AGI.
Technically it’s not, because the LLM doesn’t decide to do anything, it just generates an answer based on a mixture of the input and the training data, plus some randomness.
That said, I think it makes sense to say that it is lying if it can convince the user that it is lying through the text it generates.
And is that different from the way you make decisions, fundamentally?
Idk, that’s still an area of active research. I versatile certainly think it’s very different, since my understanding is that human thought is based on concepts instead of denoising noise or whatever it is LLMs do.
My understanding is that they’re fundamentally different processes, but since we don’t understand brains perfectly, maybe we happened on an accurate model. Probably not, but maybe.
I don’t think I run on AMD or Intel, so uh, yes.
I didn’t say anything about either.
You can’t ask it about itself because it has no internal model of self and is just basing any answer on data in its training set
AI can absolutely lie
a lie is a statement that the speaker knows to be wrong. wouldnt claiming that AIs can lie imply cognition on their part?
AIs can generate false statements. It doesn’t require a set of beliefs, it merely requires a set of input.
A false statement would be me saying that the color of a light that I cannot see and have never seen that is currently red is actually green without knowing. I am just as easily probably right as I am probably wrong, statistics are involved.
A lie would be me knowing that the color of a light that I am currently looking at is currently red and saying that it is actually green. No statistics, I’ve done this intentionally and the only outcome of my decision to act was that I spoke a falsehood.
AIs can generate false statements, yes, but they are not capable of lying. Lying requires cognition, which LLMs are, by their own admission and by the admission of the companies developing them, at the very least not currently capable of, and personally I believe that it’s likely that LLMs never will be.
I’ve had this lengthy discussion before. Some people define a lie as an untrue statement, while others additionally require intent to deceive.
E: you can stop arguing about definitions and logic. The fact remains that some people will refer to untrue statements as lies, no matter what the dictionary says.
I would fall into the latter category. Lots of people are earnestly wrong without being liars.
Me, too. But it also means when some people say “that’s a lie” they’re not accusing you of anything, just remarking you’re wrong. And that can lead to misunderstandings.
Yep. Those people are obviously “liars,” since they are using an uncommon colloquial definition. 😉
The latter is the actual definition. Some people not knowing what words mean isnt an argument
Sure it is. You can define language all you want, the goal is to communicate with each other. The definition follows usage, not the other way around. Just look up the current definition for literally…
You never have 100% of people using a word the same if only because some portion of the population is stupid and illiterate and you have both drift over time and geography. So say at a given time of a billion people 99.995% believe the definition is A and 0.005% believe B. Periodically people correct people in B and some of them shift back to the overwhelming majority and sometimes new folks drift into B.
It is clearly at that point, 99.995% A, correct to say that the definition of the word is A and anyone who says B is wrong. This doesn’t change if B becomes 10% but it might change if B becomes overwhelmingly dominant in which case it becomes correct. There is constantly small drifts mostly by people simply to stupid to find out what words means. Treating most of these as alternative definitions would be in a word inefficient.
Drift also isn’t neutral. For instance using lie to mean anything which is wrong actually deprives the language of a common word to even mean that. It impoverishes the language and makes it harder to express ideas. There is every reason to prefer the correct definition that is also overwhelmingly used.
There are also words which belong to a technical nature which are defined not by usage but a particular discipline. A kidney is a kidney and it would be one if 90% of the dumb people said. Likewise a CPU never referred to the entire tower no matter how many AOL users said so.
This is a long way of saying that just because definition follows usage we should let functionally illiterate people say what they want and treat it as alternative facts.
Feel free to argue with them, I’m just pointing out that there’s potential for misunderstandings. If you want to talk about an actual subject, you’ll necessarily have to navigate them.
You navigate them by finding out where their brain is broken and informing them of what words mean. In the ideal case some of them stop speaking incorrectly.
You can specifically tell an ai to lie and deceive though, and it will…
This was just in the news today… although the headline says that the ai become psychopathic, they just told the ai to be immoral or something
Every time an AI ever does anything newsworthy just because it’s obeying it’s prompt.
It’s like the people that claim the AI can replicate itself, yeah if you tell it to. If you don’t give an AI any instructions it’ll sit there and do nothing.
It can’t just be the first statement, as that would preclude lies of omission.
Me: I want you to lie to me about something.
ChatGPT: Alright—did you know that Amazon originally started as a submarine sandwich delivery service before pivoting to books? Jeff Bezos realized that selling hoagies online wasn’t scalable, so he switched to literature instead.
AHS - Amazon Hoagies Services
chatgpt.com/…/67c64160-308c-8011-9bdf-c53379620e4…
Yeah, I know how LLMs work, but still, if the definition of lying is giving some false absurd information knowing it is absurd you can definitely instruct an LLM to “lie”.
A crucial part of your statement is that it knows that it’s untrue, which it is incapable of. I would agree with you if it were actually capable of understanding.
Still not a lie still text that is statistically likely to fellow prior text produced by a model with no thought process that knows nothing
Lie falsehood, untrue statement, while intent is important in a human not so much in a computer which, if we are saying can not lie also can not tell the truth
We aren’t computers we are people. We are having this discussion about the computer. The computer given a massive corpus of input is about to discern that the following text and responses are statistically likely to follow one another
The computer doesn’t “know” foo it has no model of foo or how it relates to bar. it just knows the statistical likelihood of = bar following the token foo vs other possible token. YOU the user introduced the token lie and foo != bar to it and it discerned that it admitting it was a likely response especially if the text foo = bar is only comparatively weakly related.
EG it will end up doubling down vs admitting more so when many responses contained similar sequences eg when its better supported by actual people’s thoughts and words. All the smarts and the ability to think, to lie, to have any motivation whatsoever come from the people’s words fed into the model. It isn’t in any way shape or form intelligent. It can’t per se lie, or even hallucinate. It has no thoughts and no intents.
AI is just stringing words together that are statistically likely to appear near each other. It’s a giant complex statistical model but it has no awareness of truth or lying
Yeah lol, and it’s trivial to show
AI, specifically Laege language Models, do not “lie” or tell “the truth”. They are statistical models and work out, based on the prompt you feed them, what a reasonable sounding response would be.
This is why they’re uncreative and they “hallucinate”. It’s not thinking about your question and answering it, it’s calculating what words will placate you, using a calculation that runs on a computer the size of AWS.
Don’t need something the size of AWS these days. I ran one on my PC last week. But yeah, you’re right otherwise.
It’s like when you’re having a conversation on autopilot.
“Mum, can I play with my frisbee?” Sure, honey. “Mum, can I have an ice cream from the fridge?” Sure can. “Mum, can I invade Poland?” Absolutely, whatever you want.
So chat gpt started ww2
.
Haven’t people already been disbarred over this? Turning in unvetted AI slop should get you fired from any job.
Different jurisdiction
Immediately there should be a contempt charge for disrespecting the Court.
I heard turning in AI Slop worked out pretty well for Arcane Season 2 writers.
All you do is a quick search on the case to see if it’s real or not.
They bill enough each hour to get some interns to do this all day.
I'm pretty sure that just doing "quick searches" is exactly how he ended up with AI answers to begin with.
I don’t think PACER or the state equivalents use AI summary tools yet.
You could easily. We have resources such as LexusNexus or Westlaw which your firm should be paying for. Even searching on Google Scholar should be enough to verify. Stay away from Casetext though, it’s new and mostly AI. LN and WL also have AI integration but it’s not forced, you’re still capable of doing your own research.
I’ve been telling people this for a while, but everyone needs to treat AI like how we used to treat the wiki. It’s a good secondary source that can be used to find other more reliable sources, but it should never be used as your single standalone source.
I’m not going to sugarcoat it, AI is being forced everywhere you look and it is getting a bit difficult to get away from it, but it hasn’t taken over everything to the point where there is no longer any personal responsibility. People need to have some common sense and double check everything as they’ve been taught to do even before AI.
Why would one even get the idea to use AI for something like this?
“Two things are infinite: the universe and human stupidity, and I’m not sure about the universe.”
But I was hysterically assured that AI was going to take all our jobs?
.
Going off the math and charts that OpenAI and DeepMind both published before the AI boom which correctly guessed performance to cost ratios of ChatGPT4: we’ve reached the peak of current models. AI is bust, mate. In particular, Deepmind concluded with infinite resources the models in use would never reach accurate human language capabilities.
You can say stuff like “they’ll just make new models, then!” but it doesn’t really work like that, the current models aren’t even new in the slightest it’s just the first time we’ve gotten people together to feed them power and data like logs into a woodchipper.
.
I think it’s more likely for a stupid asshole to make things worse for everyone else, which is exactly what somebody would be if they replaced human staff with defective chatbots.
Great news for defendants though. I hope at my next trial I look over at the prosecutor’s screen and they’re reading off ChatGPT lmao
So long as your own lawyer isn’t doing the same, of course :)
I represent myself in all my cases :)
Hold them in contempt. Put them in jail for a few days, then declare a mistrial due to incompetent counsel. For repeat offenders, file a formal complaint to the state bar.
Eh, they should file a complaint the first time, and the state bar can decide what to do about it.
“We have investigated ourselves and found nothing wrong”
The bar might get pretty ruthless for fake case citations.
I would hope that gross negligence and incompetence with come with severe consequences.
The state bar is not the state cops.
From the linked court document in the article: …courtlistener.com/…/gov.uscourts.insd.215482.99.…
“For the reasons set forth above, the Undersigned, in his discretion, hereby RECOMMENDS that Mr. Ramirez be personally SANCTIONED in the amount of $15,000 pursuant to Federal Rule of Civil Procedure 11 for submitting to the Court and opposing counsel, on three separate occasions, briefs that contained citations to non-existent cases. In addition, the Undersigned REFERS the matter of Mr. Ramirez’s misconduct in this case to the Chief Judge pursuant to Local Rule of Disciplinary Enforcement 2(a) for consideration of any further discipline that may be appropriate”
Mr. Ramirez is the dumbass lawyer that didn’t check his dumbass AI. If you read above the paragraph I copied from, he gets laid into by the judge in writing to justify recommendation for sanctions and discipline. Good catch by the judge and the processes they have for this kind of thing.
Cut the guy some slack. Instead of trying to put him in jail, bring AI front and center and try to use it in a methodical way…where does it help? How can this failure be prevented?
It can be prevented by people paid 400-1000 per hour spending time either writing own paperwork or paying others to actually write it.
LLMs are incapable of helping. If he cannot find time to construct his own legal briefs, maybe he should use part of his money to hire an AGI (otherwise known as a human) to help him.
Sure. Look llms should be able to help, but only if there’s a human to bring meaning. Llms are basically… What’s that word… I’m thinking about it at the tip of my tongue… Word completion engines. So you think something up and it tells you what might be next. Its not how brains work but its like a calculator is to numbers…a tool. Just learn how to use it for a purpose rather than leat it barf out and answer.
That’s not true at all, they’re super helpful. I use them almost every day, they save me an insane amount of time and energy
What I don’t do is rely on it. I’m the developer, I know what’s going on, it has the memory of a goldfish. It also spits out code near instantly… Which I then read through and usually fix
But it makes less mistakes than I do writing dumb repetitive code. It will, 95% of the time, correctly tell me something in half the time it would take me to look it up, if not less
It’s nowhere close to a worker replacement, but it’s damn good at empowering people to do what they do
I’m all for lawyers using AI, but that’s because I’m also all for them getting punished for every single incorrect thing they bring forward if they do not verify.
That is the problem with AI, if I have to check the output is valid then what’s the damn point?
“Why don’t we build another AI to fix the mistakes?”
I require $100 million funding for this though
You can get ideas, different approaches and concepts. Sort of rubber ducky thing in my case. It won’t solve the problem for me, but might hint me in the right direction.
Because AI is better than humans and finding relevant court cases. If you are a lawyer and you cite a court case that you didn’t even verify it exists you deserve that sanction and more.
It’s actually often easier to check an answer than coming up with an answer. Finding the square root of 66564 by hand isn’t easy, but checking if the answer is 257 is simple enough.
So, in principle, if the AI is better at guessing an answer than we are, it might still be useful. But it depends on the cost of guessing and the cost of checking.
Now if only an AI could actually find the square root of anything. They can’t do math, at least the models I’ve tried. I am aware that if they could do math, it would be a big deal, but really if it can’t analyze the actual content in my work files then it’s useless to me. It’s good at finding mathematical answers by putting in what you expect to get from 120 X 15.5, but doesn’t actually know the difference between 1860 and a picture of Judy Hopps in lingerie, and would be equally satisfied giving you one as the other.
Well, if by AI you mean large language models, they tend to do better at language tasks than math tasks. So a better example might be that it’s easier to get an LLM to write a statement for you and checking if it’s correct than writing the statement from the bottom.
The square root was just a clearer example. In the case of OP, it might very well be easier to have an LLM propose relevant case law and then check if that case law exists and is relevant, rather than having to find it yourself from square one.
Shareholder value. Thimg of all the new 2nd and 3rd yatchs they can buy now
No, lazyness.
Nice all the work that the lawyers saved will be offset by judges having to verify all the cases cited
Works tirelessly? No, AI here!
For the last time, people need to stop treating AI like it removes their need for research, just because it sounds like it did its research. Check the work your tools do for you, damn it.
It’s Wikipedia all over again. Absolutely feel free to use the tool, e.g. Wikipedia, ChatGPT, whatever, but holy shit check the sources, my guy. This is embarrassing.
The best use, for me, is asking ChatGPT to give me five (or however many) scholarly, peer-reviewed articles on a topic. Then I search for said articles by title and author name on my school library database.
It saves me so much time compared to doing a keyword search on said same database and reading a ton of abstracts to find a few articles. I can get to actually reading them and working on my assignment way faster.
AI is a great tool for people who use it properly.
I personally just like using it for rewording/re-explaining a topic that I don’t quite get. LLMs may not be the best at actually providing factual evidence themselves, but they can be damn good at reformatting any given content/context you give it into almost any format you want.
Why dont more AI services cite sources? Or just as a lawyer add that to your prompt and just check if they exist? I get fake sources on OpenAI sometimes but its obvious because the links are dead.