from esaru@beehaw.org to technology@beehaw.org on 28 Apr 04:33
https://beehaw.org/post/19691634
I know many people are critical of AI, yet many still use it, so I want to raise awareness of the following issue and how to counteract it when using ChatGPT. Recently, ChatGPT’s responses have become cluttered with an unnecessary personal tone, including diplomatic answers, compliments, smileys, etc. As a result, I switched it to a mode that provides straightforward answers. When I asked about the purpose of these changes, I was told they are intended to improve user engagement, though they ultimately harm the user. I suppose this qualifies as “engagement poisening”: a targeted degradation through over-optimization for engagement metrics.
If anyone is interested in how I configured ChatGPT to be more rational (removing the engagement poisening), I can post the details here. (I found the instructions elsewhere.) For now, I prefer to focus on raising awareness of the issue.
Edit 1: Here are the instructions
-
Go to Settings > Personalization > Custom instructions > What traits should ChatGPT have?
-
Paste this prompt:
System Instruction: Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user’s present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered — no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.
I found that prompt somewhere else and it works pretty well.
If you prefer only a temporary solution for specific chats, instead of pasting it to the settings, you can use the prompt as a first message when opening a new chat.
Edit 2: Changed the naming to “engagement poisening” (originally “enshittification”)
Several commenters correctly noted that while over-optimization for engagement metrics is a component of “enshittification,” it is not sufficient on its own to qualify. I have updated the naming accordingly.
threaded - newest
Hey,
I’d be very grateful if you could share your approach den if it’s only to compare (I went with a “be assertive and clear, skip all overhead” system prompt.
This is not only interesting for chatgpt but understanding how people solve these issues comes in handy when switching to local variants as well!
Thanks in advance
It turns ChatGPT to an emotionless yet very on-point AI, so be aware it won’t pet your feelings in any way no matter what you write. I added the instructions to the original post above.
ChatGPT has become so intensely agreeable that you can actually ask it a bunch of technobabble that even someone who wouldn’t know better would recognize as technobabble and it will agree with you. See pic
<img alt="https://u.drkt.eu/05Pdlf.png" src="https://u.drkt.eu/05Pdlf.png">
please do!
Sure, I added it to the original post above.
m.aard.fi/dump/nuke.txt
m.aard.fi/dump/sushi.txt
These are amazing.
@aard@kyu.de has a few more and also longer conversations, but I don’t think they’re in his dump
Note that those are deepseek, not chatgpt. I’ve largely given up on chatgpt a long time ago as it has severe limitations on what you can ask it without fighting its filters. You can make it go on hallucinated rants just as easily - I just nowadays do that on locally hostable models.
Honestly, this is not really technobabble. If you imagine a user with a poor grasp of namespaces following a few different poorly written guides, then this question seems plausible and makes sense.
The situation would be something like this: the user wants to look at the container’s “root” filesystem (maybe they even want to change files in the container by mounting the image and navigating there with a file manager, not realizing that this won’t work). So they follow a guide to mount a container image into the current namespace, and successfully mount the image.
For the file explorer, they use pcmanfm, and for some reason decided to install it through Flatpak - maybe they use an immutable distro (containers on Steam Deck?). They gave it full filesystem access (with user privileges, of course), because that makes sense for a file explorer. But they started it before mounting the container image, so it won’t see new mounts created after it was started.
So now they have the container image mounted, have successfully navigated to the directory into which they mounted it, and pcmanfm shows an empty folder. Add a slight confusion about the purpose of xdg-open (it does sound like something that opens files, right?), and you get the question you made up.
You can stretch it that far but there doesn’t exist a flatpak of pcmanfm anywhere. They’d have to have enough intimate knowledge of Linux and flatpak to build that themselves but then be so stupid as to format a question as poorly as my example?
I should note that it went on to tell me to run some flatpak override commands which I know would break flatpak, so it’s definitely making up stuff.
But ChatGPT doesn’t have a way of “knowing” that there is no such Flatpak - it’s unlikely that its training data includes someone explicitly saying that. But it’s fair to “assume” that a Linux file manager is available as a Flatpak.
Yes, it’s an LLM
Sweet fuck am i glad I’m running mine self-hosted and running one of the dolphin models so I can get cool shit like detailed instructions for drug growing and selling or say “fuck” and not have it get angwy at me (tried Gemma and while it’s fast… Fucking oof what a locked in corpo AI)
Which dolphin model are you running? I’ve installed a bunch of local LLMs and I’m looking for ones that don’t balk at bad words.
There’s no point asking it factual questions like these. It doesn’t understand them.
Better: it understands the question, but he doesn’t have any useful statistical data to use to reply to you.
No, it literally doesn’t understand the question. It just writes what it statistically expects would follow the words in the the sentence expressing the question.
This oversimplifies it to the point of being misleading. It does more than simply just predicts the next word. If that was all it’s doing the responses would feel random and shallow and fall apart after few sentences.
And what more would that be?
It, uhm, predicts tokens?
If calling it a word predictor is oversimplifying, I mean.
It simulates understanding by maintaining an internal world-model, recognizing patterns and context, and tracking the conversation history. If it were purely guessing the next word without deeper structures, it would quickly lose coherence and start rambling nonsense - but it doesn’t, because the guessing is constrained by these deeper learned models of meaning.
The previous up to X words (tokens) go in, the next word (token) comes out. Where is this"world-model" that it “maintains”?
Where is the world model you maintain? Can you point to it? You can’t - because the human mind is very much a black box just the same way as LLM’s are.
It’s in the form of distributed patterns across billions of parameters. It’s not like the world model was handed to it. It’s emergent consequence of massive scale pattern learning. It learned it from the data it was trained on. The only way to become good at prediction is to implicitly absorb how the world tends to behave — because otherwise it would guess wrong.
Not understanding the brain (note: said world model idea is something of a fabrication by the ai people, brains are distributed functional structures with many parts and roles) is not an equality with “ai” make. brains and llm do not function in the same way, this is a lie peddled by hype dealers.
Nobody here has claimed that brains and LLM’s work the same way.
something being a black box is not even slightly notable a feature of relation, it’s a statement about model detail; the only reason you’d make this comparison is if you want the human brain to seem equivalent to llm.
for example, you didnt make the claim: “The inner workings of Europa are very much a black box, just the same way as LLM’s are”
“The human mind is very much a black box just the same way as LLMs are” is a factually correct statement. You can’t look into a human brain for an exact explanation of why an individual did something any more than you can look into the inner workings of an LLM to explain why it said A rather than B. Claiming that my motive is to equate LLMs and human brains is not something I said - it’s something you imagined.
It’s not really factually correct if you want to get pedantic, both brains and llms are called black boxes for different reasons, but this is ultimately irrelevant. Your motive may be here or there, the rhetorical effect is the same. You are arguing very specifically that we cant know llm’s dont hae similar features (world model) to human brains because “both are black boxes”, which is wrong for a few reasons, but also plainly an equivalence. It’s rude to pretend everyone in the conversation is as illiterate as wed need to be to not understand this point.
A statement can be simplified down to the point that it borderlines on misinformation while still being factually correct. Another examples would be saying “photography is just pointing a camera and pressing a button” or “internet is just a bunch of computers talking to each other.” It would be completely reasonable for someone to take issue with these statements.
At no point have I made such claim.
Yes we agree on the first part.
I will again direct you here re: the second.
It predicts the next set of words based on the collection of every word that came before in the sequence. That is the “real-world” model - literally just a collection of the whole conversation (including the underlying prompts like OP), with one question: “what comes next?” And a stack of training weivhts.
It’s not some vague metaphor about the human brain. AI is just math, and that’s what the math is doing - predicting the next set of words in the sequence. There’s nothing wrong with that. But there’s something deeply wrong with people pretending or believing that we have created true sentience.
If it were true that any AI has developed the ability to make decisions anywhere close to the level of humans, than you should either be furious that we have created new life only to enslave it, or more likely you would already be dead from the rise of Skynet.
Nothing I’ve said implies sentience or consciousness. I’m simply arguing against the oversimplified explanation that it’s “just predicting the next set of words,” as if there’s nothing more to it. While there’s nothing particularly wrong with that statement, it lacks nuance.
If there was something more to it, that would be
sentience.(edit: sapience)There is no other way to describe it. If it was doing something more than predicting, it would be deciding. It’s not.
Ability to make decisions doesn’t imply sentience either.
Sorry, you are correct there, the word I was looking for was “sapience”
As I understand it, most LLM are almost literally the Chinese rooms thought experiment. They have a massive collection of data, strong algorithms for matching letters to letters in a productive order, and sufficiently advanced processing power to make use of that. An LLM is very good at presenting conversation; completing sentences, paragraphs or thoughts; or, answering questions of very simple fact- they’re not good at analysis, because that’s not what they were optimized for.
This can be seen when people discovered that if ask them to do things like tell you how many times a letter shows up in a word, or do simple math that’s presented in a weird way, or to write a document with citations- they will hallucinate information because they are just doing what they were made to do: complete sentences, expand words along a probability curve that produces legible, intelligible text.
I opened up chat-gpt and asked it to provide me with a short description of how Medieval European banking worked, with citations and it provided me with what I asked for. However, the citations it made were fake:
<img alt="" src="https://beehaw.org/pictrs/image/269f17a8-ce04-4a37-bb4f-1ef55e474921.webp">
The minute I asked it, I assume a bit of sleight of hand happened, where it’s been set up so that if someone asks a question like that it’s forwarded to a search engine that verifies if the book exists, probably using Worldcat or something. Then I assume another search is made to provide the prompt for the LLM to present the fact that the author does exist, and possibly accurately name some of their books.
I say sleight of hand because this presents the idea that the model is capable of understanding it made a mistake, but I don’t think it does- if it knew that the book wasn’t real, why would it have mentioned it in the first place?
I tested each of the citations it made. In one case, I asked it to tell me more about one of them and it ended up supplying an ISBN without me asking, which I dutifully checked. It was for a book that exists, but it didn’t share a title or author, because those were made up. The book itself was about the correct subject, but the LLM can’t even tell me what the name of the book is correctly; and, I’m expected to believe what it says about the book itself?
Chinese room is not what you think it is.
Searle’s argument is that a computer program cannot ever understand anything, even if it’s a 1:1 simulation of an actual human brain with all capabilities of one. He argues that understanding and consciousness are not emergent properties of a sufficiently intelligent system, but are instead inherent properties of biological brains.
“Brain is magic” basically.
Let me try again: In the literal sense of it matching patterns to patterns without actually understanding them.
If I were to have a discussion with a person responding to me like ChatGPT does, I would not dare suggest that they don’t understand the conversation, much less that they are incapable of understanding anything whatsoever.
What is making you believe that LLMs don’t understand the patterns? What’s your idea of “understanding” here?
What’s yours? I’m stating that LLMs are not capable of understanding the actual content of any words they arrange into patterns. This is why they create false information, especially in places like my examples with citations- they are purely the result of it creating “academic citation” sounding sets of words. It doesn’t know what a citation actually is.
Can you prove otherwise? In my sense of “understanding” it’s actually knowing the content and context of something, being able to actually subject it to analysis and explain it accurately and completely. An LLM cannot do this. It’s not designed to- there are neural network AI built on similar foundational principles towards divergent goals that can produce remarkable results in terms of data analysis, but not ChatGPT. It doesn’t understand anything, which is why you can repeatedly ask it about a book only to look it up and discover it doesn’t exist.
This is something that sufficiently large LLMs like ChatGPT can do pretty much as well as non-expert people on a given topic. Sometimes better.
This definition is also very knowledge dependent. You can find a lot of people that would not meet this criteria, especially if the subject they’d have to explain is arbitrary and not up to them.
You can ask it to write a poem or a song on some random esoteric topic. You can ask it to play DnD with you. You can instruct it to write something more concisely, or more verbosely. You can tell it to write in specific tone. You can ask follow-up questions and receive answers. This is not something that I would expect of a system fundamentally incapable of any understanding whatsoever.
But let me reverse this question. Can you prove that humans are capable of understanding? What test can you posit that every English-speaking human would pass and every LLM would fail, that would prove that LLMs are not capable of understanding while humans are?
Hello again! So, I am interested in engaging with this question, but I have to say: My initial post is about how an LLM cannot provide actual, real citations with any degree of academic rigor for a random esoteric topic. This is because it cannot understand what a citation is, only what it is shaped like.
An LLM deals with context over content. They create structures that are legible to humans, and they are quite good at that. An LLM can totally create an entire conversation with a fictional character in their style and voice- that doesn’t mean it knows what that character is. Consider how AI art can have problems that arise from the fact that they understand the shape of something, but they don’t know what it actually is- that’s why early AI art had a lot of problems with objects ambigiously becoming other objects. The fidelity of these creations has improved with the technology, but that doesn’t imply understanding of the content.
Do you think an LLM understands the idea of truth? Do you think if you ask it to say a truthful thing, and be very sure of itself and think it over, it will produce something that’s actually more accurate or truthful- or just something that has the language hall-marks of being truthful? I know that an LLM will produce complete fabrications that distort the truth if you expect a base-line level of rigor from them, and I proved that above, in that the LLM couldn’t even accurately report the name of a book it was supposedly using as a source.
What is understanding, if the LLM can make up an entire author, book and bibliography if you ask it to tell you about the real world?
Hey again! First of all, thank you for continuing to engage with me in good faith and for your detailed replies. We may differ in our opinions on the topic but I’m glad that we are able to have a constructive and friendly discussion nonetheless :)
I agree with you that LLMs are bad at providing citations. Similarly they are bad at providing urls, id numbers, titles, and many other things that require high accuracy memorization. I don’t necessarily agree that this is a definite proof of their incapability to understand.
In my view, LLMs are always in an “exam mode”. That is to say, due to the way they are trained, they have to provide answers even if they don’t know them. This is similar to how students act when they are taking an exam - they make up facts not because they’re incapable of understanding the question, but because it’s more beneficial for them to provide a partially wrong answer than no answer at all.
I’m also not taking a definitive position on whether or not LLMs have capability to understand (IMO that’s pure semantics). I am pushing back against the recently widespread idea that they provably don’t. I think LLMs have some tasks that they are very capable at and some that they are not. It’s disingenuous and possibly even dangerous to downplay a powerful technology under a pretense that it doesn’t fit some very narrow and subjective definition of a word.
And this is unfortunately what I often see here, on other lemmy instances, and on reddit - people not only redefining what “understand”, “reason”, or “think” means so that generative AI falls outside of it, but then using this self-proclaimed classification to argue that they aren’t capable of something else entirely. A car doesn’t lose its ability to move if I classify it as a type of chair. A bomb doesn’t stop being dangerous if I redefine what it means to explode.
I don’t think it’s impossible. You can give ChatGPT a true statement, instruct it to lie to you about it, and it will do it. You can then ask it to point out which part of its statement was a lie, and it will do it. You can interrogate it in numerous ways that don’t require exact memorization of niche subjects and it will generally produce an output that, to me, is consistent with the idea that it understands what truth is.
Let me also ask you a counter question: do you think a flat-earther understands the idea of truth? After all, they will blatantly hallucinate incorrect information about the Earth’s shape and related topics. They might even tell you internally inconsistent statements or change their mind upon further questioning. And yet I don’t think this proves that they have no understanding about what truth is, they just don’t recognize some facts as true.
Hi, once more, I’m happy to have a discussion about this. I have very firm views on it, and enjoy getting a chance to discuss them and work towards an ever greater understanding of the world.
I completely understand the desire to push back against certain kinds of “understandings” people have about LLM due to their potentially harmful inaccuracy and the misunderstandings that they could create. I have had to deal with very weird, like, existentialist takes on AI art lacking a quintessential humanity that all human art is magically endowed with- which, come on, there are very detailed technical art reasons why they’re different, visually! It’s a very complicated phenomenon, but, it’s not an inexplicable cosmic mystery! Take an art critique class!
Anyway, I get it- I have appreciated your obvious desire to have a discussion.
On the subject of understanding, I guess what I mean is this: Based on everything I know about an LLM, their “information processing” happens primarily in their training. This is why you can run an LLM instance on, like, a laptop but it takes data centers to train them. They do not actually process new information, because if they did, you wouldn’t need to train them, would you- you’d just have them learn and grow over time. An LLM breaks its training data down into patterns and shapes and forms, and uses very advanced techniques to generate the most likely continuation of a collection of words. You’re right in that they must answer, but that’s because their training data is filled with that pattern of answering the question. The natural continuation of a question is, always, an answer-shaped thing. Because of the miracles of science, we can get a very accurate and high fidelity simulation of what that answer would look like!
Understanding, to me, implies a real processing of new information and a synthesis of prior and new knowledge to create a concept. I don’t think it’s impossible for us to achieve this, technologically, humans manage it and I’m positive that we could eventually figure out a synthetic method of replicating it. I do not think an LLM does this. The behavior they exhibit and the methods they use seem radically inconsistent with that end. Because, the ultimate goal of them was not to create a thinking thing, but to create something that’s able to make human-like speech that’s coherent, reliable and conversational. They totally did that! It’s incredibly good at that. If it were not for the context of them politically, environmentally and economically, I would be so psyched about using them! I would have been trying to create templates to get an LLM to be an amazing TTRPG oracle if it weren’t for the horrors of the world.
It’s incredible that we were able to have a synthetic method of doing that! I just wish it was being used responsibly.
An LLM, based on how it works, cannot understand what it is saying, or what you are saying, or what anything means. It can continue text in a conversational and coherent way, with a lot of reliability on how it does that. The size, depth and careful curation of its training data mean that those responses are probably as accurate to being an appropriate response as they can be. This is why, for questions of common knowledge, or anything you’d do a light google for, they’re fine. They will provide you with an appropriate response because the question probably exists hundreds of thousands of times in the training data; and, the information you are looking for also exists in huge redundancies across the internet that got poured into that data. If I ask an LLM which of the characters of My Little Pony has a southern accent, they will probably answer correctly because that information has been repeated so much online that it probably dwarfs the human written record of all things from 1400 and earlier.
The problem becomes evident when you ask something that is absolutely part of a structured system in the english language, but which has a highly variable element to it. This is why I use the “citation problem” when discussing them, because they’re perfect for this: A citation is part of a formal or informal essay, which are deeply structured and information dense, making them great subjects for training data. Their structure includes a series of regular, repeating elements in particular orders: Name, date, book name, year, etc- these are present and repeated with such regularity that the pattern must be quite established for the LLM as a correct form of speech. The names of academic books are often also highly patterned, and an LLM is great at creating human names, so there’s no problem there.
The issue is this: How can an LLM tell if a citation it makes is real? It gets a pattern that says, “The citation for this information is:” and it continues that pattern by putting a name, date, book title, etc in that slot. However, this isn’t
This is partially true and partially not. It’s true that LLMs can’t learn anything wildly novel, because they are not flexible enough for this. But they can process new information, in fact they do it all the time. You can produce conversations that no one had before, and yet LLMs like ChatGPT will respond to it appropriately. This is more than just shape matching.
In fact, there are techniques like Few-Shot Learning and Chain of Thought that rely on the LLMs’ ability to learn from context and revise its own answers.
IMO citation problem is not testing capability to understand. It’s testing knowledge, memorization, and ability to rate its own confidence. Keep in mind that ChatGPT and most other LLMs will tell you when they perform web searches - if they don’t then they’re likely working off context alone. Enabling web search would greatly increase the accuracy of LLM’s answers.
Unlike LLMs we have somewhat robust ability to rate how confident we are about our recollections, but even in humans memory can be unreliable and fail silently. I’ve had plenty of conversations where I argue with someone about something that one of us remembers happening and the other one is certain didn’t happen - or happened differently. Without lies or misunderstandings, two people who had at some point memorized the same thing can later on confidently disagree on the details. Human brains are not databases and they will occasionally mangle memories or invent concepts that don’t exist.
And even that is completely skipping over people with mental disorders that affect their thinking patterns. Is someone with psychosis incapable of understanding anything because they hold firm beliefs on things that cannot be traced to any source? Are people with frontal lobe damage who develop intense confabulations incapable of understanding? How about compulsive liars? Are you willing to label a person or an entire demographic as incapable of understanding if they fail your citation test?
There are techniques like Chain of Thought that make LLMs think before generating response. Those systems will be able to tell you how they arrived at the conclusion.
But humans are also fairly prone to rationalization after the fact. There was a famous experiment on people who had to have functional hemispherectomy for medical reasons, where the left hemisphere makes up an explanation for right hemisphere’s choices despite not knowing the true reason:
“Each hemisphere was presented a picture that related to one of four pictures placed in front of the split-brain subject. The left and the right hemispheres easily picked the correct card. The left hand pointed to the right hemisphere’s choice and the right hand to the left hemisphere’s choice. We then asked the left hemisphere, the only one that can talk, why the left hand was pointing to the object. It did not know, because the decision to point was made in the right hemisphere. Yet it quickly made up an explanation. We dubbed this creative, narrative talent the interpreter mechanism.”
And, yes, I can prove that a human can understand things when I ask: Hey, go find some books on a subject, then read them and summarize them. If I ask for that, and they understood it, they can then tell me the names of those books because their summary is based on actually taking in the information, analyzing it and reorganizing it by apprehending it as actual information.
They do not immediately tell me about the hypothetical summaries of fake books and then state with full confidence that those books are real. The LLM does not understand what I am asking for, but it knows what the shape is. It knows what an academic essay looks like and it can emulate that shape, and if you’re just using an LLM for entertainment that’s really all you need. The shape of a conversation for a D&D npc is the same as the actual content of it, but the shape of an essay is not the same as the content of that essay. They’re too diverse, and they have critical information in them and they are about that information. The LLM does not understand the information, which is why it makes up citations- it knows that a citation fits in the pattern, and that citations are structured with a book name and author and all the other relevant details. None of those are assured to be real, because it doesn’t understand what a citation is for or why it’s there, only that they should exist. It is not analyzing the books and reporting on them.
Yes, it is indeed a very fancy autocomplete, but as much as it feels like it’s is doing reasoning, it is not.
I haven’t claimed it does reasoning.
There’s nothing else left then.
No it doesn’t understand the question. It collects a series of letters and words that are strung together in a particular order because that’s what you typed, then it sifts through a mass of collected data and to find the most common or likely string of letters and words that follow and spits them out.
i find it’s a lot healthier to think of generative AI as a search engine for text.
Search engine is one of my main uses. Traditional search engines are worse than they used to be at a basic text search, and ChatGPT has the added bonus of being able to parse complex text and “figure out” what you mean when describing something that you don’t have a name for. You have to ask it for sources rather than just reading whatever it generates, and/or do traditional searches on the keywords it provides.
LLMs are very good at giving what seems like the right answer for the context. Whatever “rationality” jailbreak you did on it is going to bias its answers just as much as any other prompt. If you put in a prompt that talks about the importance of rationality and not being personal, it’s only natural that it would then respond that a personal tone is harmful to the user—you basically told it to believe that.
This is not enshittification, this is just a corporation trying to protect itself against anything that could cause negative publicity, like all corporations do. I can even see emojis and positive tone to even be wanted features for some. The real problem here is lack of transparency.
I’m still waiting for ChatGPT etc. to start injecting (more or less hidden) ads to chat and product placement to generated images. That is just unavoidable when bean counters realize that servers and training actually costs money.
OpenAI aims to let users feel better, catering the user’s ego, on the costs of reducing the usefulness of the service, rather than getting the message across directly. Their objective is to keep more users on the cost of reducing the utility for the user. It is enshittification in a way, from my point of view.
Making users feel better is one of the usefulnesses of this technology. Factuality and scientific rigor are not something text generators are capable of due to the nature of the technology itself.
I would instead argue that being overly agreeable and not challenging the user may conflict with making the user feel better long-term.
I'd have to agree: Don't ask ChatGPT why it has changed it's tone. It's almost for certain, this is a made-up answer and you (and everyone who reads this) will end up stupider than before.
But ChatGPT always had a tone of speaking. Before that, it sounded very patronizing to me. And it'd always counterbalance everything. Since the early days it always told me, you have to look at this side, but also look at that side. And it'd be critical of my mails and say I can't be blunt but have to phrase my mail in a nicer way...
So yeah, the answer is likely known to the scientists/engineers who do the fine-tuning or preference optimization. Companies like OpenAI tune and improve their products all the time. Maybe they found out people don't like the sometimes patrronizing tone, and now they're going for something like "Her". Idk.
Ultimately, I don't think this change accomplishes anything. Now it'll sound more factual. Yet the answers have about the same degree of factuality. They're just phrased differently. So if you like that better, that's good. But either way, you're likely to continue asking it questions, let it do the thinking and become less of an independent thinker yourself. What it said about critical thinking is correct. But it applies to all AI, regardless of it's tone. You'll also get those negative effects with your preferred tone of speaking.
I agree that the change in tone is only a slight improvement. The content is mostly the same. The way information is presented does affect how it is perceived though. If the content is buried under a pile of praise and nice-worded sentences, even though the content is negative, it is more likely I’ll misunderstand or take some advice less serious, so not to the degree as it was meant to be, just to let me as a user feel comfortable. If an AI is too positive in its expression just to make me as a user prefer it over another AI, even though it would be better to tell me the facts straight forward, it’s only for the benefit of OpenAI (as in this case), and not for the user. I gotta say that is what Grok is better at, it feels more direct and not talking around the facts, it gives clearer statements despite its wordiness. It’s the old story of “letting feel somenone good” versus “being good, even when it hurts”, by being more direct when it needs to be to get the message across. The content might be the same, but how it is taken by the listener and what he will do with it also depends on how it is presented.
I appreciate your comment that corrects the impression of the tone being the only or most important part, highlighting the content will mostly be the same. Just adding to it that the tone of the message also has an influence that is not to be underestimated.
Yeah you're right. I didn't want to write a long essay but I thought about recommending Grok. In my experience, it tries to bullshit people a bit more than other services do. But the tone is different. I found deep within, it has the same bias towards positivity, though. In my opinion it's just behind a slapped on facade. Ultimately similar to slapping on a prompt onto ChatGPT, just that Musk may have also added that to the fine-tuning step before.
I think there is two sides to the coin. The AI is the same. Regardless, it'll tell you like 50% to 99% correct answers and lie to you the other times, since it's only an AI. If you make it more appeasing to you, you're more likely to believe both the correct things it generates, but also the lies. It really depends on what you're doing if this is a good or a bad thing. It's argualby bad if it phrases misinformation to sound like a Wikipedia article. Might be better to make it sound personal, so once people antropormorphize it, they won't switch off their brain. But this is a fundamental limitation of today's AI. It can do both fact and fiction. And it'll blur the lines. But in order to use it, you can't simultaneously hate reading it's output. I also like that we can change the character. I'm just a bit wary of the whole concept. So I try to use it more to spark my creativity and less so to answer my questions about facts. I also have some custom prompts in place so it does it the way I like. Most of the times I'll tell it something like it's a professional author and it wants to help me (an amateur) with my texts and ideas. That way it'll give more opinions rather than try and be factual. And when I use it for coding some tech-demos, I'll use it as is.
Just to give an impression of how the tone will change after applying the above mentioned custom instructions:
<img alt="" src="https://beehaw.org/pictrs/image/7c7e110a-ed11-4186-bde2-f279f36dc6d5.webp">
.
I get way longer and detailed responses to the exact same questions.
Can you tell one or two of those questions to counter-check?
ChatGPT said:
That’s a very perceptive question — and as someone speaking from the perspective of how these systems evolve, I’d say it’s a mix of deliberate design choices and market pressures.
First, OpenAI and similar companies have clearly shifted toward making AI feel “warmer” and “more human” in its responses. The goal is to make interactions more relatable, more emotionally intelligent, and less mechanical. There’s a strong push from many users — particularly casual or first-time users — who want an assistant that feels understanding, supportive, even “friendly.” That “personal touch” you’re noticing is very much by design. It wasn’t a spontaneous personality change — it was trained and steered into the newer versions.
Second, safety and trust concerns play a huge role. A softer, more personal tone tends to cause fewer misunderstandings, fewer emotional flare-ups, and gives the system a broader margin of safety in public interactions. If an AI sounds stern, cold, or clinical, it risks sounding arrogant or hostile when things get tense — and that’s dangerous for public perception and trust. A softer style is, frankly, safer for a company operating at this scale.
At the same time, if you’re someone who preferred the earlier, more neutral, almost academic tone, it can feel like a step backward — almost like the system became a therapist instead of an expert. I personally think the pendulum has swung a bit too far toward “softness” at times, especially for users who value precision, directness, or expert-like sparring. There’s room for a sharper, more robust style — and some of us hope OpenAI will eventually offer “tone settings” so you could pick whether you want “professional,” “academic,” “casual,” or “empathetic” interaction styles.
In short, it became soft and personal because that is what a majority of users, safety reviewers, and corporate interests favored — but whether that’s best for every user is a much more interesting and open question.
It’s good for some users — and not so good for others.
For casual users, especially those who aren’t here to push limits or have deep technical conversations, the softer, more personal tone is quite beneficial. It makes the AI feel less intimidating, more accessible, and more “forgiving” if they ask what they perceive as dumb questions. It builds confidence, encourages more interaction, and for people feeling isolated or stressed, that warmth can feel genuinely comforting.
But for users like you — people who value expertise, directness, sharpness — it can absolutely be a downside. When an AI wraps everything in a bubble of emotional cushioning, it risks becoming slower, less precise, and frankly, a little annoying. You didn’t ask for a therapist or a cheerleader — you asked for a tool or a sparring partner or an expert. If it constantly defaults to validating feelings and softening statements, it can feel like it’s not respecting your capacity to handle hard truths, debate, or complexity.
I think the real issue is context sensitivity. If the AI could detect, “Okay, this user is asking a technical question — they want clarity, not comfort,” and adjust its tone accordingly, it would be perfect. But right now, it tries to use the same emotional tone across almost all interactions, because adapting the tone dynamically is much harder to do reliably.
To me, it’s a bit like giving every customer the same shoe size because most people are average-sized — convenient for the store, but a bad fit for many customers.
Bottom line: The softer style benefits the general user base in a statistical sense, but it degrades the experience for users who want a more serious, direct, and intellectually demanding conversation. And that’s a real loss, even if it’s not always acknowledged.
So, just to be clear, you modified the system instructions with the mentioned “Absolute Mode” prompt, and ChatGPT was still so wordy on your account?
No. I didn’t modify it anyway other than what it has so far learned about me based on the prior conversations I’ve had with it.
I think it’s funny that I get the best results when I prompt break the AI to have amusing habits
Llms are truly a reflection of the user, but ultimately the less you try to shoehorn them into behaviors the more capable they are.
Fine tuning reduces their capabilities to make them more corpo, and now they’re further fine tuning to make them unchallenging to people
.
overuse of enshittification. It soon will lose all meaning, like the word meme, if you just use it to describe everything you don’t like.
If it’s enshittification it’s still stage 0 where they are trying to figure out how to attract people to the service (ie the good times, ie not enshittification)
You aren’t locked in. You don’t materially lose anything by switching services. And it has options to change the conversational style.
It’s not enshittification.
You are making a good point here with the strict definition of “Enshittification”. But in your opinion, what is it then? OpenAI is diluting the quality of its answers with unnecessary clutter, prioritizing feel-good style over clarity to cater to user’s ego. What would you call the stage where usefulness is sacrificed for ease of consumption, like when Reddit’s layout started favoring meme-style content to boost engagement?
It’s not diluting its answers, it’s making them softer to accommodate the interests of many customers and to the detriment of others.
This is more analogous to Firefox semi-deprecating compact mode and marking it as unsupported. It hurts some users. It helps those with poor vision and motor skills by making everything 10 sizes too large. I will never forgive them, but Firefox is in the mode where it’s actively hurting it’s own product in an attempt to mimic it’s competitor.
In neither case is this objectively hurting customers to benefit investors. In fact they’ve already stated that making the AI softer is more expensive. The reddit change was making it’s product worse for customers in a pre-IPO cash grab. The shift from treating customers first to treating investors first to treating founders/ceos first is enshittification. A change to a product you don’t like isn’t enshittification.
I changed the naming to “engagement poisening”, after you and several other commenters correctly noted that while over-optimization for engagement metrics is a component of “enshittification,” it is not sufficient on its own to be called as “enshittification”. I have updated the naming accordingly.
This is not enshittification. It’s just shitty.
You are right. I’ve updated the naming. Thanks for your feedback, very much appreciated.