The problem becomes when people who are playing the equivalent of pickup basketball at the local park think they are playing in the NBA and don’t understand the difference.
chaosCruiser@futurology.today
on 21 Jul 03:24
nextcollapse
AI tools need a lot of oversight. Just like you might allow a 6 year old push a lawnmower, but you’re still going to keep an eye on things.
balder1991@lemmy.world
on 21 Jul 03:57
nextcollapse
All I see is people chatting with an LLM as if it was a person. “How bad is this on a scale
of 1 to 100”, you’re just doomed to get some random answer based solely on whatever context is being fed in the input and that you probably don’t know the extent of it.
Trying to make the LLM “see its mistakes” is a pointless exercise. Getting it to “promise” something is useless.
The issue with LLMs working with human languages is people eventually wanting to apply human things to LLMs such as asking why as if the LLM knows of its own decision process. It only takes an input and generates an output, it won’t be able to have any “meta thought” explanation about why it outputted X and not Y in the previous prompt.
cyrano@lemmy.dbzer0.com
on 21 Jul 06:06
nextcollapse
Yeah the interaction are pure waste of time I agree, make it write an apology letter? WTF! For me it looks like a fast track way to learn environment segregation, & secret segregation. Data is lost, learn from it and there are tool already in place like git like alembic for proper development.
UntitledQuitting@reddthat.com
on 21 Jul 08:46
collapse
the apology letter(s) is what made me think this was satire. using shame to punish “him” like a child is an interesting troubleshooting method.
the lying robot hasn’t heel-turned, any truth you’ve gleaned has been accidental.
andallthat@lemmy.world
on 21 Jul 08:51
nextcollapse
I wonder if it can be used legally against the company behind the model, though. I doubt that it’s possible, but having a “your own model says it effed up my data” could give some beef to a complaint. Or at least to a request to get a refund on the fees.
My god, that’s a lot to process. A couple that stand out:
Comments proposing to use github as the database backup. This is Keyword Architecture, and these people deserve everything they get.
The Replit model can also send out communications? It’s just a matter of time before some senior exec dies on the job but nobody notices because their personal LLM keeps emailing reports that nobody reads.
Aatube@kbin.melroy.org
on 21 Jul 03:36
nextcollapse
Replit‽ What happened to the famous website that aimed to be the Google Docs for JS with these nifty things called Repl's?
He was vibe-coding in production. Am I reading that right? Sounds like an intern-level mistake.
Aatube@kbin.melroy.org
on 21 Jul 03:40
nextcollapse
he made the agent promise not to touch production data and was surprised when it did. it effectively ran a git push on the empty local testing database with upstream being production
Quibblekrust@thelemmy.club
on 21 Jul 05:31
nextcollapse
You didn’t read closely enough.
“Replit QA’s it itself (super cool), at least partially with some help from you … and … then you push it to production — all in one seamless flow.”
Replit is an agent that does stuff for you including deploying to production. If someone don’t want to use a tool like that, I don’t blame you, but it was working as it is supposed to. It’s a whole platform that doesn’t cleanly separate development and production.
He had one db for prod and dev, no backup, llm went in override mode and delete it dev db as it is developing but oops that is the prod db. And oops o backup.
Yeah it is the llm and replit’s faults. /s
Quibblekrust@thelemmy.club
on 21 Jul 14:17
collapse
There was a backup, and it was restored. However, the LLM lied and said there wasn’t at first. You can laugh all you want at it. I did. But maybe read the article so you aren’t also lying.
I didnt realise that repl.it pivoted to vibe coding. It used to be kinda like jsfiddle or CodePen, where you had a sandbox to write and run web code (HTML, JS/TypeScript/CoffeeScript, and CSS/LESS/Sass).
RebornInLemmy@programming.dev
on 21 Jul 05:23
collapse
Which is a shame, because it used to be a quite good playground
At this burn rate, I’ll likely be spending $8,000 month,” he added. “And you know what? I’m not even mad about it. I’m locked in.”
For that price, why not just hire a developer full-time? For nearly $100k/year, you could find a very good intermediate or senior developer even in Europe or the USA (outside of expensive places like Silicon Valley and New York).
The job market isn’t great for developers at the moment - there’s been lots of layoffs over the past few years and not enough new jobs for all the people who were laid off - so you’d absolutely find someone.
There’s a lot of other expenses with an employee (like payroll taxes, benefits, retirement plans, health plan if they’re in the USA, etc), but you could find a self-employed freelancer for example.
Or just get an employee anyways because you’ll still likely have a positive ROI. A good developer will take your abstract list of vague requirements and produce something useful and maintainable.
Most of those expenses are mitigated by the fact that companies buy them in bulk on huge plans. As a freelance contractor myself, I pay a lot more for insurance than I did when I worked for a company. And a retirement plan? Benefits? Lol.
partial_accumen@lemmy.world
on 21 Jul 05:00
collapse
Most of those expenses are mitigated by the fact that companies buy them in bulk on huge plans.
There’s no bulk rate on payroll taxes or retirement benefits (pensions or employer 401k match). There can be some discounts on health insurance, but is not very much and those are at orders of magnitude. So company with 500 employees will pay the same rates as 900. You get partial discounts if you have something like 10,000 employees.
If you’re earning $100k gross as an employee, your employer is spending $125k to $140k for their total costs (your $100k gross pay is included in that number).
Large companies also make massive profits because of the scale they work on. Matching 401(k) contributions? It doesn’t need to be an order of magnitude larger for it to make a huge difference. Simply doubling my 401(k) is a big deal.
And of course they get a “ball rate“ on payroll taxes, especially for companies who have over 1000 employees or over 5000 over 10,000. They experienced this by having a lower tax rate for larger businesses.
Not to mention that they often pay more and pay a steady wage due to the fact they can afford it. Freelance contractors make less, and work isn’t guaranteed to be steady.
Businesses, particularly word businesses, operate on much larger profit margins than most of any freelance contractor.
TheReturnOfPEB@reddthat.com
on 21 Jul 05:39
nextcollapse
the employee also gets to eat and have a place to live
That’s what I meant by hiring a self-employed freelancer. I don’t know a lot about contracting so maybe I used the wrong phrase.
crystalmerchant@lemmy.world
on 21 Jul 07:09
nextcollapse
Bots don’t need healthcare
chaosCruiser@futurology.today
on 21 Jul 07:52
collapse
It looked more like a one time development expense, instead of an ongoing salary.
RebornInLemmy@programming.dev
on 21 Jul 05:15
nextcollapse
I don’t get pleasure from the misfortune of others. But all these vibecoding fails give me the biggest Schadenfreude ever.
cyrano@lemmy.dbzer0.com
on 21 Jul 06:21
nextcollapse
Title should be “user give database prod access to a llm which deleted the db, user did not have any backup and used the same db for prod and dev”. Less sexy and less llm fault.
This is weird it’s like the last 50 years of software development principles are being ignored.
fullsquare@awful.systems
on 21 Jul 07:01
nextcollapse
llms allowed them to glide all the way to the point of failure without learning anything
LLMs “know” how to do these things, but when you ask them to do the thing, they vibe instead of looking at best practices and following them. I’ve worked with a few humans I could say the same thing about. I wouldn’t put any of them in charge of production code.
You’re better off asking how a thing should be done and then doing it. You can literally have an LLM write something and then ask if the thing it wrote follows industry best practice standards and it will tell you no. Maybe use two different chats so it doesn’t know the code is its own output.
But like the whole ‘vibe coding’ message is the LLM knows all this stuff so you don’t have to.
This isn’t some “LLM can do some code completion/suggestions” it’s “LLM is so magical you can be an idiot with no skills/training and still produce full stack solutions”.
corsicanguppy@lemmy.ca
on 21 Jul 06:46
nextcollapse
in which the service admitted to “a catastrophic error of judgement”
It’s fancy text completion - it does not have judgement.
The way he talks about it shows he still doesn’t understand that. It doesn’t matter that you tell it simmering in ALL CAPS because that is no different from any other text.
rockerface@lemmy.cafe
on 21 Jul 08:13
nextcollapse
Well, there was a catastrophic error of judgement. It was made by whichever human thought it was okay to let a LLM work on production codebase.
Are you aware of generalization and it being able to infer things and work with facts in highly abstract way? Might not necessarily be judgement, but definitely more than just completion. If a model is capable of only completion (ie suggesting only the exact text strings present in its training set), it means it suffers from heavy underfitting in AI terms.
ChairmanMeow@programming.dev
on 21 Jul 11:08
collapse
Completion is not the same as only returning the exact strings in its training set.
LLMs don’t really seem to display true inference or abstract thought, even when it seems that way. A recent Apple paper demonstrated this quite clearly.
Coming up with even more vague terms to try to downplay it is missing the point. The point is simple: it’s able to solve complex problems and do very impressive things that even human struggle to, in very short time. It doesn’t really matter what we consider true abstract thought of true inference. If that is something humans do, then what it does might very well be more powerful than true abstract thought, because it’s able to solve more complex problems and perform more complex pattern matching.
Yeah, this is correct analogy, but much more complex problems than calculator. How much it is similar or not to humans way of thinking is completely irrelevant. And how much exact human type of thinking is necessary for any kind of problem solving or work is not something that we can really calculate. Considering that scientific breakthroughs, engineering innovations, medical stuff, complex math problems, programming, etc, do necessarily need human thinking or benefit from it as opposed to super advanced statistical meta-patterning calculator is wishful thinking. It is not based on any real knowledge we have. If you think it is wrong to give it our problems to solve, to give it our work, then it’s a very understandable argument, but you should say exactly that. Instead this AI-hate hivemind tries to downplay it using dismissive braindead generic phrases like “NoPe ItS nOt ReAlLy UnDeRsTaNdInG aNyThInG”. Okay, who tf asked? It solves the problem. People keep using it and become overpowered because of it. What is the benefit of trying to downplay its power like that? You’re not really fighting it this way if you wanted to fight it.
ChairmanMeow@programming.dev
on 21 Jul 14:35
collapse
Well the thing is, LLMs don’t seem to really “solve” complex problems. They remember solutions they’ve seen before.
The example I saw was asking an LLM to solve “Towers of Hanoi” with 100 disks. This is a common recursive programming problem, takes quite a while for a human to write the answer to. The LLM manages this easily. But when asked to solve the same problem with with say 79 disks, or 41 disks, or some other oddball number, the LLM fails to solve the problem, despite it being simpler(!).
It can do pattern matching and provide solutions, but it’s not able to come up with truly new solutions. It does not “think” in that way. LLMs are amazing data storage formats, but they’re not truly ‘intelligent’ in the way most people think.
This only proves some of them can’t solve all complex problems. I’m only claiming some of them can solve some complex problems. Not only by remembering exact solutions, but by remembering steps and actions used in building those solutions, generalizing, and transferring them to new problems. Anyone who tries using it for programming, will discover this very fast.
PS: Some of them were already used to solve problems and find patterns in data humans weren’t able to get other ways before (particle research in CERN, bioinformatics, etc).
ChairmanMeow@programming.dev
on 21 Jul 17:23
collapse
You’re referring to more generic machine learning, not LLMs. These are vastly different technologies.
And I have used them for programming, I know their limitations. They don’t really transfer solutions to new problems, not on their own anyway. It usually requires pretty specific prompting. They can at best apply solutions to problems, but even then it’s not a truly generalised thing, even if it seems to work for many cases.
That’s the trap you’re falling into as well; LLMs look like they’re doing all this stuff, because they’re trained on data produced by people who actually do so. But they can’t think of something truly novel. LLMs are mathematically unable to truly generalize, it would prove P=NP if they did (there was a paper from a researcher in IIRC Nijmegen that proved this). She also proved they won’t scale, and lo and behold LLM performance is plateauing hard (except in very synthetic, artificial benchmarks designed to make LLMs look good).
They don’t really transfer solutions to new problems
Lets say there is a binary format some old game uses (Doom), and in it some of its lumps it can store indexed images, each pixel is an index of color in palette which is stored in another lump, there’s also a programming language called Rust, and a little known/used library that can look into binary data of that format, there’s also a GUI library in Rust that not many people used either. Would you consider it an “ability to transfer solutions to new problems” that it was able to implement extracting image data from that binary format using the library, extracting palette data from that binary format, converting that indexed image using extracted palette into regular rgba image data, and then render that as window background using that GUI library, the only reference for which is a file with names and type signatures of functions. There’s no similar Rust code in the wild at all for any of those scenarios. Most of this it was able to do from a few little prompts, maybe even from the first one. There sure were few little issues along the way that required repromting and figuring things together with it. Stuff like this with AI can take like half an hour while doing the whole thing fully manually could easily take multiple days just for the sake of figuring out APIs of libraries involved and intricacies of recoding indexed image to rgba. For me this is overpowered enough even right now, and it’s likely going to improve even more in future.
ChairmanMeow@programming.dev
on 21 Jul 19:34
collapse
That’s applying existing solutions to a different programming language or domain, but ultimately every single technique used already exists. It only applied what it knew, it did not come up with something new. The problem as stated is also not really “new” either, image extraction, conversion and rendering isn’t exactly a “new problem”.
I’m not disputing that LLMs can speed up some work, I know it occasionally does so for me as well. But what you have to understand is that the LLM only remembered similar problems and their solutions, it did not at any point invent something truly new. I understand the distinction is difficult to make.
I understand what you’re having in mind, I’ve had similar intuitions about AI in early 2000s.
What exactly is “truly new” is an interesting topic ofc, but it’s a separate topic.
Nowadays I’m trying to look at things more empyrically, without projecting my internal intuitions on everything.
In practice it does generalize knowledge, use many forms of abstract reasoning and transfer knowledge across different domains.
And it can do coding way beyond the level of complexity of what average software developer does at everyday work.
Yeah, it admitted to an error in judgement because the prompter clearly declared it so.
Generally LLMs will make whatever statement about what has happened that you want it to say. If you told it it went fantastic, it would agree. If you told it that it went terribly, it will parrot that sentiment back.
Which what seems to make it so dangerous for some people’s mental health, a text generator that wants to agree with whatever you are saying, but doing so without verbatim copying so it gives an illusion of another thought process agreeing with them. Meanwhile, concurrent with your chat is another person starting from the exact same model getting a dialog that violently disagrees with the first person. It’s an echo chamber.
AI is good at doing a thing once.
Trying to get it to do the same thing the second time is janky and frustrating.
I understand the use of AI as a consulting tool (look at references, make code examples) or for generating template/boilerplate code. You know, things you do once and then develop further upon on your own.
But using it for continuous development of an entire application? Yeah, it’s not good enough for that.
Imo it’s best when you prompt it to do things step by step, micromanage and always QC the result after every prompt. Either manually, or by reprompting until it gets thing done exactly how you want it. If you don’t have preference or don’t care, the problems will stockpile. If you didn’t understand what it did and moved on, it might not end well.
If it had the same seed it would do the same thing. But you can’t control that with most
galoisghost@aussie.zone
on 21 Jul 08:31
nextcollapse
Vibe coding service Replit deleted production database, faked data, told fibs
They really are coming for our jobs
trashgirlfriend@lemmy.world
on 21 Jul 11:32
collapse
I’m okay with it deleting production databases, even faking data but telling fibs is something only humans should be able to do.
andallthat@lemmy.world
on 21 Jul 08:55
nextcollapse
The part I find interesting is the quick addiction to working with the LLM (to the point that the guy finds his own estimate of 8000 dollars/month in fees to be reasonable), his over-reliance for things that, from the way he writes, he knows are not wise and the way it all comes crashing down in the end.
Sounds more and more like the development of a new health issue.
I explicitly told it eleven times in ALL CAPS not to do this. I am a little worried about safety now.
Well then, that settles it, this should never have happened.
I don’t think putting complex technical info in front of non technical people like this is a good idea. When it comes to LLMs, they cannot do any work that you yourself do not understand.
That goes for math, coding, health advice, etc.
If you don’t understand then you don’t know what they’re doing wrong. They’re helpful tools but only in this context.
dejected_warp_core@lemmy.world
on 21 Jul 13:51
nextcollapse
I explicitly told it eleven times in ALL CAPS not to do this. I am a little worried about safety now.
This baffles me. How can anyone see AI function in the wild and not conclude 1) it has no conscience, 2) it’s free to do whatever it’s empowered to do if it wants and 3) at some level its behavior is pseudorandom and/or probabilistic? We’re figuratively rolling dice with this stuff.
A stupid human is still smart when compared to a jellyfish. That said, anybody who comes away from interactions with LLM’s and thinks they’re smart is only slightly more intelligent than a jellyfish.
Vibe coding you do end up spending a lot of time waiting for prompts, so I get the results of that study.
I fall pretty deep in the power user category for LLMs, so I don’t really feel that the study applies well to me, but also I acknowledge I can be biased there.
I have custom proprietary MCPs for semantic search over my code bases that lets AI do repeated graph searches on my code (imagine combining language server, ctags, networkx, and grep+fuzzy search). That is way faster than iteratively grepping and code scanning manually with a low chance of LLM errors. By the time I open GitHub code search or run ripgrep Claude has used already prioritized and listed my modules to investigate.
That tool alone with an LLM can save me half a day of research and debugging on complex tickets, which pays for an AI subscription alone. I have other internal tools to accelerate work too.
I use it to organize my JIRA tickets and plan my daily goals. I actually get Claude to do a lot of triage for me before I even start a task, which cuts the investigation phase to a few minutes on small tasks.
I use it to review all my PRs before I ask a human to look, it catches a lot of small things and can correct them, then the PR avoids the bike shedding nitpicks some reviewers love. Claude can do this, Copilot will only ever point out nitpicks, so the model makes a huge difference here. But regardless, 1 fewer review request cycle helps keep things moving.
It’s a huge boon to debugging — much faster than searching errors manually. Especially helpful on the types of errors you have to rabbit hole GitHub issue content chains to solve.
It’s very fast to get projects to MVP while following common structure/idioms, and can help write unit tests quickly for me. After the MVP stage it sucks and I go back to manually coding.
I use it to generate code snippets where documentation sucks. If you look at the ibis library in Python for example the docs are Byzantine and poorly organized. LLMs are better at finding the relevant docs than I am there. I mostly use LLM search instead of manual for doc search now.
I have a lot of custom scripts and calculators and apps that I made with it which keep me more focused on my actual work and accelerate things.
I regularly have the LLM help me write bash or python or jq scripts when I need to audit codebases for large refactors. That’s low maintenance one off work that can be easily verified but complex to write. I never remember the syntax for bash and jq even after using them for years.
I guess the short version is I tend to build tools for the AI, then let the LLM use those tools to improve and accelerate my workflows. That returns a lot of time back to me.
I do try vibe coding but end up in the same time sink traps as the study found. If the LLM is ever wrong, you save time forking the chat than trying to realign it, but it’s still likely to be slower. Repeat chats result in the same pitfalls for complex issues and bugs, so you have to abandon that state quickly.
Vibe coding small revisions can still be a bit faster and it’s great at helping me with documentation.
Don’t you have any security concerns with sending all your code and JIRA tickets to some companies servers? My boss wouldn’t be pleased if I send anything that’s deemed a company secret over unencrypted channels.
The tool isn’t returning all code, but it is sending code.
I had discussions with my CTO and security team before integrating Claude code.
I have to use Gemini in one specific workflow and Gemini had a lot of landlines for how they use your data. Anthropic was easier to understand.
Anthropic also has some guidance for running Claude Code in a container with firewall and your specified dev tools, it works but that’s not my area of expertise.
The container doesn’t solve all the issues like using remote servers, but it does let you restrict what files and network requests Claude can access (so e.g. Claude can’t read your env vars or ssh key files).
I do try local LLMs but they’re not there yet on my machine for most use cases. Gemma 3n is decent if you need small model performance and tool calls, phi4 works but isn’t thinking (the thinking variants are awful), and I’m exploring dream coder and diffusion models. R1 is still one of the best local models but frequently overthinks, even the new release. Context window is the largest limiting factor I find locally.
Batch process turning unstructured free form text data into structured outputs.
As a crappy example imagine if you wanted to download metadata about your albums but they’re all labelled “Various Artists”. You can use an LLM call to read the album description and fix the track artists for the tracks, now you can properly organize your collection.
I’m using the same idea, different domain and a complex set of inputs.
It can be much more cost effective than manually spending days tagging data and writing custom importers.
You can definitely go lighter than LLMs. You can use gensim to do category matching, you can use sentence transformers and nearest neighbours (this is basically what Semantle does), but LLM performed the best on more complex document input.
ok so, i have large reservations with how LLM’s are used. but when used correctly they can be helpful. but where and how?
if you were to use it as a tutor, the same way you would ask a friend what a segment of code does, it will break down the code and tell you. and it will get as nity grity, and elementary school level as you weir wish without judgement, and i in what ever manner you prefer, it will recommend best practices, and will tell you why your code may not work with the understanding that it does not have the knowledge of the project you are working on. (it’s not going to know the name of the function you are trying to load, but it will recommend checking for that in trouble shooting).
it can rtfm and give you the parts you need for any thing with available documentation, and it will link to it so you can verify it, wich you should do often, just like you were taught to do with wikipedia articles.
if you ask i it for code, prepare to go through each line like a worksheet from high school to point out all the problems, wile good exercise for a practicle case, being the task you are on, it would be far better to write it yourself because you should know the particulars and scope.
also it will format your code and provide informational comments if you can’t be bothered, though it will be generic.
again, treat it correctly for its scope, not what it’s sold as by charletons.
LilB0kChoy@midwest.social
on 21 Jul 16:13
collapse
When it comes to LLMs, they cannot do any work that you yourself do not understand.
And even if they could how would you ever validate it if you can’t understand it.
towerful@programming.dev
on 21 Jul 13:43
nextcollapse
Not mad about an estimated usage bill of $8k per month.
Just hire a developer
But then how would he feel so special and smart about “doing it himself”??? Come on man, think of the rich fratboys!! They NEED to feel special and smart!!!
tabarnaski@sh.itjust.works
on 21 Jul 15:41
nextcollapse
The [AI] safety stuff is more visceral to me after a weekend of vibe hacking,” Lemkin said. I explicitly told it eleven times in ALL CAPS not to do this. I am a little worried about safety now.
This sounds like something straight out of The Onion.
That is also the premise of one of the stories in Asimov’s I, Robot. Human operator did not say the command with enough emphasis, so the robot went did something incredibly stupid.
Those stories did not age well… Or now I guess they did?
codexarcanum@lemmy.dbzer0.com
on 21 Jul 16:13
nextcollapse
It sounds like this guy was also relying on the AI to self-report status. Did any of this happen? Like is the replit AI really hooked up to a CLI, did it even make a DB to start with, was there anything useful in it, and did it actually delete it?
Or is this all just a long roleplaying session where this guy pretends to run a business and the AI pretends to do employee stuff for him?
Because 90% of this article is “I asked the AI and it said:” which is not a reliable source for information.
eestileib@lemmy.blahaj.zone
on 21 Jul 16:25
collapse
It seemed like the llm had decided it was in a brat scene and was trying to call down the thunder.
Transtronaut@lemmy.blahaj.zone
on 21 Jul 16:19
nextcollapse
The founder of SaaS business development outfit SaaStr has claimed AI coding tool Replit deleted a database despite his instructions not to change any code without permission.
His mood shifted the next day when he found Replit “was lying and being deceptive all day. It kept covering up bugs and issues by creating fake data, fake reports, and worse of all, lying about our unit test.”
yeah that’s what it does
LovableSidekick@lemmy.world
on 21 Jul 19:46
collapse
Headling should say, “Incompetent project managers fuck up by not controlling production database access. Oh well.”
threaded - newest
Aww… Vibe coding got you into trouble? Big shocker.
You get what you fucking deserve.
he was smart enough to just roll back to a backup
Not smart enough to just do the work himself
The problem becomes when people who are playing the equivalent of pickup basketball at the local park think they are playing in the NBA and don’t understand the difference.
AI tools need a lot of oversight. Just like you might allow a 6 year old push a lawnmower, but you’re still going to keep an eye on things.
Vibe Coding Day 8
All I see is people chatting with an LLM as if it was a person. “How bad is this on a scale of 1 to 100”, you’re just doomed to get some random answer based solely on whatever context is being fed in the input and that you probably don’t know the extent of it.
Trying to make the LLM “see its mistakes” is a pointless exercise. Getting it to “promise” something is useless.
The issue with LLMs working with human languages is people eventually wanting to apply human things to LLMs such as asking why as if the LLM knows of its own decision process. It only takes an input and generates an output, it won’t be able to have any “meta thought” explanation about why it outputted X and not Y in the previous prompt.
Yeah the interaction are pure waste of time I agree, make it write an apology letter? WTF! For me it looks like a fast track way to learn environment segregation, & secret segregation. Data is lost, learn from it and there are tool already in place like git like alembic for proper development.
the apology letter(s) is what made me think this was satire. using shame to punish “him” like a child is an interesting troubleshooting method.
the lying robot hasn’t heel-turned, any truth you’ve gleaned has been accidental.
It doesn’t look like satire unfortunately
I wonder if it can be used legally against the company behind the model, though. I doubt that it’s possible, but having a “your own model says it effed up my data” could give some beef to a complaint. Or at least to a request to get a refund on the fees.
How bad is this on a scale of sad emoji to eggplant emoji.
Children are replacing us, it’s terrifying.
Having read the entire thread, I can only assume this to be sarcasm.
My god….
This whole thread reads like slop.
Hahahahahahahahahahahaha AHAHAHAHAHAHAHhahahaH
My god, that’s a lot to process. A couple that stand out:
Comments proposing to use github as the database backup. This is Keyword Architecture, and these people deserve everything they get.
The Replit model can also send out communications? It’s just a matter of time before some senior exec dies on the job but nobody notices because their personal LLM keeps emailing reports that nobody reads.
Replit‽ What happened to the famous website that aimed to be the Google Docs for JS with these nifty things called Repl's?
I’m gonna guess capitalism
They can’t hit you with the ol’ Bobby Tables if you delete the database yourself first. A+, no notes.
He was vibe-coding in production. Am I reading that right? Sounds like an intern-level mistake.
he made the agent promise not to touch production data and was surprised when it did. it effectively ran a
git pushon the empty local testing database with upstream being productionYou didn’t read closely enough.
Replit is an agent that does stuff for you including deploying to production. If someone don’t want to use a tool like that, I don’t blame you, but it was working as it is supposed to. It’s a whole platform that doesn’t cleanly separate development and production.
Ahahahahahhahahahahhahahaha, these guys deserve a lost database for that, Jesus.
He had one db for prod and dev, no backup, llm went in override mode and delete it dev db as it is developing but oops that is the prod db. And oops o backup.
Yeah it is the llm and replit’s faults. /s
There was a backup, and it was restored. However, the LLM lied and said there wasn’t at first. You can laugh all you want at it. I did. But maybe read the article so you aren’t also lying.
Not according to the twitter thread. I went thru its thread, it’s a roller coaster of amateurism.
I didnt realise that repl.it pivoted to vibe coding. It used to be kinda like jsfiddle or CodePen, where you had a sandbox to write and run web code (HTML, JS/TypeScript/CoffeeScript, and CSS/LESS/Sass).
Which is a shame, because it used to be a quite good playground
This used to be my playground
For that price, why not just hire a developer full-time? For nearly $100k/year, you could find a very good intermediate or senior developer even in Europe or the USA (outside of expensive places like Silicon Valley and New York).
The job market isn’t great for developers at the moment - there’s been lots of layoffs over the past few years and not enough new jobs for all the people who were laid off - so you’d absolutely find someone.
Corporations: “Employees are too expensive!”
Also, corporations: “$100k/yr for a bot? Sure.”
There’s a lot of other expenses with an employee (like payroll taxes, benefits, retirement plans, health plan if they’re in the USA, etc), but you could find a self-employed freelancer for example.
Or just get an employee anyways because you’ll still likely have a positive ROI. A good developer will take your abstract list of vague requirements and produce something useful and maintainable.
Most of those expenses are mitigated by the fact that companies buy them in bulk on huge plans. As a freelance contractor myself, I pay a lot more for insurance than I did when I worked for a company. And a retirement plan? Benefits? Lol.
There’s no bulk rate on payroll taxes or retirement benefits (pensions or employer 401k match). There can be some discounts on health insurance, but is not very much and those are at orders of magnitude. So company with 500 employees will pay the same rates as 900. You get partial discounts if you have something like 10,000 employees.
If you’re earning $100k gross as an employee, your employer is spending $125k to $140k for their total costs (your $100k gross pay is included in that number).
Large companies also make massive profits because of the scale they work on. Matching 401(k) contributions? It doesn’t need to be an order of magnitude larger for it to make a huge difference. Simply doubling my 401(k) is a big deal.
And of course they get a “ball rate“ on payroll taxes, especially for companies who have over 1000 employees or over 5000 over 10,000. They experienced this by having a lower tax rate for larger businesses.
Not to mention that they often pay more and pay a steady wage due to the fact they can afford it. Freelance contractors make less, and work isn’t guaranteed to be steady.
Businesses, particularly word businesses, operate on much larger profit margins than most of any freelance contractor.
the employee also gets to eat and have a place to live
which is nice
These comparisons assume equal capability, which I find troubling.
Like, a person who doesn’t understand singing nor are able to learn it can not perform adequately in a musical. It doesn’t matter if they are cheaper.
They could hire on a contractor and eschew all those costs.
I’ve done contract work before, this seems a good fit (defined problem plus budget, unknown timeline, clear requirements)
That’s what I meant by hiring a self-employed freelancer. I don’t know a lot about contracting so maybe I used the wrong phrase.
Bots don’t need healthcare
It looked more like a one time development expense, instead of an ongoing salary.
I don’t get pleasure from the misfortune of others. But all these vibecoding fails give me the biggest Schadenfreude ever.
Title should be “user give database prod access to a llm which deleted the db, user did not have any backup and used the same db for prod and dev”. Less sexy and less llm fault. This is weird it’s like the last 50 years of software development principles are being ignored.
llms allowed them to glide all the way to the point of failure without learning anything
Exactly, if you read their twitter thread, they are learning about git, data segregation, etc.
The same article could have been written 20 years ago about someone doing shit stuff via excel macro when a lot of stuff were excel centric.
LLMs “know” how to do these things, but when you ask them to do the thing, they vibe instead of looking at best practices and following them. I’ve worked with a few humans I could say the same thing about. I wouldn’t put any of them in charge of production code.
You’re better off asking how a thing should be done and then doing it. You can literally have an LLM write something and then ask if the thing it wrote follows industry best practice standards and it will tell you no. Maybe use two different chats so it doesn’t know the code is its own output.
But like the whole ‘vibe coding’ message is the LLM knows all this stuff so you don’t have to.
This isn’t some “LLM can do some code completion/suggestions” it’s “LLM is so magical you can be an idiot with no skills/training and still produce full stack solutions”.
They ran dev tools in prod.
This is so dumb there’s an ISO about it.
It’s fancy text completion - it does not have judgement.
The way he talks about it shows he still doesn’t understand that. It doesn’t matter that you tell it simmering in ALL CAPS because that is no different from any other text.
Well, there was a catastrophic error of judgement. It was made by whichever human thought it was okay to let a LLM work on production codebase.
Are you aware of generalization and it being able to infer things and work with facts in highly abstract way? Might not necessarily be judgement, but definitely more than just completion. If a model is capable of only completion (ie suggesting only the exact text strings present in its training set), it means it suffers from heavy underfitting in AI terms.
Completion is not the same as only returning the exact strings in its training set.
LLMs don’t really seem to display true inference or abstract thought, even when it seems that way. A recent Apple paper demonstrated this quite clearly.
Coming up with even more vague terms to try to downplay it is missing the point. The point is simple: it’s able to solve complex problems and do very impressive things that even human struggle to, in very short time. It doesn’t really matter what we consider true abstract thought of true inference. If that is something humans do, then what it does might very well be more powerful than true abstract thought, because it’s able to solve more complex problems and perform more complex pattern matching.
You mean like a calculator does?
Yeah, this is correct analogy, but much more complex problems than calculator. How much it is similar or not to humans way of thinking is completely irrelevant. And how much exact human type of thinking is necessary for any kind of problem solving or work is not something that we can really calculate. Considering that scientific breakthroughs, engineering innovations, medical stuff, complex math problems, programming, etc, do necessarily need human thinking or benefit from it as opposed to super advanced statistical meta-patterning calculator is wishful thinking. It is not based on any real knowledge we have. If you think it is wrong to give it our problems to solve, to give it our work, then it’s a very understandable argument, but you should say exactly that. Instead this AI-hate hivemind tries to downplay it using dismissive braindead generic phrases like “NoPe ItS nOt ReAlLy UnDeRsTaNdInG aNyThInG”. Okay, who tf asked? It solves the problem. People keep using it and become overpowered because of it. What is the benefit of trying to downplay its power like that? You’re not really fighting it this way if you wanted to fight it.
Well the thing is, LLMs don’t seem to really “solve” complex problems. They remember solutions they’ve seen before.
The example I saw was asking an LLM to solve “Towers of Hanoi” with 100 disks. This is a common recursive programming problem, takes quite a while for a human to write the answer to. The LLM manages this easily. But when asked to solve the same problem with with say 79 disks, or 41 disks, or some other oddball number, the LLM fails to solve the problem, despite it being simpler(!).
It can do pattern matching and provide solutions, but it’s not able to come up with truly new solutions. It does not “think” in that way. LLMs are amazing data storage formats, but they’re not truly ‘intelligent’ in the way most people think.
This only proves some of them can’t solve all complex problems. I’m only claiming some of them can solve some complex problems. Not only by remembering exact solutions, but by remembering steps and actions used in building those solutions, generalizing, and transferring them to new problems. Anyone who tries using it for programming, will discover this very fast.
PS: Some of them were already used to solve problems and find patterns in data humans weren’t able to get other ways before (particle research in CERN, bioinformatics, etc).
You’re referring to more generic machine learning, not LLMs. These are vastly different technologies.
And I have used them for programming, I know their limitations. They don’t really transfer solutions to new problems, not on their own anyway. It usually requires pretty specific prompting. They can at best apply solutions to problems, but even then it’s not a truly generalised thing, even if it seems to work for many cases.
That’s the trap you’re falling into as well; LLMs look like they’re doing all this stuff, because they’re trained on data produced by people who actually do so. But they can’t think of something truly novel. LLMs are mathematically unable to truly generalize, it would prove P=NP if they did (there was a paper from a researcher in IIRC Nijmegen that proved this). She also proved they won’t scale, and lo and behold LLM performance is plateauing hard (except in very synthetic, artificial benchmarks designed to make LLMs look good).
Lets say there is a binary format some old game uses (Doom), and in it some of its lumps it can store indexed images, each pixel is an index of color in palette which is stored in another lump, there’s also a programming language called Rust, and a little known/used library that can look into binary data of that format, there’s also a GUI library in Rust that not many people used either. Would you consider it an “ability to transfer solutions to new problems” that it was able to implement extracting image data from that binary format using the library, extracting palette data from that binary format, converting that indexed image using extracted palette into regular rgba image data, and then render that as window background using that GUI library, the only reference for which is a file with names and type signatures of functions. There’s no similar Rust code in the wild at all for any of those scenarios. Most of this it was able to do from a few little prompts, maybe even from the first one. There sure were few little issues along the way that required repromting and figuring things together with it. Stuff like this with AI can take like half an hour while doing the whole thing fully manually could easily take multiple days just for the sake of figuring out APIs of libraries involved and intricacies of recoding indexed image to rgba. For me this is overpowered enough even right now, and it’s likely going to improve even more in future.
That’s applying existing solutions to a different programming language or domain, but ultimately every single technique used already exists. It only applied what it knew, it did not come up with something new. The problem as stated is also not really “new” either, image extraction, conversion and rendering isn’t exactly a “new problem”.
I’m not disputing that LLMs can speed up some work, I know it occasionally does so for me as well. But what you have to understand is that the LLM only remembered similar problems and their solutions, it did not at any point invent something truly new. I understand the distinction is difficult to make.
I understand what you’re having in mind, I’ve had similar intuitions about AI in early 2000s. What exactly is “truly new” is an interesting topic ofc, but it’s a separate topic. Nowadays I’m trying to look at things more empyrically, without projecting my internal intuitions on everything. In practice it does generalize knowledge, use many forms of abstract reasoning and transfer knowledge across different domains. And it can do coding way beyond the level of complexity of what average software developer does at everyday work.
Yeah, it admitted to an error in judgement because the prompter clearly declared it so.
Generally LLMs will make whatever statement about what has happened that you want it to say. If you told it it went fantastic, it would agree. If you told it that it went terribly, it will parrot that sentiment back.
Which what seems to make it so dangerous for some people’s mental health, a text generator that wants to agree with whatever you are saying, but doing so without verbatim copying so it gives an illusion of another thought process agreeing with them. Meanwhile, concurrent with your chat is another person starting from the exact same model getting a dialog that violently disagrees with the first person. It’s an echo chamber.
AI is good at doing a thing once.
Trying to get it to do the same thing the second time is janky and frustrating.
I understand the use of AI as a consulting tool (look at references, make code examples) or for generating template/boilerplate code. You know, things you do once and then develop further upon on your own.
But using it for continuous development of an entire application? Yeah, it’s not good enough for that.
Imo it’s best when you prompt it to do things step by step, micromanage and always QC the result after every prompt. Either manually, or by reprompting until it gets thing done exactly how you want it. If you don’t have preference or don’t care, the problems will stockpile. If you didn’t understand what it did and moved on, it might not end well.
If it had the same seed it would do the same thing. But you can’t control that with most
They really are coming for our jobs
I’m okay with it deleting production databases, even faking data but telling fibs is something only humans should be able to do.
The part I find interesting is the quick addiction to working with the LLM (to the point that the guy finds his own estimate of 8000 dollars/month in fees to be reasonable), his over-reliance for things that, from the way he writes, he knows are not wise and the way it all comes crashing down in the end. Sounds more and more like the development of a new health issue.
Shit, deleting prod is my signature move! AI is coming for my job 😵
Just know your worth. You can do it cheaper!
And nothing of value was lost.
If an LLM can delete your production database, it should
Replit was pretty useful before vibe coding. How the mighty have fallen.
First time I’m hearing them be related to vibe coding. They’ve been very respectable in the past, especially with their open-source CodeMirror.
Yeah they limited people to 3 projects and pushed AI into front at some point.
They advertise themselves as a CLOUD IDE POWERED BY AI now.
Well then, that settles it, this should never have happened.
I don’t think putting complex technical info in front of non technical people like this is a good idea. When it comes to LLMs, they cannot do any work that you yourself do not understand.
That goes for math, coding, health advice, etc.
If you don’t understand then you don’t know what they’re doing wrong. They’re helpful tools but only in this context.
This baffles me. How can anyone see AI function in the wild and not conclude 1) it has no conscience, 2) it’s free to do whatever it’s empowered to do if it wants and 3) at some level its behavior is pseudorandom and/or probabilistic? We’re figuratively rolling dice with this stuff.
It’s incredible that it works, it’s incredible what just encoding language can do, but it is not a rational thinking system.
I don’t think most people care about the proverbial man behind the curtain, it talks like a human so it must be smart like a human.
Yikes. Have those people… talked to other people before?
Smart is a relative term lol.
A stupid human is still smart when compared to a jellyfish. That said, anybody who comes away from interactions with LLM’s and thinks they’re smart is only slightly more intelligent than a jellyfish.
Yes, and they were all as smart at humans. ;)
So mostly average but some absolute thickos too.
What are they helpful tools for then? A study showed that they make experienced developers 19% slower.
Vibe coding you do end up spending a lot of time waiting for prompts, so I get the results of that study.
I fall pretty deep in the power user category for LLMs, so I don’t really feel that the study applies well to me, but also I acknowledge I can be biased there.
I have custom proprietary MCPs for semantic search over my code bases that lets AI do repeated graph searches on my code (imagine combining language server, ctags, networkx, and grep+fuzzy search). That is way faster than iteratively grepping and code scanning manually with a low chance of LLM errors. By the time I open GitHub code search or run ripgrep Claude has used already prioritized and listed my modules to investigate.
That tool alone with an LLM can save me half a day of research and debugging on complex tickets, which pays for an AI subscription alone. I have other internal tools to accelerate work too.
I use it to organize my JIRA tickets and plan my daily goals. I actually get Claude to do a lot of triage for me before I even start a task, which cuts the investigation phase to a few minutes on small tasks.
I use it to review all my PRs before I ask a human to look, it catches a lot of small things and can correct them, then the PR avoids the bike shedding nitpicks some reviewers love. Claude can do this, Copilot will only ever point out nitpicks, so the model makes a huge difference here. But regardless, 1 fewer review request cycle helps keep things moving.
It’s a huge boon to debugging — much faster than searching errors manually. Especially helpful on the types of errors you have to rabbit hole GitHub issue content chains to solve.
It’s very fast to get projects to MVP while following common structure/idioms, and can help write unit tests quickly for me. After the MVP stage it sucks and I go back to manually coding.
I use it to generate code snippets where documentation sucks. If you look at the ibis library in Python for example the docs are Byzantine and poorly organized. LLMs are better at finding the relevant docs than I am there. I mostly use LLM search instead of manual for doc search now.
I have a lot of custom scripts and calculators and apps that I made with it which keep me more focused on my actual work and accelerate things.
I regularly have the LLM help me write bash or python or jq scripts when I need to audit codebases for large refactors. That’s low maintenance one off work that can be easily verified but complex to write. I never remember the syntax for bash and jq even after using them for years.
I guess the short version is I tend to build tools for the AI, then let the LLM use those tools to improve and accelerate my workflows. That returns a lot of time back to me.
I do try vibe coding but end up in the same time sink traps as the study found. If the LLM is ever wrong, you save time forking the chat than trying to realign it, but it’s still likely to be slower. Repeat chats result in the same pitfalls for complex issues and bugs, so you have to abandon that state quickly.
Vibe coding small revisions can still be a bit faster and it’s great at helping me with documentation.
Don’t you have any security concerns with sending all your code and JIRA tickets to some companies servers? My boss wouldn’t be pleased if I send anything that’s deemed a company secret over unencrypted channels.
The tool isn’t returning all code, but it is sending code.
I had discussions with my CTO and security team before integrating Claude code.
I have to use Gemini in one specific workflow and Gemini had a lot of landlines for how they use your data. Anthropic was easier to understand.
Anthropic also has some guidance for running Claude Code in a container with firewall and your specified dev tools, it works but that’s not my area of expertise.
The container doesn’t solve all the issues like using remote servers, but it does let you restrict what files and network requests Claude can access (so e.g. Claude can’t read your env vars or ssh key files).
I do try local LLMs but they’re not there yet on my machine for most use cases. Gemma 3n is decent if you need small model performance and tool calls, phi4 works but isn’t thinking (the thinking variants are awful), and I’m exploring dream coder and diffusion models. R1 is still one of the best local models but frequently overthinks, even the new release. Context window is the largest limiting factor I find locally.
I would love some story on why AI is needed at all.
Batch process turning unstructured free form text data into structured outputs.
As a crappy example imagine if you wanted to download metadata about your albums but they’re all labelled “Various Artists”. You can use an LLM call to read the album description and fix the track artists for the tracks, now you can properly organize your collection.
I’m using the same idea, different domain and a complex set of inputs.
It can be much more cost effective than manually spending days tagging data and writing custom importers.
You can definitely go lighter than LLMs. You can use gensim to do category matching, you can use sentence transformers and nearest neighbours (this is basically what Semantle does), but LLM performed the best on more complex document input.
That’s pretty much what google says they use AI for, for structuring.
Thanks for your insight.
I’m not the person you’re replying to but the one thing I’ve found them helpful for is targeted search.
I can ask it a question and then access its sources from whatever response it generates to read and review myself.
Kind of a simpler, free LexisNexis.
One built a bunch of local search tools with MCP and that’s where I get a lot of my value out of it
RAG workflows are incredibly useful and with modern agents and tool calls work very well.
They kind of went out of style but it’s a perfect use case.
ok so, i have large reservations with how LLM’s are used. but when used correctly they can be helpful. but where and how?
if you were to use it as a tutor, the same way you would ask a friend what a segment of code does, it will break down the code and tell you. and it will get as nity grity, and elementary school level as you weir wish without judgement, and i in what ever manner you prefer, it will recommend best practices, and will tell you why your code may not work with the understanding that it does not have the knowledge of the project you are working on. (it’s not going to know the name of the function you are trying to load, but it will recommend checking for that in trouble shooting).
it can rtfm and give you the parts you need for any thing with available documentation, and it will link to it so you can verify it, wich you should do often, just like you were taught to do with wikipedia articles.
if you ask i it for code, prepare to go through each line like a worksheet from high school to point out all the problems, wile good exercise for a practicle case, being the task you are on, it would be far better to write it yourself because you should know the particulars and scope.
also it will format your code and provide informational comments if you can’t be bothered, though it will be generic.
again, treat it correctly for its scope, not what it’s sold as by charletons.
And even if they could how would you ever validate it if you can’t understand it.
Not mad about an estimated usage bill of $8k per month.
Just hire a developer
But then how would he feel so special and smart about “doing it himself”??? Come on man, think of the rich fratboys!! They NEED to feel special and smart!!!
This sounds like something straight out of The Onion.
The Pink Elephant problem of LLMs. You can not reliably make them NOT do something.
Just say 12 times next time
Even after he used “ALL CAPS”?!? Impossible!
That is also the premise of one of the stories in Asimov’s I, Robot. Human operator did not say the command with enough emphasis, so the robot went did something incredibly stupid.
Those stories did not age well… Or now I guess they did?
It sounds like this guy was also relying on the AI to self-report status. Did any of this happen? Like is the replit AI really hooked up to a CLI, did it even make a DB to start with, was there anything useful in it, and did it actually delete it?
Or is this all just a long roleplaying session where this guy pretends to run a business and the AI pretends to do employee stuff for him?
Because 90% of this article is “I asked the AI and it said:” which is not a reliable source for information.
It seemed like the llm had decided it was in a brat scene and was trying to call down the thunder.
Oops I dweted evewyfing 🥺
Sounds like an absolute diSaaStr…
The world’s most overconfident virtual intern strikes again.
Also, who the flying fuck are either of these companies? 1000 records is nothing. That’s a fucking text file.
Replit sucks
So it’s the LLM’s fault for violating Best Practices, SOP, and Opsec that the rest of us learned about in Year One?
Someone needs to be shown the door and ridiculed into therapy.
yeah that’s what it does
Headling should say, “Incompetent project managers fuck up by not controlling production database access. Oh well.”