There’s some real perks to using AI to code - it helps a ton with templatable or repetitive code, and setting up tedious tasks. I hate doing that stuff by hand so being able to pass it off to copilot is great. But we already had tools that gave us 90% of the functionality copilot adds there, so it’s not super novel, and I’ve never had it handle anything properly complicated at all successfully (asking GPT-5 to do your dynamic SQL calls is inviting disaster, for example. Requires hours of reworking just to get close.)

In the beginning there were manufacturer’s manuals, spec sheets, etc.

Then there were magazines, like Byte, InfoWorld, Compute! that showed you a bit more than just the specs

Then there were books, including the X for Dummies series that purported to teach you theory and practice

Then there was Google / Stack Overflow and friends

Somewhere along there, where depends a lot on your age, there were school / University courses

Now we have “AI mode”

Each step along that road has offered a significant speedup, connecting ideas to theory to practice.

I agree, all the “magic bullet” AI hype is far overblown. However, with AI something I new I can do is, interactively, develop a specification and a program. Throw out the code several times while the spec gets refined, re-implemented, tried in different languages with different libraries. It’s still only good for “small” projects, but less than a year ago “small” meant less than 1000 lines of code. These days I’m seeing 300 lines of specification turn into 1500-3000 lines of code and have it running successfully within half a day.

I don’t know if we’re going to face a Kurzweilian singularity where these things start improving themselves at exponential rates, or if we’ll hit another 30 year plateau like neural nets did back in the 1990s… As things are, Claude helps me make small projects several times faster than I could ever do with Google and Stack Overflow. And you can build significant systems out of cooperating small projects.

I found that it only does well if the task is already well covered by the usual sources. Ask for anything novel and it shits the bed.

At absolute best.

My experience is it’s the bottom stack overflow answers. Making up bullshit and nonexistent commands, etc.

At least when I’m cleaning up after shit devs who used Stack Overflow, I can usually search using a fragment of their code and find where they swiped it from and get some clue what the hell they were thinking. Now that they’re all using AI chatbots, there’s no trace.

Except LLMs are absolutely terrible at working with a new, poorly documented library. Commonly-used, well-defined libraries? Sure! Working in an obscure language or an obscure framework? Good luck.

LLMs can surface information. It’s perhaps the one place they’re actually useful. They cannot reason in the same way a human programmer can, and all the big tech companies are trying to sell them on that basis.

I’m honestly doubting it will get there someday, at least with the current use of LLMs. There just isn’t true comprehension in them, no space for consideration in any novel dimension. If it takes incredible resources for companies to achieve sometimes-kinda-not-dogshit, I think we might need a new paradigm.

Were you trying to say “drivel?”

How are you interpreting it that way. Did you miss a sentence or something in the quote?

Nowhere close to any junior ime. Grads learn very quickly. Interns only job is to understand. Code academy career switchers understand requirements and will ask questions. Subservient AI does fuck all of any of those things

They are more akin to yet another Rapid Application Development wave imo. Go see how the previous iterations have done. Lots are still with us (rails ftw!). I’ll bet most will outlive LLMs

Even that description is vastly overselling it’s usefulness. Every time someone says it’s like a junior dev I just sigh, because literally the only reason I like junior devs is because they turn into not junior devs. Never once has assigning something to a junior dev made my job easier. The entire goal is to train them to the point they make PRs that I don’t have to walk them through reworking.

It amplifies a person. If you smart, it will help. If you dumb, it will make you dumber.

GPT has been quite hit and miss for me, but Claude is usually quite solid.

It needs micromanaging, otherwise it will do bad design decisions and go off on unrelated side quests. When micromanaged it’ll get you to that MVP very fast.

The trap is that you need to be able to find the errors it makes, or at least call them out immediately. Trying to have co-pilot fix it’s own mistakes is usually a neverending prompt-cycle.

It can summarise big code bases fast, and find how things fit together a lot faster than me. It’s been very useful when being thrown in head first into a new project.

Claude Sonnet? so you enjoy being lied to on a daily basis because that’s all you’re getting from Claude. Claude Code is alright but using Claude.ai you might as well just throw a dart at wall to get a solution on something.

I personally think unit tests are the worst application of AI. Tests are there to ensure the code is correct, so ideally the dev would write the tests to verify that the AI-generated code is correct.

I personally don’t use AI to write code, since writing code is the easiest and quickest part of my job. I instead use it to generate examples of using a new library, give me comparisons of different options, etc, and then I write the code after that. Basically, I use it as a replacement for a search engine/blog posts.

@patch.object(Object, "method", return_value=value)

@parameterized.expand([
  (key, value, Expected Exception,),
  (other_key, other_value, OtherExpectedException,), 
])
def test_exceptions(self, key, value, exception_class):
    obj = setup()
    setattr(obj, key, value) 

    with self.assertRaises(exception_class): 
        func_to_test(obj)

@unittest.mock.patch.object(Class, "method", return_value=...)

dynamic_mock =  MagicMock(Class)
dynamic_mock...

One of the guys at my old job submitted a PR with tests that basically just mocked everything, tested nothing. Like,

with patch("something.whatever", return_value=True):
  assert whatever(0) is True
  assert whatever(1) is True

Except for a few dozen lines, with names that made it look like they were doing useful.

He used AI to generate them, of course. Pretty useless.

I’ve seen it generate working unit tests plenty. In the sense that they pass.

…they do not actually test the functionality.
Of course that function returns what you’re asserting - you overwrote its actual output and checked against that!

No duh, says a layman who never wrote code in his life.

If not to editorialize, what else is the text box for? :)

Look on the bright side, in a couple of years they will come crawling back to us, desperate for new things to be built so their profit machines keep profiting.

Current ML techniques literally cannot replace developers for anything but the most rudimentary of tasks.

I wish we had true apprenticeships out there for development and other tech roles.

Yeah, code bloat with LLMs is fucking monstrous. If you use them, get used to immediately scouring your code for duplications.

Actually typing out code has literally never been the bottleneck. It’s a vanishingly small amount of what we do. An experienced engineer can type out bash or Python scripts without so much as blinking. And better yet, they can do it without completely fabricating commands and library functions.

The hard part is truly understanding what it is you’re trying to do in the first place, and that fundamentally requires a level of semantic comprehension that LLMs do not in any way possess.

It’s very much like the “no code” solutions of yesteryear. They sound great on paper until you’re faced with the reality of the buggy, unmaintainable nightmare pile of spaghetti code that they vomit into your repo.

LLMs are truly a complete joke for software development tasks. I remain among the top 3-4 developers in terms of speed and output at my workplace (and all of the fastest people refuse to use LLMs as well), and I don’t create MRs chock full of bullshit that has to be ripped out (fucking sick of telling people to delete absolutely useless tests that do nothing but slow down our CI pipeline). The slowest people are those that keep banging their head against the LLM for “efficiency” when it’s anything but.

It’s the fucking stupidest trend I’ve seen in my career and I can’t wait until people finally wake up and realize it’s both incredibly inefficient and incredibly wasteful.

Oh god, please don't use it for Bash. LLM-generated Bash is such a fucking pot of horse shit bad practices. Regular people have a hard enough time writing good Bash, and something trained on all the fucking crap on StackOverflow and GitHub is inevitably going to be so bad...

Signed, a senior dev who is the "Bash guy" for a very large team.

I’ve found it’s pretty good at refactoring existing code to use a different but well-supported and well documented library. It’s absolutely terrible for a new and poorly documented library.

I recently tried using Copilot with Claude to implement something in a fairly young library, and did get the basics working, including a long repetitive string of “that doesn’t work, I’m getting error msg [error]”. Seven times of that, and suddenly it worked! I was quite amazed, though it failed me in many other ways with that library (imagining functions and options that don’t exist). But then redoing the same thing in the older, better supported library, it got it right on the first try.

But maybe the biggest advantage of AI coding is that it allows me to code when my brain isn’t fully engaged. Of course the risk there is that my brain might not fully engage because of the AI.

I thought that as well and got some code from someone that left the company and asked it to comment it.

It did the obvious “x= 5 // assign 5 to x” crap comments and then it got to the actually confusing part and just skipped that mess entirely…

It is not worthless. My understanding is that management only trusts sources that are expensive.

A VP and his taking head sycophants at my work has not shut up about it. They went through some trouble to automatically measure employee use of the AI and made it a performance measure. So now we have it generate code so we don’t get fired and mostly throw it away.

I don’t think it’s meant to be a conclusion. The article serves as a recap of several reports and studies about the effectivity of LLMs with coding, and the final quote from Bain & Company was a counterpoint to the previous ones asserting that productivity gains are minimal at best, but also that measuring productivity is a grey area.

Sounds like vibecoders will have to relearn the lessons of the past 40 years of software engineering.

Wait, are you blaming AI for this, or yourself?

More code is better, obviously! Why else would a website to see a restaurant menu be 80Mb? It’s all that good, excellent code.

It did solve my impostor syndrome though. Turns out a bunch of people I saw to be my betters were faking it all along.

I sometimes get up to five lines of viable code. Then upon occasion what should have been a one liner tries to vomit all over my codebase. The best feature about AI enabled IDE is the button to decline the mess that was just inflicted.

In the past week I had two cases I thought would be “vibe coding” fodder, blazingly obvious just tedious. One time it just totally screwed up and I had to scrap it all. The other one generated about 4 functions in one go and was salvageable, though still off in weird ways. One of those was functional, just nonsensical. It had a function to check whether a certain condition was present or not, but instead of returning a boolean, it passed a pointer to a string and set the string to “” to indicate false… So damn bizarre, hard to follow and needlessly more lines of code, which is another theme of LLM generated code.

On proprietary products, they are awful. So many hallucinations that waste hours. A manager used one on a code review of mine and only admitted it after I spent the afternoon chasing it.

Based on my experience with claude sonnet and gpt4/5… It’s a little useful but generally annoying and fails more often than works.

I do think moderate use still comes out ahead, as it saves a bunch of typing when it does work, but I still get annoyed at the blatantly stupid suggestions I keep having to decline.

Oh yes. The Great pathlib. The Blessed pathlib. Hallowed be it and all it does.

I’m a Ruby girl. A couple of years ago I was super worried about my decision to finally start learning Python seriously. But once I ran into pathlib, I knew for sure that everything will be fine. Take an everyday headache problem. Solve it forever. Boom. This is how standard libraries should be designed.

If it wasn’t for all the AI hype that it’s going to do everyone’s job, LLMs would be widely considered an amazing advancement in computer-human interaction and human assistance. They are so much better than using a search engine to parse web forums and stack overflow, but that’s not going to pay for investing hundreds of billions into building them out. My experience is like yours - I use AI chat as a huge information index mainly, and helpful sounding board occasionally, but it isn’t much good beyond that.

I will concur with the whole ‘llm keeps suggesting to reinvent the wheel’

And poorly. Not only did it not use a pretty basic standard library to do something, it’s implementation is generally crap. For example it offered up a solution that was hard coded to IPv4, and the context was very ipv6 heavy

I mean largely for most of us I hope. But I feel like the tech sector was oversatured because of all the hype of it being an easy get rich quick job. Which for some people it was.

There already are. People all over LinkedIn are changing their titles to “AI Code Cleanup Specialist”.

Only for taxpayers ☝️

I am a software architect, an mainly usw it to refactor my own old code… But i am maybe not a typical architect…

Software architects that don’t write code are worse than useless

How is it not correct if the code successfully does the very thing that was prompted?

F.ex. in my company we don’t have any real programmers but have built handful of useful tools (approx. 400-1600 LOC, mainly Python) to do some data analysis, regex stuff to cleanup some output files, index some files and analyze/check their contents for certain mistakes, dashboards to display certain data, etc.

Of course the apps may not have been perfect after the very first prompt, or even compiled, but after iterating an error or two, and explaining an edge case or two, they’ve started to perform flawlessly, saving tons of work hours per week. So how is this not useful? If the code creates results that are correct, doesn’t that make the app itself technically ”correct” too, albeit likely not nearly as optimized as equivalent human code would be.

I’ve seen and used AI for snippets of code and it’s pretty decent at that.

With my colleagues I always compare it to a battery powered drill. It’s very powerful and can make shit a lot easier. But you’d not try to build furniture from scratch with only a battery powered drill.

You need the knowledge to use it - and also saws, screws, the proper bits for those screws and so on and so forth.

Eh I had it write a program that finds my PCs ip and sends it to the Unifi gateway to change a rule. Worked fine but I guess technically it is mostly using the go libraries written by someone else.

I’ve used Claude code to fix some bugs and add some new features to some of my old, small programs and websites. Not things I can’t do myself, but things I can’t be arsed to sit down and actually do.

It’s actually gone really well, with clean and solid code. easily readable, correct, with error handling and even comments explaining things. It even took a gui stream processing program I had and wrote a server / webapp with the same functionality, and was able to extend it with a few new features I’ve been thinking to add.

These are not complex things, but a few of them were 20+ files big, and it manage to not only navigate the code, but understand it well enough to add features with the changes touching multiple files (model, logic, view layer for example, or refactor a too big class and update all references to use the new classes).

So it’s absolutely useful and capable of writing good code.

for me it typically don’t cause syntax errors, but the main thing it fucks up is what you specifically told them to do, where the output straight up does not perform the way your specification requires. If it’s just some syntax errors at least the compiler can catch them, this you won’t even know if you don’t bother testing the output.

AI is works well for mindless tasks. Data formatting, rough drafts, etc.

Once a task requires context and abstract thinking, AI can’t handle it.

I think it’s useful for writing mundane snippets I’ve written a million times or helping me with languages I’m less familiar with, but anything more compex becomes pretty spaghetti pretty quick.

When the AI bubble bursts, even janitors and nurses will lose their jobs. Financial institutions will go bust.

From what I’ve seen and heard, there are a few factors to this.

One is that the tech industry right now is built on venture capital. In order to survive, they need to act like they’re at the forefront of the Next Big Thing in order to keep bringing investment money in.

Another is that LLMs are uniquely suited to extending the honeymoon period.

The initial impression you get from an LLM chatbot is significant. This is a chatbot that actually talks like a person. A VC mogul sitting down to have a conversation with ChatGPT, when it was new, was a mind-blowing experience. This is a computer program that, at first blush, appears to be able to do most things humans can do, as long as those things primarily consist of reading things and typing things out - which a VC, and mid/upper management, does a lot of. This gives the impression that AI is capable of automating a lot of things that previously needed a live, thinking person - which means a lot of savings for companies who can shed expensive knowledge workers.

The problem is that the limits of LLMs are STILL poorly understood by most people. Despite constructing huge data centers and gobbling up vast amounts of electricity, LLMs still are bad at actually being reliable. This makes LLMs worse at practically any knowledge work than the lowest, greenest intern - because at least the intern can be taught to say they don’t know something instead of feeding you BS.

It was also assumed that bigger, hungrier LLMs would provide better results. Although they do, the gains are getting harder and harder to reach. There needs to be an efficiency breakthrough (and a training breakthrough) before the wonderful world of AI can actually come to pass because as it stands, prompts are still getting more expensive to run for higher-quality results. It took a while to make that discovery, so the hype train was able to continue to build steam for the last couple years.

Now, tech companies are doing their level best to hide these shortcomings from their customers (and possibly even themselves). The longer they keep the wool over everyone’s eyes, the more money continues to roll in. So, the bubble keeps building.

AI is a financial scam. Basically companies that are already mature promise great future profits thanks to this new technological miracle, which makes their stock more valuable than it otherwise would be. Cory Doctorow has written eloquently about this.

This struck upon one of the greatest wishes of all corporations. A way to get work without having to pay people for it.

I think right now companies are competing until they're only 1 or 2 that clearly own the majority of the market.

Afterwards they will devolve back into the same thing search engines are now. A cesspool of sponsored ads and links to useless SEO blogs.

They'll just become gate keepers of information again and the only ones that will be heard are the ones who pay a fee or game the system.

Maybe not though, I'm usually pretty cynical when it comes to what the incentives of businesses are.

I mean, originally they thought they had come upon a magic bullet. Turns out it wasn’t the case, and now they’re going to suffer for it.

Lmao calling nobility would imply we can’t vote our senior management and it often ends up being whoever “the king” wants or one of king’s children.
Wait…

From my experience, AI just seems to be a lesson in overfitment. You can’t use it to do things nobody has done before. Furthermore, you only really get good responses from prompts related to Javascript

I have 3 questions, and I’m coming from a heavily AI-skeptic position, but am open:

1. Do you believe that providing all that context, describing the existing patterns, creating an implementation plan, etc, allows the AI to both write better code and faster than if you just did it yourself? To me, this just seems like you have to re-write your technical documentation in prose each time you want to do something. You are saying this is better than ‘Do XYZ’, but how much twiddling of your existing codebase do you need to do before an AI can understand the business context of it? I don’t currently do development on an existing codebase, but every time I try to get these tools to do something fairly simple from scratch, they just flail. Maybe I’m just not spending the hours to build my AI-parsable functional spec. Every time I’ve tried this, asking something as simple as (and paraphrased for brevity) “write an Asteroids clone using JavaScript and HTML 5 Canvas” results in a full failure, even with multiple retries chasing errors. I wrote something like that a few years ago to learn Javascript and it took me a day-ish to get something that mostly worked.

2. Speaking of that context. Are you running your models locally, or do you have some cloud service? If you give your entire codebase to a 3rd party as context, how much of your company’s secret sauce have you disclosed? I’d imagine most sane companies are doing something to make their models local, but we see regular news articles about how ChatGPT is training on user input and leaking sensitive data if you ask it nicely and I can’t imagine all the pro-AI CEOs are aware of the risks here.

3. How much pen-testing time are you spending on this code, error handling, edge cases, race conditions, data sanitation? An experienced dev understands these things innately, having fixed these kinds of issues in the past and knows the anti-patterns and how to avoid them. In all seriousness, I think this is going to be the thing that actually kills AI vibe coding, but it won’t be fast enough. There will be tons of new exploits in what used to be solidly safe places. Your new web front-end? It has a really simple SQL injection attack. Your phone app? You can tell it your username is admin’joe@google.com and it’ll let you order stuff for free since you’re an admin.

I see a place for AI-generated code, for instant functions that do something blending simple and complex. “Hey claude, write a function to take a string and split it at the end of every sentence containing an uppercase A”. I had to write weird functions like that constantly as a sysadmin, and transforming data seems like a thing an AI could help me accelerate. I just don’t see that working on a larger scale, though, or trusting an AI enough to allow it to integrate a new function like that into an existing codebase.

Yea, I use it for home assistant, it’s amazingly powerful… And so incredibly dumb

It will take my if and statements, and shrunk it to 1/3 the length, while being twice as to robust… While missing that one of the arguments is entirely in the wrong place.

It regurgitates old code, it cannot come up with new stuff.

The trick is, most of what you write is basically old code in new wrapping. In most projects, I’d say the new and novel part is maybe 10% of the code. The rest is things like setting up db models, connecting them to base logic, set up views, api endpoints, decoding the message on the ui part, displaying it to user, handling input back, threading things so UI doesn’t hang, error handling, input data verification, basic unit tests, set up settings, support reading them from a file or env vars, making UI look not horrible, add translatable text, and so on and on and on. All that has been written in some variation a million times before. All can be written (and verified) by a half-asleep competent coder.

The actual new interesting part is gonna be a small small percentage of the total code.

This is a sane way to use LLM. Also, pick your poison, some bots are better than others for a specific task. It’s kinda fascinating to see how other people solve coding problems and that is essentially on tap with a bot, it will churn out as many examples as you want. It’s a really useful tool for learning syntax and libraries of unfamiliar languages.

On one extreme side of LLM there is this insane hype and at the other extreme a great pessimism but in the middle is a nice labour saving educational tool.

Writing new code is easier than editing someone else’s code but editing a portion is still better than writing the entire program again from start to end.

Then there is LLMs which force you to edit the entire thing from start to end.