At least in my experience, it excels in blank canvas projects. Where you've got nothing and want something pretty basic. The tools can probably set up a fresh React project faster than me. But at least every time I've tried them on an actual work repo they get reduced to almost useless.

Which is why they generate so much hype. They are perfect for tech demos, then management wonders why they aren't seeing results in the real world.

Exactly. It quickly builds a lot of technical debt that must be paid down, especially for people writing code in areas they aren't deep in.

For tight tasks it can be super helpful -- like for me, an AI/Data Science guy, setting up a basic reverse proxy. But I do so with a ton of scrutiny -- pushing it, searching on Kagi or docs to at least confirm the code, etc. This is helpful because I don't have a mental map about reverse proxy -- so it can help fill in gaps but only with a lot of reticence.

That type of use really doesn't justify the billion dollar valuations of any companies, IMO.

What do you mean by you don't have a mental map about a reverse proxy?

That I don't have a deep understanding about nginx and how it's many options fit together across an OS with confidence that I made it secure, accurate, and/or fast. Give me python, Matlab, rust, I can put something together, but something like nginx and I've simply never dived deep enough for a solid understanding before using an LLM to understand more.

Even scaffolding a new project is not easy work, especially a new stack or new versions of existing tools. For example, I have never been able to create a Vue 3 project with Vite and Tailwind setup correctly. I tried top SOTA models. Maybe my prompting skills are not good, but everytime it fails to set up a project correctly. Everytime it gives me some old configurations that's not relevant anymore.

LLMs are probably the worst tool for the job. Code generators have been a thing forever. Why use a LLM when you can do "npm create vite@latest my-vue-app -- --template vue" ?

It's always more tedious than that. You have to pick all these libraries, install and set them up, build login pages, etc. Stuff that is all simple work but takes ages. I've never used an LLM for it but it seems like the kind of work that should be easy enough to automate and would save a week of setting everything up if it worked.

Why though? Vite supplies a project scaffolder and lists a one liner under getting started;

ie pnpm create vite

Tailwind is similarly a one liner to initialize(might be a vite create option now).

Edit: My bad, you are talking about the LLMs! I'm always surprised how still for past years, even though we have great projects scalfolding across the node verse, people are still complaining about how hard setting up projects is..

The reason LLMs suck in plenty of brownfield projects is because those codebases likely either implemented frameworks in a proprietary way, maybe did not rely on any public framework at all, or were in general done in an esoteric way and therefor few (if any) similar codebases exist within the LLMs training data. Which is problematic because LLMs aren't capable of reasoning or learning they're literally just predicting the next most likely token in a chain similarly to how autocomplete works. Without you supplying additional context and explicitly defining guardrails for preforming common tasks the LLM has no frame of reference for working with your codebase.

I've had great success with GPT5 in existing projects because its agent mode is very good (the best I've seen so far) at analyzing the existing codebase and then writing code that feels like it fits in already (without prompt engineering on my part). I still agree that AI is particularly good on fresh projects though.

Could be that there is a huge difference in the products. Last few companies have given me Github Copilot which I find entirely useless, I found the automatic suggestions more distracting than useful, and the fix and explain functions never work. But maybe if you burn $1000/day on Claude Code it works a lot better. And then companies see the results from that and wonder why they aren't getting it spending a couple of dollars on Copilot.

I use GitHub Copilot from work in agent mode with GPT5 and it’s great! I don’t use the suggestions, fix, or explain features, I agree they’re almost always not helpful.

Is that a separate product you have to pay for? I've never seen an agent feature in the VS code plugin.

It's there next to the model selector in chat.

I actually completely disagree with this, and IMO it works best with projects that are templated with AI development in mind. Lots of documentation and comments, working tests, etc.

You want as much context as possible _right in the code_.

By "knowing what you're doing" do you mean "have enough experience to it by hand", "have experience with a specific AI tool and its limitations" or a combination?

You don't need sotware engineering to build successful software, until you do.

In my experience you don't need to know a whole lot about LLM's to work them. You need to know that everything they spit out is potential garbage, and if you can't tell the good from the garbage then whatever you're using them for is going to be terrible. In terms of software terrible is fine for quite a lot of systems. One of the first things I build out of university in the previous millennium is still in production today and it's horrible. It's inefficient, horribly outdated since it hasn't been updated ever. It runs 10 times a day and at least 1 of them will need to automatically restart itself because it failed. It's done it's job without the need for human intervention for the past many decades though. I know because one of my old colleagues still works there. It could've been improved, but the inefficiency cost over all those years is probably worth about two human hours, and it would likely take quite a while to change it. A lot of software is like that, though a lot of it doesn't live for so long. LLM's can absolutely blast that sort of thing. It's when the inefficiency cost isn't less than a few human hours that LLM's become a liability if you don't know how to do the engineering.

I use LLM's to write a lot of the infrastructure as code we use today. I can do that because I know exactly how that should be engineered. What the LLM can do that I can't, is that it can spit out the k8s yaml for an ingress point with 200 lines of port settings in a couple of seconds. I've yet to have it fail, probably because those configurations are basically all the same depending on the service. What a LLM can't do, however, is write the entire yaml config.

Similarily it can build you a virtual network with subnets in bicep based on a couple of lines of text with address prefixes. At the sametime it couldn't build you a reasonable vnet with subnets if you asked it to do it from scractch. That doesn't mean it can't build you one that works though, it's just that you're likely going to claim 65534 ip addresses for a service which uses three.

The first one.

I mean the truth should be fairly obvious to people given a lot of the talk around AI stuff rings very much like the ifls/mainstream media style "science" articles which always make some outrageous "right around the corner" claim based off some small tidbit out of a paper they only skimmed the abstract of.