I don't get the obsession with more code. Whatever happened to good development practices? Many of which, if followed allow LLMs to work more efficiently ironically enough. I would rather hear that you've fixed bugs that have stumped your dev team for years, and have had next to no outages, to me that's more valuable than just slopping features out the door.
While I agree with your sentiment, very strongly actually, there has always been a tension between what I'll call "engineering teams" and "product teams". The engineering teams want the long-term platform stability, performance, etc. The product teams want to push that new feature out, quick! The overall business needs both (types of thinking and doing) to survive, of course, but not all leadership sees the world through the lens engineering team lens. I'd argue most actually lean the other way, which is understandable though unfortunate. I'd also posit that since these product teams are often "closer to the business", those people typically win favor with the real decision makers.
This tension is why I refer to myself as a product engineer. Teams where non-technical product managers (ie. your "product team") become feature factories that eventually drown in their own tech debt. Teams where navel-gazing engineering managers (ie. your "engineering team") have too much power end up with premature platformization, and beautiful architecture diagrams while user needs go unmet.
The key is to understand that computers exist to serve human needs, and the less code / UI / hoops to solve a problem the better, and to think end-to-end. If you are able to do that, most systems built by committee have a lot of low-hanging fruit that can be improved without much sacrifice. In practice, political realities and operational concerns may come in the way, but I firmly believe this is the only way to successfully operate as an engineer, and it will only become more important in the age of AI.
And I feel that stuff like agile is now even more important than before. Now it’s role should be more about protecting customers from bad changes and following whole processes through where you actually measure the impact. It’s just too easy to do those 100k lines changes to products because it’s some vice presidents pet project.
If you build your platform well, then you can execute fast, but also safe.
I push "money and data are sacred"; if we lose either of those, someone cries. Build the platform slow and fast. Governance, scaling etc sits here.
Provide reference APIs and let the product team one shot whatever they want and need, build fast and experimental, but your platform enforces constraints.
Best of both worlds (though feels like we're back in backend vs frontend development)
>If you build your platform well, then you can execute fast, but also safe.
this is extremely true, but it seems less and less like anyone wants to invest in a solid platform and just wants to jump to executing fast and unsafe. and then pretends that ai can definitely do both when the experience is quite a bit different.
I have seen at least one example develop in real time. You typically spent years building before you had a platform, then lack the enthusiasm from senior leadership to invest the next 12-18 months with no new features and hard to quantify - maybe no - returns. Then you get displaced by newcomers who are "platform native"; what this really means is their engineering misteps haven't been discovered yet, or even if they guessed correctly the world has turned and made their previously correct solution now a failing one. And now we have something called "AI-native" that is going to solve this once and for all with some sort of magic, and you can't lead these companies unless you truly believe in the magic.
“ You typically spent years building before you had a platform, then lack the enthusiasm from senior leadership to invest the next 12-18 months with no new features and hard to quantify - maybe no - returns.” —> this is failure from the team and especially people who should do product management job on that team.
I'm lucky I guess, I feel like the product team is on "my side" with respect to building a resilient system. They understand that bad architecture shows up as a fragmented and friction-full UX.
The demand for new features come from sales and executive people who want to check more boxes on a customer's checklist.
Ultimately your paycheck exists through deals that involve those boxes on checklists. It doesn't simply materialize out of thin air. If there are no deals, it doesn't matter how robust and elegant the architecture is.
Features don’t usually come from product teams. They come from above: some Head Of, VP Of, and the like. Those people only ask, they know very little about execution. They want their wishes come true in a timely manner
Management/Product/Product Engineer - When you are responsible for getting the work done then you look for any and every option to speed it up.
Owner/Platform/Senior Engineer - When you are responsible for ensuring it works and can be maintained then you have to pay attention to how to build it, aka the trade offs
I've spent most of the past two weeks trying to get my AI to stop writing so goddamned much code. It's great that I can, say, completely switch between two databases, but I wish the AIs would develop a concept of energy spend or something and spend maybe a moment checking whether something is already implemented in the code base before blasting out 2000 new lines defining a new database access layer, and also rewriting random code it shouldn't be touching for some task hundreds of lines at a time just because it can. I don't care how easy it is to write some tests that sort of kind of cover it hopefully maybe... no new code is as tested as the stuff that has been doing it in the field for years already.
Just because it has the power doesn't mean it should be used.
I'm not asking for everything to be some sort of beautiful code whose concision would lead Knuth to tears. I'm just asking that you not slather dozens of existing functions with three new parameters and dozens of lines of code each that basically do nothing, to cover a use case that I accidentally vaguely alluded to, if you read my prompt a certain way, but isn't really what I was interested in.
Right. There IS a way to convince ai to write less code.. but it obviously spends 10x tokens, whether it’s LLMs as judge and automated review or just write 10 solutions and keep 1. Partly ai companies like it this way but partly it’s unintentional consequences that happen to work in their favor.
Training for good taste in architecture actually causes premature optimization at high effort allowance, but at the same time, it is combined with an inability to even recognize / exploit that later if it should ever become actually useful. Dial the effort down, and it skips the reasonable abstraction and nothing is dry, factoring is insane.
Goldilocks for simple stuff barely works with frontier models after they are out of their comfort zone for topic or line limits.. you kind of just have to decide whether it’s worth it anyway. In practice.. constant supervision for high quality code, or such high quality system design that the code quality is less of a down side.
One thing I've spotted several times throughout this year: the larger volume of code leads to exactly the sort of problems it always has. It just happens real fast now, with fewer people. Problems that used to take years to erupt now manifest before an MVP ships, but people just sweep it all under the rug in the name of moving fast.
The rate at which software is produced has never been higher, nor has the rate at which it is thrown away for being broken and useless. It’s not at all obvious to me that if you control for both of those rate changes that the amount of useful software with a lifespan measured in years created per developer (ie, the software that actually has long term value) has actually changed all that much.
I drastically improved the reliability of an internal IoT device so it can tolerate all kinds of abuse (abrupt power loss during update, unreliable SD card, bootstrapping over network when the SD card is missing/damaged, fail-safe updates, recovery in a network isolated environment). This is the first release that I really leaned into AI, having been pretty skeptical before.
I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
> I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
It always works best when layering on top of good human code. The thing is, you start to run out of that as you lean in. Tricky.
Heh, you should look at the reply I left for your sibling comment. I think it is just a regression to the mean of all open source software upon which the models were trained. Sometimes the mean is an improvement.
Hell, if the Dunning–Kruger effect is to be believed, it would be bold me to assume I am capable of good human code. Maybe the average is the best I'll ever get.
> The thing is, you start to run out of that as you lean in. Tricky.
I think it depends how you lean in. I generated a few thousand lines of Rust code today. It was actually pretty high quality. On my initial review, I only identified one change I wanted to make.
Most functions were pretty short and well factored. I wonder if using Rust as a target helps, because (1) the training data doesn't include all the terrible Javascript and Python etc. found on the internet, and (2) Rust and many of its libraries encourage good architecture - code factored into traits, functional pipelines, a powerful type system, etc.
LLMs trained on bad code for mediocre languages naturally write corresponding quality code.
A big part of it still boils down to prompting - what you're asking it to do and how much freedom you give them. If you let them make all the architectural decisions, you're likely to end up with issues.
> It can be used to make existing products better and more secure. You just won't naturally hear about that.
Why? I keep asking for examples of LLM being used to make code more simple and elegant (which I would say is adjacent to making software better, if not more secure), and never get a reply, not a single one. I'm not claiming it doesn't exist, I know it must exist. If only via a lots of careful massaging of system prompts and harnesses, even if it takes too much effort to be worth it, I would like to see it.
Of course people can't share internal examples, but this is all over the world for many months now, where are the open source examples? Not "something new someone couldn't have done before", but someone making their precious even neater (while also adding things to it, but not primarily). Put differently, can LLM delete code, can they curate it well, can they not write weird comments nobody wants or needs, that sound like echoes of the user prompts and responses to them, but just explain tricky parts of the code, like a human would?
Also, IMO a VERY useful thing to learn about a tool or a chemical or whatever is, "what happens if I keep applying it to the same small thing over and over?" -- because that's the level of scientist I never evolved past, the kind of stuff you do as a kid -- , whereas a lot of this seems to be more about how it feels to apply it here and there until it looks good, and then move on and never look back. Which really isn't much at all, especially given we're such a short time in. I mean, how useful were mugs made of lead? Are people aware of the (in hindsight) insane things that were done with x-rays as those were invented, before the potential for harm was known? I bet they had a great time playing with that stuff, for a while.
What if in 5 years it turns out that a few years ago, LLM got into this habit of implementing certain things in a certain way because it seemed the best to "them" (the training, whatever), but there's a security issue with that -- what do you do? You can only let more agents loose, and hope they don't miss anything, because you cannot even do something like "patch PHP" (which is not my thought, Casey Muratori mentioned that in on of those podcast things).
Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though. It just does, like suggesting we go to Mars, because everything is better there, and to do that, we have to blow up Earth. Or saying to magically be able to fly from now on, we need to saw off the branch we're sitting on. Even if I thought life on Mars was the future or that we could fly, if we just thought of it the right way, this isn't a way forward IMO.
> LLM being used to make code more simple and elegant
Simple and elegant generally doesn't come from the LLM. It comes from me saying no to the LLM. I've discarded many branches that didn't feel like they were going the right way. Hell, I aggressively do that with my own code also. I rarely commit the first thing that works.
> where are the open source examples?
My general impression of open source is that it is, in part, a market driven by good software engineering. There's certainly below average code, but that tends to get mocked out of existence before it grows too much. Proprietary code that doesn't have a strong forcing function used to be able to hide poor engineering principles.
I think AI generated code is a regression to the mean, and the mean can be an improvement sometimes. I inherited this project. It was not originally written with AI. A lot of hard work and good engineering went into the hardware/firmware. The IoT software connecting it to our intranet, less so. Some examples that I'm fixing/fixed: ignoring HTTPS certificates, RCE by design, in-place file updating JSON files, a hand-rolled (but ineffective) fair mutex, busy waiting for a device to appear, and worst of all, no tests. None of these are hard to avoid, and even the small AI models would have come up with a better implementation.
This doesn't seem unique to my project either. I think a lot of software is just silently getting closer to the average of all open source code upon which the models were trained. I don't think I want things like SQLite to get closer to the average.
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I think you might be reading too much into what I said. When I say "fully lean into", it's probably less than you're thinking of. I still feel that I understand what was written and can stake my reputation/career on it.
You want them to make code more simple and elegant by themselves, no other direction? "Make this code simple and elegant, no mistakes, no hallucinations"? I don't think that's realistic at the moment.
But with suitable direction, they can definitely improve codebases - especially a typical enterprise codebase, which, if you don't like LLM code, you would hate with a passion. (I can only assume you've never seen such code otherwise your perspective would be different.)
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I essentially agree, but there's a huge spectrum before you get to that point, and lots of ways to use it effectively that doesn't create tech debt.
> You want them to make code more simple and elegant by themselves, no other direction?
No, he wants a project where AI is used as a major help, but the result tend towards good code, i.e readable and well abstracted. Even if the cost is high, just an example where you can extract the best practices of using AI coding.
You are responding to someone that just today generated thousands of lines of code, claims to have reviewed it all, thought it was good and only wanted one minor change. Surely they have an example of the good code it produces.
We used to understand that code was harder to read than it is to write. It follows, directly, that better engineering discipline means shipping LESS code. Less to support, less to understand, means a product that is easier to maintain and extend.
Now, token calculators spew out code at a pace no human can match, which completely removes the barrier of effort - which was one of the largest natural counters to shipping massive amounts of slop.
I'll be impressed by a coding model when it writes less code than a human, and spends more time refactoring than it does just adding new layers. Until then, this vibecoding is just madness.
1. The issue is that most people and organizations do not care about maintenance. Maintenance doesn’t get you the promotion, accolades, or immediate quarterly revenue because great maintenance tends to be invisible.
2. This is exacerbated by the lack of loyalty people have towards their organizations because their orgs don’t value or reward loyalty. That’s because leadership doesnt care about the long term view because of item #1. Hence, everyone is only doing short 1 - 4 year stints in order to get meaningful raises
3. The exceptions in the private sector are businesses that not publicly traded.
4. Even then, very few people care about quality that no one else can see
it really isn't a flex ppl think it is when they're burning tokens churning agents that are working for "hours" doing work
we don't need to read or write code anymore, but we do need to write (as humans) the instructions for agents.
having ADRs, product specs, acceptance criteria, etc. that are properly maintained leads to agents working for just a few minutes and pumping out reliable code.
the more vague the work for agents, the more randomness and inefficiency there will be. that leads to bugs, constantly re-writing code, etc.
imo ofc, i'm not yet running a billion dollar company so maybe i'm wrong.
Can you please point me to any project based around spec-driven development with agents which has worked well long-term, ideally one that isn't another agentic devleopment tool?
I think the HN crowd overestimates the quality of software. There is a (very) long tail of truly abysmal software that people are forced to use all the time. The reason it gets to be abysmal is because it’s often back-office admin software that helps people file forms for, say, logistics paperwork or school district expense tracking. AI-native engineering teams can create dramatic improvements for that long tail because the people running that software are either incompetent or checked out, and a pull towards average quality by AIs would yield great improvement.
One of the most uncomfortable truths about our profession is that there is literally no floor to how shitty software can be and yet still make people billions of dollars.
Reminds me of an incident where I had to go pick up a missed delivery (an expensive photo printer) from the dispatch warehouse because the shipping company couldn’t get a signature after three attempts.
Whatever, I drove down to this set of huge industrial warehouses with trucks coming and going by the dozen and found the customer service office.
“That’ll be half an hour, I recommend waiting at the cafe.”
“Half an hour for what? Aren’t all the packages right here?”
“They are, but looking up the package ID in the system takes at least 30 minutes, maybe 33.”
“33!? Wait… don’t tell me, let me guess: it has been getting slowly and steadily worse over years? Predictably taking longer and longer, right?”
“Yeah, six months ago it was 27 minutes! How did you know?”
“Umm… isn’t this a bit of drag on your operations?”
“Absolutely! Everything takes forever, every shipment is delayed, … <long rant> …”
“I… bet. Why hasn’t anyone fixed it!? It’s a one line fix!”
“No budget!”
This was one of the biggest logistics companies in the country about to undergo a multi-billion-dollar merger.
Their core business software was missing database indexes even on primary key columns!
Usability of internal tools is just never considered. Priorities are driven by management. They invariably just don’t care about feedback from the peons that actually have to use them in their day-to-day jobs, even when it results in tremendous inefficiency.
This also allows companies like Workforce to make billions selling the most godawful software imaginable.
> There is a (very) long tail of truly abysmal software that people are forced to use all the time.
We generally call it “enterprise software,” and it’s usually abysmal because the people buying/funding it are VP-level executives who care about compliance/reporting/visibility and who never have to actually use the software.
This actually comes up on HN pretty regularly. Or used to; like many topics it’s taken a back seat to AI.
...and now with AI much more of that dreadful software can be built with all the bullshit featuritis that said managers who don't use it think they want.
none of these types of pieces of content focus on what was created for users, its always about how sky high their "shipping" metrics are (PRs merged, LoC, etc). it makes it extremely hard for me to care, as they all boil down to "im a dev who made some loops on loops on loops that writes a lot of code and spends a lot of tokens."
i will say, at least this article says "we merge 5x more and our revert rate has stayed the same." so thats at least a metric i can start to respect
Any good product leader will tell you, sometimes not building something is actually good for the customer. Slower dev cycles have you the ability to take time with decisions and make adjustments. The obsession with shipping more code is making things worse for users and I would argue, damages the company in the long run.
Eveyone has a backlog. What happens when you clear it in a month? Do you suddenly have better ideas about what you want to build? Do you suddenly have better intuitive designs? Do you have more users or the same users using the product twice as much? In most cases, the answer is no.
AI gives you the speed to execute when you need to, but knowing what to build and how will still remain a bottleneck.
Now the real question is "is the revert rate the same because you still have the same process, or do you fix-forward more often now, so your revert-rate is because of higher-severity incidents?"
I mean we (huge tech company with ~2000 devs) literally never actually revert anything, in my 2 years here I have quite literally never reverted a commit. We have a staging/release cycle that means nothing ever needs a revert, you just fix issues in followup commits on trunk
This is another one of the infinite meaningless metrics that AI cash burners love to use without context
When that happens we revert immediately. But to be fair, this kind of commits have to be looked at in detail, and you often need the CI export because those yaml files are a pain to write...
Perhaps it shows that it does not matter how good your engineering is, or if you have sped up feature development 10x, if the shop is not providing value then who cares?
You're thinking about this at the wrong level. This org has made the decision to be AI-native. They aren't A/B testing being AI-native vs not being AI-native. That's the kind of thing "the market" does; if a company's markup is higher than another's it can attract more investment or its equity can trade for higher pricing on public markets.
I don't think it makes sense to A/B test "AI-native" vs other regimes of development unless you are a large, decentralized company like an F5 company or a Big Tech. How will a team of 25 (the size of the company if you read the thread) have enough output to even A/B test the difference?
It was the article that used metrics related to engineering velocity as evidence of success. It makes sense to ask how this is impacting their business metrics. The company is not in the business of shipping more code. I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
> It was the article that used metrics related to engineering velocity as evidence of success
Success for what? The article opened with: "Last week Will Larson (@Lethain
) asked how leaders are evaluating engineering managers for AI skills. My reply
, a list of questions to think about, got a lot of bookmarks.
More broadly, I've been thinking about how to build an AI-native engineering org for a while. Sharing some lessons from the last nine months of building at FurtherAI."
The opener is "how to build an AI-native engineering org" and the context is a company of roughly 22 engineers (from a reply.) Nothing here says "evidence of success"
> I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
In an company of 22 engineers, how would you make these measurements? Would you split your company into a velocity focused group and a quality focused group and make them compete? How? What would a customer motion look like? What would these groups share? This makes no sense. This is like asking a small company to A/B test teams that used a computer vs didn't use a computer.
A larger, more mature organization has the luxury of splitting business units into AI-native vs not-AI-native workflows and comparing COGS/business objectives against each other. I know of very large tech consulting services who did exactly this around the beginning of the cloud era to figure out which aspects of their business they wanted to place on clouds and which aspects they kept on-prem, and even then that was messy because part of the value of on-prem is the scale benefit you get by owning and operating your own DC. But no 22 person engineering team I saw A/B tested this out. Companies were founded either with a cloud-first philosophy or an on-prem philosophy.
This is a 25 eng startup. Anything can go wrong here. A hike in interest rates can lead to a down round causing the company to go into cost cutting mode, leading to a decrease in product velocity and prod incidents as infra is cut to the bone, in order to cope. Your top engineer can get sick and quit and your next in line isn't strong enough to keep the ship going.
I'm curious if you've ever worked at a startup, tried to invest in a startup, or considered purchasing a startup's product to know what it's like? The whole point is you're taking a risk trying something new and different. If you're just doing what a much bigger, established company is doing, you're fighting against an incumbent who already has much lower COGS than you.
Taking a bet on being AI-native as a startup is taking a bet that use of AI will result in lower COGS than the incumbent. As the old saying goes, "Nobody got fired for buying IBM", and yet Amazon was able to crack into that space by doing something novel. If Amazon was just IBM, why not buy IBM? The ultimate test of AI-native business models will be if these companies become the Amazon of today, or whether we'll write about them the way we wrote about "fifth generation computing" in Japan.
>> I've started largely using this site in a read-only fashion. The SNR here now is too low and I'm too busy. Found my time better spent in more focused offline and online communities.
since when does an engineering org care about revenue or customer satisfaction? Most software devs have no idea how the business they work for operates and think their direct deposit every two weeks is a fundamental law of nature. Also, many software devs are anti-customer and think customers are wrong even when the customer is talking to them one-on-one describing their needs in plain english.
I've gone through a few recruiter screens lately and the number of people who mention "number of deploys per day" as a metric they are proud of causes my eyes to roll.
What can you possibly be doing as a product that justifies you releasing 60 times in a working day?
It's certainly better than the old models of releasing on a quarterly cadence with the test team and all that, but it feels like the metric has become the goal when you are releasing that often.
This was the case at a previous job. Someone read some "state of devops report" or similar BS where they wrote that good teams ship often. They took from that it was shipping often that made good teams...
And as a result we had some contrived deploy process on Slack with buttons and emojis and what not to deploy.
Not a single customer saw any value out of this of course, because shipping something is not the same as shipping something valuable.
So I guess releasing 60 times in a working day proves that you're part of a good team /s
That's why I'm sceptical about the value of ai in software development. Much more can now be done but I don't see costumers paying more for it. The only way it can work is by laying off software engineers and saving money on their salaries.
Tend to agree. AI has failed to produce more business outcome. The only thing left is cost efficiency by reducing headcount. Thus 5x more productivity means needing just 1/5th your current staffing.
and IME the #1 rule for us is work in a profit center, not a cost center. When you're cutting there's a law of diminishing returns, but the floor is so incredibly low you don't want to be near it just to clear the blade.
This post is funny to me, because I feel it falls well short of what a proper, forward-thinking "AI-native" engineering org would actually look like. That is, the author is presenting this as kind of a radical departure from normal engineering, but I just see it as normal engineering with more AI. I don't see that as AI-native.
To me, true "AI-native" engineering will be one where questions regarding how humans will do things like review code are irrelevant. You explicitly throw away any concept of typical software engineering and, instead, differ to the AI. Your job, as a human, is solely to do what the AI can't physically do. That's not the same as closing gaps between what AI is currently capable of doing and what you'd need it to be capable of doing. It's literally enforcing that anything an AI is able to do is treated as correct in itself, and you, as a human, only support it by being a human.
If that sounds outlandish, that's because it is. We don't have such capable AI systems yet. It's possible we never will. But that's beyond the point. The reality is trying to make an "AI-native" engineering org by just layering AI on top of existing processes and systems is futile. If your expectation is that AI will continue to improve in capabilities at an accelerating rate, then these stop-gaps are pointless. You might as well just ignore AI altogether until it is able to completely take over, because that's the inevitable goal, right?
If you're willing to buy that will be possible, then my guess is that it will occur in the form of small scale startups and actual mavericks rather than someone trying to run an existing business who will simply be too risk adverse to go all in when the time comes. Some guy in his basement is more likely to leverage AI "properly" to build an actual "AI-native" engineering org because that person won't have much to lose.
> If you're willing to buy that will be possible, then my guess is that it will occur in the form of small scale startups and actual mavericks rather than someone trying to run an existing business who will simply be too risk adverse to go all in when the time comes. Some guy in his basement is more likely to leverage AI "properly" to build an actual "AI-native" engineering org because that person won't have much to lose.
FWIW I think the company writing this post is one of those small scale startups. But that being said, you're not the only one who thinks this. A decent amount of VCs and funds think that only smaller, newer companies have the potential to be truly AI native because there's too much organizational inertia in an existing business in keeping its existing human processes and so only a newer company can achieve the true "scaling potential" or using AI to scale without human labor.
I think this is all fair, but somewhat just semantics.
Most engineers are working in "normal" engineering orgs, but can feel the ground underneath them shifting, and companies believe that if they are caught flat footed, they will be out competed in the marketplace, so they are all trying to make sure they are adopting it at least in a way that lets them keep pace if not accelerate revenue.
If you are not impacted by economic incentives, you can continue to not care about this transitional state, but most people do.
Are there any companies outside of the "AI shovels" business that make more money because of AI? Every time I ask someone why they went so hard on adopting AI, I hear the same story about the fear of losing the market share or becoming less competitive. Honest question, who is benefitting from all of this? The revenues at majority of companies seem flat at best. There is a growing concern about professionals getting deskilled, the job satisfaction is not improving, wages are stagnant. Mind-boggling that we are actively engaged and often push for this experiment that hurts our own profession. Cheering for our own demise.
Not keen on outing myself by providing evidence, but I think we do generate more revenue by using AI. We launched new products with a small team and also optimized our outcome-pricing based product using AI several times. We also reduced costs by self-hosting more software.
But we're a tiny baby startup, so things like "spin up a new product with 1 engineer and get to $1m in revenue on it" or "reduce our IaaS bill by 300k/yr" is a meaningful contribution for us.
We're also building new AI-enabled software that was not possible before.
The big breakout hits have been AI Infra, but that makes sense from a technological perspective, Cisco made a lot of money well before Google did, but Cisco is not where the money concentrated in the end.
We don't really know what the equivalent of Google as an application business will be in the AI era, but there's a pretty big chance it's not actual Google. But Google still has to do it's best to hold on.
I use AI compatibility as a metric when evaluating new software to add to my stack. If an AI agent isn’t able to do most of what you can do in the app by opening it, I prefer to look for other options.
Does not apply to everything though. E.G: I don’t care about AI support for my non work chats/email, since I’d rather not give it access.
Reminds me of a few things, like being a meat proxy, as well as books like Company of One or Million Dollar One Person Business, and these were published before AI so right now we will see companies with only one founder and hundreds to thousands of AI agents. There already was that drug delivery company with the one founder but apparently a lot of their revenue was fraudulent. Still, it proves the point that it is possible.
I was misremembering, they were sued for spam laws, not for their revenue being fraudulent so it seems it's legitimate, making a billion in revenue with one person (and later on his brother who was hired), the rest is all AI.
IMO most modern engineering practices are cargo culture. I’m sure in 10 years we look back on today the same way we’d look back on the days before version control was common. We are in the emailing files around stage of development
> I feel it falls well short of what a proper, forward-thinking "AI-native" engineering org would actually look like. That is, the author is presenting this as kind of a radical departure from normal engineering, but I just see it as normal engineering with more AI. I don't see that as AI-native.
A company which proclaims they are "AI-native" isn't actually interested in engineering, they are trying to ride the hype wave. Companies who care about engineering just... do engineering, and don't make a big deal out of whether they are chasing trends.
The reality here is even more idiotic: someone criticizing a company for a terrible strategy they aren’t using that wouldn’t work if they did. According to the critic!
If you follow this logic, we get to the true "AI-native" engineer which is ecstatically floating in a bath of dopamine-stimulating liquid, while AI reads their brain pattern and instantly builds everything they didn't even form a full thought about.
"If that sounds outlandish, that's because it is. We don't have such capable AI systems yet."
It this point, a whole lot of teams have been leaning into AI-first, not-entirely-reviewed code since around March this year.
That means we are already 6-8 months in, and not far from having a full year of experience of what that actually looks like, albeit weirdly influenced by the increase in model capabilities during that period.
I'm looking forward to those retrospectives. Did this turn out to work OK for the teams doing it? What were the biggest failings and/or surprises? With hindsight, did this produce good quality software?
I think the biggest shift is in predictability. Will something be done today, next week, or next month - nobody knows. Will we converge on a bug free state? not sure.
So far AI works great at fast moving work which you will probably recode or throw away in a month or two anyway (e.g. a startup) but is weak at maintaining a large code base that you rely on. At least part of this is cultural, as the cost of misalignments between team members can be quickly translated into broken software.
Software robustness, at least in terms of "has the app crashed on any client machines today", is achievable. It's just expensive enough to achieve that most people (rightly or not) don't prioritize it, and the more moving parts (like third party libraries or language runtimes) your application has the harder it is to achieve. Maybe AI will lower the cost of achieving that.
When I worked on Guild Wars every server or user crash went to a shared inbox and was bucketed so we could identify and act on new crashes. IIRC we maintained very, very low production crash rates and could prove that many of the crashes were due to hardware faults.
Similarly, Mozilla tries to analyze and act on every crash report they get in the field. If a webpage visited by even a small % of the customer base was causing crashes, the team worked to try and fix the crash to deliver stability. If it was a high-traffic webpage it got treated as a high-priority task - I remember Farmville breaking was basically a 3-alarm fire back when it was extremely popular.
"No bugs at all" is an even higher bar, but it probably is possible for simpler applications with the use of stuff like theorem provers. I'd love to see someone try it in production but I think the return on investment for that is probably super super negative.
I operate on a "no known bugs" basis with quite a bit of effort invested to try to find new, unknown bugs.
The idea that every crash isn't acted on is kind of a crazy one, to me. Not every crash is actually actionable in e.g. a mobile app, but every one should at least get an eyeball in my opinion.
This, to me anyway, is the bare minimum of "caring about software quality". If you do this regularly you'll have very solid code, and it's interesting that I have the perception at least that Guild Wars was a quality product.
There's another layer, where you do that for subthreshold issues like UI inconvenience, etc, but that's bonus, I think.
For all the talk about AI making teams super productive, I don't see much talk about AI improving software quality.
It's a shame, though, because I believe there is a lot of potential there.
I mean, quality is easier with the right tooling (monitoring, linters, property-based tests, theorem prover...). But, given limited resources, it can feel difficult to prioritize investing in quality when business is pressing devs to ship, ship, ship.
But now the cost of writing custom tooling has changed. These custom tools might be sloppy because vibe-coded but we wouldn't have them otherwise! Or we might finally have decided to make that investment after too much time suffering from their absence. Now the cost/benefit changed, so why not invest now?
> But now the cost of writing custom tooling has changed. These custom tools might be sloppy because vibe-coded but we wouldn't have them otherwise
There’s a lot of good tools out there that are not used, either because of ignorance of their existence or lack of expertise. An hour of config can pay real dividend later.
And sometimes you just use some unix utilities and hack a bash script while testing things out.
After 15 years on the field, I still lack a lot of expertise. Many teams do, as you say.
LLM can save hours of doc reading for non-critical tools that I would otherwise not use. Its also pretty good at "hacking a bash script" in order to test things out. Not everyone is confortable writing bash scripts.
Have been at it since Nov 23rd 2025 with a first commit with Opus 4.5 - it was rough at first but after about year well worth it. You can also tell where so many just get stuck in the mud due to lack of direct experience with all the engineering silos (from bare metal sys admin to QA eng silos outside of standard software engineering core) it takes to roll out a new kind of technology company while scaling a new kind of commercialization model, also managed by AI and AI native from the ground up.
Working at an "AI native" company and these questions and stats are far too hand-wavvy for me.
What works:
- Empowering engineers. Giving engineers more trust, ownership and product decision-making leads to more engaged engineers vs. micromanaged tickets. Engineers who are treated like owners tend to notice cross-cutting problems and push to get them cleaned up.
- Hire for diversity. Not race or gender, but diversity of thought. Different ages, backgrounds, approaches to AI, focus on frontend/backend, etc. Race/gender diversity tends to come along for free. This is a huge edge because every LLM is spinning up the same next token prediction, so approaching problems from different perspectives gives more robust coverage across your codebase.
- Flat org structure, low ego. Interns and 20 year vets work side-by-side and teach each other things - hire for openmindedness.
- Communicate about product. What should it do? How do we want a feature to work, and how to handle certain edge cases? If the product is well understood, the code follows. In-person/hybrid is valuable.
- Architecture matters more than ever, but it is easier than ever to fix. Listen when engineers mention tech debt. Notice when a simple change touches 27 files. Senior engineers shine for this, because they tend to have a better nose for code smell.
- Reviewing still matters. Anthropic may throw tokens at reviewing, but it is a great way to share ambient product/code knowledge across the team. 200 engineers throwing code against the wall is chaos and leads to YOLO codebases. You can move fast AND keep a rough mental model of the codebase intact.
Been moving our 17-person engineering team to adopt ai and this comment summarizes all of my findings perfectly.
This specifically summarizes a huge gap I think some people have around ai:
> Architecture matters more than ever, but it is easier than ever to fix.
You can prompt away to your heart’s content, but if you’re not pushing it in the right direction architecturally, you’re bound to face frustration eventually.
The most successful way I’ve found to convert those leaning anti-AI or AI-skeptical on my team is just showing them how much tech debt I can clean up with AI. They get excited to pitch in, and they see that it does actually benefit them in some meaningful ways. That set of endpoints that’s been neglected for years? Let’s stop complaining about them and DO something about them, today. I’ve actually seen life be blown into some devs who were coasting for a while. It’s an interesting time.
I was just looking at some user-facing error messages Claude generated for me which contained unnecessary/irrelevant technical details while failed to provide clear instructions or fix.
Earlier, I reviewed code that does way more than specified scope, changes public API and introduces new behavior. I had to almost redo the whole thing.
I thought to myself, no way I am going to ship the code as written without actually reading through it (or as some people put it, only looking at the "intent"). I'll get more complaints from users while having absolutely no idea what's going on with the code or actual behavior.
It's a puzzle to me how these orgs can pull it off.
It really depends on the kind of product you're working on. A lot of developers (and managers and business leaders) assume that their workflows will apply to other kinds of products; and are often very wrong.
For example: A few days ago I saw a quote that over 90% of current Y-Combinator startups vibecode their products. The problem is the author misunderstands context: Startups sacrifice quality for time to market. In other cases, if you're building something that isn't technically challenging, (because it's 80% similar to many other products out there,) you can probably vibecode it.
In contrast, if you're doing something technically challenging, it's to your benefit to be a lot more hands-on. I recently started getting excellent reviews from copilot of my hand-written code. (Or, I should say, the hand-written parts of an AI-assisted pull request.)
I think it also depends on how well defined requirements are. I can have Claude nearly one shot a Terraform provider but it's just taking 2 API contracts and combining them together (the provider contract and the API you're automating)
On the other hand, some things are incredibly product intensive and poorly defined and it seems LLMs will latch on to an arbitrary direction with much lower success rate.
It is also worse in "brown field" where there's lots of tribal knowledge and technical debt. Two systems may both exist that look very similar but oftentimes there's the one on life support that almost no one uses or cares about and the LLM doesn't know it should largely ignore that until it's thoroughly documented (which can be a significant undertaking)
The people / orgs that can actually codify their strong software engineering principles and practices into "a harness"[1] are the people / orgs that find success with these tools.
Even then, the actual development of a harness for interesting problems is a difficult task. It requires more skill & experience than people realize, and gets harder as you scale things up.
Spent few minutes reading the infomercial -- regretted it. Engineering principles form the bedrock first, and AI-tools are made to fit in them, not the other way around as this infomercial propounds. Its linkedin vibe is a big turnoff too.
It describes a team spending 30x payroll on tokens - $3,000,000 in token spend per month - building a multiplayer game where incidents and bug reports from users are passed directly to Fable 5/5.1 agents who then act on them and deploy the changes to production.
"We consider it an outage if we don't ship every hour"
It's worth reading because it's either a complete disaster or a glimpse of the future, or quite possibly both.
well, i did go check out the company spawn from the article. its certainly interesting how hands off their process is and how much they ship, but holy shit the games were fucking horrible. there was a direct WoW ripoff (right down to the assets) that was unplayable, a racing simulator where my controls stopped working 10 second in, and some sort of RPG that i started and just never loaded. so i will take what they are saying with the heaviest possible grain of salt.
Yes exactly this is my standard for everything now. You can spend an unbelievable amount of tokens doing all kinds of things that are interesting as processes in their own right.
at the end of it: is anything of value delivered to customers? the company might say "well users make the games with AI, we just serve them and host the multiplayer." okay, but again, what value was created? they brag about being one of anthropic's top customers; congrats you are handing VC money to anthropic. why should i care? theres no answer for why anyone should care, because to them the system and clout is the product, the product doesnt actually matter.
at least thats my read on it. i dont know how else to understand whats going on here.
Oh it's Jacob - I used to work w/ him on his acquired startup, Pluto. He was very AI-pilled; I'm not surprised at the results. I remember wading through the AI-generated Python he had written. I think Pluto had something like 500k or 700k lines after 18mo. Everything was duplicated; there was no reuse of anything. Dozens of features, which mostly worked, not all did.
As a product that I would regularly use, Pluto missed the mark for me. As a SWE in 2023, I couldn't meet Jacob's mentality. Today, it's how a lot of us ship software.
I didn't love the product but I have to respect the process: he was able to sell his company. For some, what he did is success.
That actually sounds really cool. Assuming it has access to a well-tuned predictive analytics mechanism that allows it to derive that changing the game in this way would increase satisfaction and revenue from you and other players like you, sufficiently to offset development costs and dissatisfaction to others, then it really should.
We are already - as in today - doing this with friends while playing Valheim, the process is semi-automatic with me in the middle as the admin, though.
We make 10+ gameplay changes a day in our custom kitchensink mod that steers everything. 70%+ are bugfixes (we have plenty of mods that conflict/are buggy), the the rest are QoL improvements, small new features and rebalancing. If we see some playstyle is too strong or boring, we spice it up the same or the next day. Mobs too weak/strong? Update the same day. A ton of performance fixes with actual in-game benchmarking. All this while I keep playing.
I built sort of an automatic modding pipeline (including public release and all) with parallel test game clients + servers used by LLMs to develop and test the changes and then merge before releasing a new mod version. We have had very few issues, I'm still shocked how reliable this is. They follow guidance well and test everything in real game clients on wipeable clones of our live world and our actual characters. Josh has a bugged chest at his place? The LLM will reproduce it, develop a patch, yes, but then also log in as him to his actual in-game base and test the fix using game inputs + OCR. At the end I get a .html page report on each change with screenshots and technical details if I ask for them. I don't care if it takes 4h to do a simple fix because it's not my time and game stability is critical. And up to 8 of them can work in parallel, so there isn't a bottleneck caused by excessive testing/re-testing.
I think this will be the future of gaming for many sandbox/easily moddable games. It's a bit too technical at the moment for regular players to handle the updates and models' quirks, but we're only going up from here. Just crazy.
I could definitely see potential in an async co-op game based on this idea. A sim harness, some pbt to avoid broken states, and something like a modified nomic ruleset might get pretty far.
whoa like an ai calvinball. Or, you compete to get the right to one prompt that changes the rules/physics/map etc. Whoever gets to the genie's lamp first …
This is amazing and somewhat insane. It's more like they are building an organism with an immune system than a code base. Interesting that they say only Fable works for them.
I’m building the same but with a malleable SaaS. Basically when you sign up, you get the software deployed and an agent that can control that deployment and code.
But the process is slow. The user wants to add a feature to their app so they ask the AI. The AI has access to a terminal where it can clone, fix, and push. This takes time and isn’t instant.
Once AI can abstract this part, software will change forever. It’s what this post alludes to, does code, PR, etc need to reinvented for this?
Once so, no one would ever use cookie cutter software because you could build your own and maintain it using just an agent.
Yes but now that cookie cutter software can be the starting point. I’m not saying to build it from scratch. There’s probably an open source alternatives to the software you are trying to replace. Use that directly or tell AI to clone it and build it in a stack you understand.
Before AI, the businesses that used Zapier and automation were able to pull ahead of businesses that were still doing things manually. A business that has a custom built software with an agent that can change it on the go will provide even a greater advantage.
This is a complete and utter disaster. They are making spam games. Like miniclips but for spam. A semi functional website to make what is basically youtube playables.
And it's devouring tens of millions. I know what tens of millions can make in the video game industry, it can keep a small team of good devs and designers and artists making something special that's _really_ enjoyed by tens of thousands of people.
We don't want more games, we want more great games. This will never, ever, make great games.
Look on ye mighty, and despair. I really don't care how this sausage was made because the only thing you need to know is that it shouldn't ever have been made.
As a (light) gamer for two decades who has been reading about the gaming industry for the past few months, I cannot understate how hard it is to create a good game, let alone a great game. So much money was spent/wasted on hopeless endeavors.
Was thinking of that exact article while I was reading this, we must be following the same folks. Highly recommend folks reading it, even if it feels insane in some regards it's a fascinating look at what the limits of this stuff could be and I found myself with a lot of new ideas to consider after reading it. Interesting times.
Edit - after writing "we must be following the same folks" I realized that you are, in fact, Simon Willison - I saw this article because you posted about it on X, haha.
This works as we've been doing it since Opus 4.5 (Nov 2025) with every user request, feature and bug. It's the future without a doubt. The feedback loops evolve and get stronger depending on how well you fine-tune or train your system to optimize on its own.
That’s the dumbest thing I’ve ever read. Spaffing millions per month on this slop is the kind of thing people suffering from AI psychosis do. There is no way you need “thousands of agents” to build any product.
The more agents you throw at something, the more they collide. Those collisions generate more slop, which requires further slop processing agents.
> A default that made Savi think longer on every message, landed at 10:22pm and reverted by 10:40pm: that's not right, undo it it's too slow and not jazz.
I giggled on that one. It feels like children playing having a job. If they pay that much for tokens they have to have at least 80k customers paying 50$ a month. Seems huge for something like that. Or they have a lot of investors. Anyway does not feel like something going anywhere.
This from that tweet stood out to me: "In general, we've seen that OpenAI's models game their own checks"
In the past when working on agentic stuff I've observed models gaming checks aggressively, to the point that when we were validating log files to see what the agent had done, the agent responded by editing the log files to pass validation. I can only imagine how bad that kind of behavior is at scale, and it sounds like they ran into it.
Won't there also be a point were you're shipping so fast that that in itself deteriorates UX? What if a page literally changes every time I reload? What if it changes mid-checkout process and the deal I agreed to 5 seconds ago does not exist anymore?!
I have come to realize that code (and I think possibly productivity in general) being expensive was always a feature. I am sure that a few great ideas did get lost to obscurity due to lack of access to resources to build them out. However, we now have an absolute deluge of complete and utter shite.
We didn't need LLMs to come up with the genius idea of making every desktop app a web browser.
Software quality was in terminal decline already, the only hope is that LLMs get good enough to dig outselves out of the several miles deep hole that has been dug.
Agreed, for the individual LLMs can be powerful to let less technical folks build and customize their software - at an org level I reckon we'll start seeing more sinking ships as that bottleneck was actually helpful at limiting the flow of lackluster features to prod.
I see a lot of builders (myself included) falling into the trap of focusing on the wrong things. Development velocity doesn't matter per se, what matters are things like growth in profit or traction. It's really easy to conflate automation/systems/process with actual business results. Movement ≠ Progress.
> None of this was free. Through the summer, much of our engineering effort went into tests and tooling, and customer-facing changes per engineer stayed between 17 and 30 a month
This is the reality I think for a lot of orgs and is absorbing a lot of the productivity gains. The gap between vibe coding a green field app from nothing which is inherently "AI native" and implementing AI tooling into complex orgs with large amounts of legacy infrastructure, integrations, ingrained practices etc is enormous. For us, it has been a big effort to enable agentic coding over a significant footprint of our codebase. And all this is happening while the plane keeps flying, with near zero tolerance for defects in critical systems. There is a lot more to it than you will understand if you just look at twitter or Hacker News.
I remember optimizing a huge build that took days. We optimised it to only hours on a cluster - it took years to get this right. We realised at the end of this that the packaging system (handled by another team) was taking about 2x longer than the build because there was one particularly huge zipfile that sat there compressing for ever and ever and ever while the rest of the cluster was idle.
As in this story we blew enormous effort optimizing something that was very difficult and because of our blinkers (and our management's blinkers) we missed that it was only able to achieve a limited result. We didn't look at the whole system. In the end it was never fixed because......siloed thinking...or something.
> None of this was free. Through the summer, much of our engineering effort went into tests and tooling, and customer-facing changes per engineer stayed between 17 and 30 a month.
What it cost you is that you helped the AI labs become better engineers. At a certain point the Big Flip will happen, and the AI labs will stop their offerings and instead start competing with the engineers.
This assumes that it’s more profitable for these labs to compete with their own customers, rather than increasing the productivity for their customers and - assuming this increase justifies it - increasing their profits.
For the argument lets say that this approach works and results in high quality apps. But will be users happy that app is updated 3 times a week and all updates add major changes? Because code generation is so cheap and fast it means we should even more thing about what to build to not overwhelm our users and actually solve their problems.
First you take your Blendtech and "will it blend" your engineers, then upload the result to the cloud inside what is called an S3 bucket (don't forget to set access permissions on your buckets kids!) to be digested by the clankers, sit back and profit!
Dark humor aside I think disconnecting a bunch of problem solvers from their computing equipment could really enhance the world, putting that analytical power into the world in person could move a lot of cogs, rather then marshalling JSON.
I know many (big) places that are already back-paddling.
From token leadership boards (more = better) to now saving on AI cost, because the benefit was less than expected. As if this wasn't obvious. Sigh.
Sure MS, Meta, etc, are still laying off engineers to have money for data centers. But I am 99.9% sure that they will thoroughly regret that move in less than 2 years.
We shipped more code!
We merged 5x faster!
We spent more tokens!
>> Every time we sped up one step, work piled up at the next... we started going through the entire process of shipping software, one step at a time...
I feel like I'm being gaslit here. You can become CTO with this lack of business fundamentals, and a basic misunderstanding of systems engineering?
Can we ban twitter links? I have to use a hack to even read it and it’s always just slop. If you can’t be bothered to write an actual article, why should we bother to read it.
Here we are again, thirty years after I was taught the "waterfall" paradigm at business school.
When asked about "customer satisfaction" we were told, awkwardly, that became "political"
The paradigm did not survive its encounter with reality, and I do not think this one will either
The "only" 175% increase in productivity is very impressive, and seems realistic to me, when you remove the chores of writing code (thank you to coding agents)
What you build is still very dependant on what you need, and you do not know that until you build it
It is a paradox only for the "waterfall" paradigm. We rejected it, in the 1990s for a reason
Not a single mention of value or revenue.
Only three metrics went up: LOC, token spend and presumably release speed/frequency.
What's the markup (profit/COGS)? is it up?
Customer satisfaction? Is NPS moving one way or another?
I don't get the obsession with more code. Whatever happened to good development practices? Many of which, if followed allow LLMs to work more efficiently ironically enough. I would rather hear that you've fixed bugs that have stumped your dev team for years, and have had next to no outages, to me that's more valuable than just slopping features out the door.
While I agree with your sentiment, very strongly actually, there has always been a tension between what I'll call "engineering teams" and "product teams". The engineering teams want the long-term platform stability, performance, etc. The product teams want to push that new feature out, quick! The overall business needs both (types of thinking and doing) to survive, of course, but not all leadership sees the world through the lens engineering team lens. I'd argue most actually lean the other way, which is understandable though unfortunate. I'd also posit that since these product teams are often "closer to the business", those people typically win favor with the real decision makers.
These new tools only exacerbate this issue.
This tension is why I refer to myself as a product engineer. Teams where non-technical product managers (ie. your "product team") become feature factories that eventually drown in their own tech debt. Teams where navel-gazing engineering managers (ie. your "engineering team") have too much power end up with premature platformization, and beautiful architecture diagrams while user needs go unmet.
The key is to understand that computers exist to serve human needs, and the less code / UI / hoops to solve a problem the better, and to think end-to-end. If you are able to do that, most systems built by committee have a lot of low-hanging fruit that can be improved without much sacrifice. In practice, political realities and operational concerns may come in the way, but I firmly believe this is the only way to successfully operate as an engineer, and it will only become more important in the age of AI.
And I feel that stuff like agile is now even more important than before. Now it’s role should be more about protecting customers from bad changes and following whole processes through where you actually measure the impact. It’s just too easy to do those 100k lines changes to products because it’s some vice presidents pet project.
If you build your platform well, then you can execute fast, but also safe.
I push "money and data are sacred"; if we lose either of those, someone cries. Build the platform slow and fast. Governance, scaling etc sits here.
Provide reference APIs and let the product team one shot whatever they want and need, build fast and experimental, but your platform enforces constraints.
Best of both worlds (though feels like we're back in backend vs frontend development)
>If you build your platform well, then you can execute fast, but also safe.
this is extremely true, but it seems less and less like anyone wants to invest in a solid platform and just wants to jump to executing fast and unsafe. and then pretends that ai can definitely do both when the experience is quite a bit different.
I have seen at least one example develop in real time. You typically spent years building before you had a platform, then lack the enthusiasm from senior leadership to invest the next 12-18 months with no new features and hard to quantify - maybe no - returns. Then you get displaced by newcomers who are "platform native"; what this really means is their engineering misteps haven't been discovered yet, or even if they guessed correctly the world has turned and made their previously correct solution now a failing one. And now we have something called "AI-native" that is going to solve this once and for all with some sort of magic, and you can't lead these companies unless you truly believe in the magic.
“ You typically spent years building before you had a platform, then lack the enthusiasm from senior leadership to invest the next 12-18 months with no new features and hard to quantify - maybe no - returns.” —> this is failure from the team and especially people who should do product management job on that team.
I'm lucky I guess, I feel like the product team is on "my side" with respect to building a resilient system. They understand that bad architecture shows up as a fragmented and friction-full UX.
The demand for new features come from sales and executive people who want to check more boxes on a customer's checklist.
Ultimately your paycheck exists through deals that involve those boxes on checklists. It doesn't simply materialize out of thin air. If there are no deals, it doesn't matter how robust and elegant the architecture is.
Features don’t usually come from product teams. They come from above: some Head Of, VP Of, and the like. Those people only ask, they know very little about execution. They want their wishes come true in a timely manner
They have been vibecoding via a harness of human LLMs this whole time.
I think this is the key to some of this debate.
Management/Product/Product Engineer - When you are responsible for getting the work done then you look for any and every option to speed it up.
Owner/Platform/Senior Engineer - When you are responsible for ensuring it works and can be maintained then you have to pay attention to how to build it, aka the trade offs
Well support teams are even closer to the customer than product and I think they would advocate for perf/stability over more features.
How would that change when support teams themselves are replaced by LLM agents
Any evidence that this can be done without terrible self-harm to the business?
I've spent most of the past two weeks trying to get my AI to stop writing so goddamned much code. It's great that I can, say, completely switch between two databases, but I wish the AIs would develop a concept of energy spend or something and spend maybe a moment checking whether something is already implemented in the code base before blasting out 2000 new lines defining a new database access layer, and also rewriting random code it shouldn't be touching for some task hundreds of lines at a time just because it can. I don't care how easy it is to write some tests that sort of kind of cover it hopefully maybe... no new code is as tested as the stuff that has been doing it in the field for years already.
Just because it has the power doesn't mean it should be used.
I'm not asking for everything to be some sort of beautiful code whose concision would lead Knuth to tears. I'm just asking that you not slather dozens of existing functions with three new parameters and dozens of lines of code each that basically do nothing, to cover a use case that I accidentally vaguely alluded to, if you read my prompt a certain way, but isn't really what I was interested in.
Totally agree but I suspect that our desires are in contrast with ai companies profits
Right. There IS a way to convince ai to write less code.. but it obviously spends 10x tokens, whether it’s LLMs as judge and automated review or just write 10 solutions and keep 1. Partly ai companies like it this way but partly it’s unintentional consequences that happen to work in their favor.
Training for good taste in architecture actually causes premature optimization at high effort allowance, but at the same time, it is combined with an inability to even recognize / exploit that later if it should ever become actually useful. Dial the effort down, and it skips the reasonable abstraction and nothing is dry, factoring is insane.
Goldilocks for simple stuff barely works with frontier models after they are out of their comfort zone for topic or line limits.. you kind of just have to decide whether it’s worth it anyway. In practice.. constant supervision for high quality code, or such high quality system design that the code quality is less of a down side.
One thing I've spotted several times throughout this year: the larger volume of code leads to exactly the sort of problems it always has. It just happens real fast now, with fewer people. Problems that used to take years to erupt now manifest before an MVP ships, but people just sweep it all under the rug in the name of moving fast.
The rate at which software is produced has never been higher, nor has the rate at which it is thrown away for being broken and useless. It’s not at all obvious to me that if you control for both of those rate changes that the amount of useful software with a lifespan measured in years created per developer (ie, the software that actually has long term value) has actually changed all that much.
I drastically improved the reliability of an internal IoT device so it can tolerate all kinds of abuse (abrupt power loss during update, unreliable SD card, bootstrapping over network when the SD card is missing/damaged, fail-safe updates, recovery in a network isolated environment). This is the first release that I really leaned into AI, having been pretty skeptical before.
I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
> I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
It always works best when layering on top of good human code. The thing is, you start to run out of that as you lean in. Tricky.
Heh, you should look at the reply I left for your sibling comment. I think it is just a regression to the mean of all open source software upon which the models were trained. Sometimes the mean is an improvement.
Hell, if the Dunning–Kruger effect is to be believed, it would be bold me to assume I am capable of good human code. Maybe the average is the best I'll ever get.
> The thing is, you start to run out of that as you lean in. Tricky.
I think it depends how you lean in. I generated a few thousand lines of Rust code today. It was actually pretty high quality. On my initial review, I only identified one change I wanted to make.
Most functions were pretty short and well factored. I wonder if using Rust as a target helps, because (1) the training data doesn't include all the terrible Javascript and Python etc. found on the internet, and (2) Rust and many of its libraries encourage good architecture - code factored into traits, functional pipelines, a powerful type system, etc.
LLMs trained on bad code for mediocre languages naturally write corresponding quality code.
A big part of it still boils down to prompting - what you're asking it to do and how much freedom you give them. If you let them make all the architectural decisions, you're likely to end up with issues.
> It can be used to make existing products better and more secure. You just won't naturally hear about that.
Why? I keep asking for examples of LLM being used to make code more simple and elegant (which I would say is adjacent to making software better, if not more secure), and never get a reply, not a single one. I'm not claiming it doesn't exist, I know it must exist. If only via a lots of careful massaging of system prompts and harnesses, even if it takes too much effort to be worth it, I would like to see it.
Of course people can't share internal examples, but this is all over the world for many months now, where are the open source examples? Not "something new someone couldn't have done before", but someone making their precious even neater (while also adding things to it, but not primarily). Put differently, can LLM delete code, can they curate it well, can they not write weird comments nobody wants or needs, that sound like echoes of the user prompts and responses to them, but just explain tricky parts of the code, like a human would?
Also, IMO a VERY useful thing to learn about a tool or a chemical or whatever is, "what happens if I keep applying it to the same small thing over and over?" -- because that's the level of scientist I never evolved past, the kind of stuff you do as a kid -- , whereas a lot of this seems to be more about how it feels to apply it here and there until it looks good, and then move on and never look back. Which really isn't much at all, especially given we're such a short time in. I mean, how useful were mugs made of lead? Are people aware of the (in hindsight) insane things that were done with x-rays as those were invented, before the potential for harm was known? I bet they had a great time playing with that stuff, for a while.
What if in 5 years it turns out that a few years ago, LLM got into this habit of implementing certain things in a certain way because it seemed the best to "them" (the training, whatever), but there's a security issue with that -- what do you do? You can only let more agents loose, and hope they don't miss anything, because you cannot even do something like "patch PHP" (which is not my thought, Casey Muratori mentioned that in on of those podcast things).
Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though. It just does, like suggesting we go to Mars, because everything is better there, and to do that, we have to blow up Earth. Or saying to magically be able to fly from now on, we need to saw off the branch we're sitting on. Even if I thought life on Mars was the future or that we could fly, if we just thought of it the right way, this isn't a way forward IMO.
> LLM being used to make code more simple and elegant
Simple and elegant generally doesn't come from the LLM. It comes from me saying no to the LLM. I've discarded many branches that didn't feel like they were going the right way. Hell, I aggressively do that with my own code also. I rarely commit the first thing that works.
> where are the open source examples?
My general impression of open source is that it is, in part, a market driven by good software engineering. There's certainly below average code, but that tends to get mocked out of existence before it grows too much. Proprietary code that doesn't have a strong forcing function used to be able to hide poor engineering principles.
I think AI generated code is a regression to the mean, and the mean can be an improvement sometimes. I inherited this project. It was not originally written with AI. A lot of hard work and good engineering went into the hardware/firmware. The IoT software connecting it to our intranet, less so. Some examples that I'm fixing/fixed: ignoring HTTPS certificates, RCE by design, in-place file updating JSON files, a hand-rolled (but ineffective) fair mutex, busy waiting for a device to appear, and worst of all, no tests. None of these are hard to avoid, and even the small AI models would have come up with a better implementation.
This doesn't seem unique to my project either. I think a lot of software is just silently getting closer to the average of all open source code upon which the models were trained. I don't think I want things like SQLite to get closer to the average.
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I think you might be reading too much into what I said. When I say "fully lean into", it's probably less than you're thinking of. I still feel that I understand what was written and can stake my reputation/career on it.
You want them to make code more simple and elegant by themselves, no other direction? "Make this code simple and elegant, no mistakes, no hallucinations"? I don't think that's realistic at the moment.
But with suitable direction, they can definitely improve codebases - especially a typical enterprise codebase, which, if you don't like LLM code, you would hate with a passion. (I can only assume you've never seen such code otherwise your perspective would be different.)
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I essentially agree, but there's a huge spectrum before you get to that point, and lots of ways to use it effectively that doesn't create tech debt.
> You want them to make code more simple and elegant by themselves, no other direction?
No, he wants a project where AI is used as a major help, but the result tend towards good code, i.e readable and well abstracted. Even if the cost is high, just an example where you can extract the best practices of using AI coding.
You are responding to someone that just today generated thousands of lines of code, claims to have reviewed it all, thought it was good and only wanted one minor change. Surely they have an example of the good code it produces.
We used to understand that code was harder to read than it is to write. It follows, directly, that better engineering discipline means shipping LESS code. Less to support, less to understand, means a product that is easier to maintain and extend.
Now, token calculators spew out code at a pace no human can match, which completely removes the barrier of effort - which was one of the largest natural counters to shipping massive amounts of slop.
I'll be impressed by a coding model when it writes less code than a human, and spends more time refactoring than it does just adding new layers. Until then, this vibecoding is just madness.
Idk imo code is and has always been a liability. Like the toilet paper bear says "less is more"
1. The issue is that most people and organizations do not care about maintenance. Maintenance doesn’t get you the promotion, accolades, or immediate quarterly revenue because great maintenance tends to be invisible.
2. This is exacerbated by the lack of loyalty people have towards their organizations because their orgs don’t value or reward loyalty. That’s because leadership doesnt care about the long term view because of item #1. Hence, everyone is only doing short 1 - 4 year stints in order to get meaningful raises
3. The exceptions in the private sector are businesses that not publicly traded.
4. Even then, very few people care about quality that no one else can see
I think you are overcorrecting. LOC is highly correlated with productive output. I’m willing to bet on it.
agree 100%
it really isn't a flex ppl think it is when they're burning tokens churning agents that are working for "hours" doing work
we don't need to read or write code anymore, but we do need to write (as humans) the instructions for agents.
having ADRs, product specs, acceptance criteria, etc. that are properly maintained leads to agents working for just a few minutes and pumping out reliable code.
the more vague the work for agents, the more randomness and inefficiency there will be. that leads to bugs, constantly re-writing code, etc.
imo ofc, i'm not yet running a billion dollar company so maybe i'm wrong.
Can you please point me to any project based around spec-driven development with agents which has worked well long-term, ideally one that isn't another agentic devleopment tool?
i would second this. at $company they are jumping head-first into this and its not going well at all...
I think the HN crowd overestimates the quality of software. There is a (very) long tail of truly abysmal software that people are forced to use all the time. The reason it gets to be abysmal is because it’s often back-office admin software that helps people file forms for, say, logistics paperwork or school district expense tracking. AI-native engineering teams can create dramatic improvements for that long tail because the people running that software are either incompetent or checked out, and a pull towards average quality by AIs would yield great improvement.
One of the most uncomfortable truths about our profession is that there is literally no floor to how shitty software can be and yet still make people billions of dollars.
Reminds me of an incident where I had to go pick up a missed delivery (an expensive photo printer) from the dispatch warehouse because the shipping company couldn’t get a signature after three attempts.
Whatever, I drove down to this set of huge industrial warehouses with trucks coming and going by the dozen and found the customer service office.
“That’ll be half an hour, I recommend waiting at the cafe.”
“Half an hour for what? Aren’t all the packages right here?”
“They are, but looking up the package ID in the system takes at least 30 minutes, maybe 33.”
“33!? Wait… don’t tell me, let me guess: it has been getting slowly and steadily worse over years? Predictably taking longer and longer, right?”
“Yeah, six months ago it was 27 minutes! How did you know?”
“Umm… isn’t this a bit of drag on your operations?”
“Absolutely! Everything takes forever, every shipment is delayed, … <long rant> …”
“I… bet. Why hasn’t anyone fixed it!? It’s a one line fix!”
“No budget!”
This was one of the biggest logistics companies in the country about to undergo a multi-billion-dollar merger.
Their core business software was missing database indexes even on primary key columns!
Madness.
Usability of internal tools is just never considered. Priorities are driven by management. They invariably just don’t care about feedback from the peons that actually have to use them in their day-to-day jobs, even when it results in tremendous inefficiency.
This also allows companies like Workforce to make billions selling the most godawful software imaginable.
> There is a (very) long tail of truly abysmal software that people are forced to use all the time.
We generally call it “enterprise software,” and it’s usually abysmal because the people buying/funding it are VP-level executives who care about compliance/reporting/visibility and who never have to actually use the software.
This actually comes up on HN pretty regularly. Or used to; like many topics it’s taken a back seat to AI.
...and now with AI much more of that dreadful software can be built with all the bullshit featuritis that said managers who don't use it think they want.
It's similar with AI-generated writing: AI can't transcend mediocrity, but it can elevate terrible writing or terrible code to be merely mediocre.
none of these types of pieces of content focus on what was created for users, its always about how sky high their "shipping" metrics are (PRs merged, LoC, etc). it makes it extremely hard for me to care, as they all boil down to "im a dev who made some loops on loops on loops that writes a lot of code and spends a lot of tokens."
i will say, at least this article says "we merge 5x more and our revert rate has stayed the same." so thats at least a metric i can start to respect
Any good product leader will tell you, sometimes not building something is actually good for the customer. Slower dev cycles have you the ability to take time with decisions and make adjustments. The obsession with shipping more code is making things worse for users and I would argue, damages the company in the long run.
Eveyone has a backlog. What happens when you clear it in a month? Do you suddenly have better ideas about what you want to build? Do you suddenly have better intuitive designs? Do you have more users or the same users using the product twice as much? In most cases, the answer is no.
AI gives you the speed to execute when you need to, but knowing what to build and how will still remain a bottleneck.
Now the real question is "is the revert rate the same because you still have the same process, or do you fix-forward more often now, so your revert-rate is because of higher-severity incidents?"
The way we do this is to have period "post project" metric checks, looking for
- are the users using the new feature (repeatedly)
- what sentiment is associated with it on customer calls (extracted via call transcripts, but ideally the relevant engineer joins the call)
I mean we (huge tech company with ~2000 devs) literally never actually revert anything, in my 2 years here I have quite literally never reverted a commit. We have a staging/release cycle that means nothing ever needs a revert, you just fix issues in followup commits on trunk
This is another one of the infinite meaningless metrics that AI cash burners love to use without context
Don't you ever merge commits that wreck CI and are too complex to fix immediately? That happens here all the time.
When that happens we revert immediately. But to be fair, this kind of commits have to be looked at in detail, and you often need the CI export because those yaml files are a pain to write...
Ehhhh:
1) You can merge a "new change" to fix a merge, instead of reverting
2) It's unclear whether the substance of their individual PRs are the same. Never thought I'd want an LOC metric.
3) Really, I'm going to double down on the idea that merges could be fixing merges. Also happens to increase the merge metric.
Is your company making 5x more? Are your customers 5x happier? Are you getting paid 5x more?
What’s the point of doing 5x more?
The “more features != better” is still a valid principle IMO, and AI doesn’t help with that. Probably is making it worse.
Perhaps it shows that it does not matter how good your engineering is, or if you have sped up feature development 10x, if the shop is not providing value then who cares?
You're thinking about this at the wrong level. This org has made the decision to be AI-native. They aren't A/B testing being AI-native vs not being AI-native. That's the kind of thing "the market" does; if a company's markup is higher than another's it can attract more investment or its equity can trade for higher pricing on public markets.
I don't think it makes sense to A/B test "AI-native" vs other regimes of development unless you are a large, decentralized company like an F5 company or a Big Tech. How will a team of 25 (the size of the company if you read the thread) have enough output to even A/B test the difference?
It was the article that used metrics related to engineering velocity as evidence of success. It makes sense to ask how this is impacting their business metrics. The company is not in the business of shipping more code. I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
> It was the article that used metrics related to engineering velocity as evidence of success
Success for what? The article opened with: "Last week Will Larson (@Lethain ) asked how leaders are evaluating engineering managers for AI skills. My reply , a list of questions to think about, got a lot of bookmarks. More broadly, I've been thinking about how to build an AI-native engineering org for a while. Sharing some lessons from the last nine months of building at FurtherAI."
The opener is "how to build an AI-native engineering org" and the context is a company of roughly 22 engineers (from a reply.) Nothing here says "evidence of success"
> I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
In an company of 22 engineers, how would you make these measurements? Would you split your company into a velocity focused group and a quality focused group and make them compete? How? What would a customer motion look like? What would these groups share? This makes no sense. This is like asking a small company to A/B test teams that used a computer vs didn't use a computer.
A larger, more mature organization has the luxury of splitting business units into AI-native vs not-AI-native workflows and comparing COGS/business objectives against each other. I know of very large tech consulting services who did exactly this around the beginning of the cloud era to figure out which aspects of their business they wanted to place on clouds and which aspects they kept on-prem, and even then that was messy because part of the value of on-prem is the scale benefit you get by owning and operating your own DC. But no 22 person engineering team I saw A/B tested this out. Companies were founded either with a cloud-first philosophy or an on-prem philosophy.
If you can’t tell value at a 25 (or 10 or 5) person org, you will not be able to at a larger one.
If you weren’t making intentional decisions about cloud vs on prem… why? I worked with many small businesses that did.
This feels like a value-divorced outsourced engineering mindset. You have the power to understand if things are good ideas or working!
Idk, seems like it’s important to A/B test being “AI native” because it could go badly.
This is a 25 eng startup. Anything can go wrong here. A hike in interest rates can lead to a down round causing the company to go into cost cutting mode, leading to a decrease in product velocity and prod incidents as infra is cut to the bone, in order to cope. Your top engineer can get sick and quit and your next in line isn't strong enough to keep the ship going.
I'm curious if you've ever worked at a startup, tried to invest in a startup, or considered purchasing a startup's product to know what it's like? The whole point is you're taking a risk trying something new and different. If you're just doing what a much bigger, established company is doing, you're fighting against an incumbent who already has much lower COGS than you.
Taking a bet on being AI-native as a startup is taking a bet that use of AI will result in lower COGS than the incumbent. As the old saying goes, "Nobody got fired for buying IBM", and yet Amazon was able to crack into that space by doing something novel. If Amazon was just IBM, why not buy IBM? The ultimate test of AI-native business models will be if these companies become the Amazon of today, or whether we'll write about them the way we wrote about "fifth generation computing" in Japan.
>> I've started largely using this site in a read-only fashion. The SNR here now is too low and I'm too busy. Found my time better spent in more focused offline and online communities.
How's that going for you?
Better. I've implemented a system to block commenters that I don't like since I wrote that. You're about to go on that list, thanks.
They probably did not even think about asking customers.
why talk to the customer when a chatbot can do it for you? your obviously not thinking ai-native... /s
since when does an engineering org care about revenue or customer satisfaction? Most software devs have no idea how the business they work for operates and think their direct deposit every two weeks is a fundamental law of nature. Also, many software devs are anti-customer and think customers are wrong even when the customer is talking to them one-on-one describing their needs in plain english.
I'm sorry you had to work at (or with) a crappy engineering org. Good ones really do care about the business.
Funny you should say this… there another post on HN about engineers using AI are going to make PMs redundant!
LOC has no impact on revenue whatsoever. We've known about that for decades.
I've gone through a few recruiter screens lately and the number of people who mention "number of deploys per day" as a metric they are proud of causes my eyes to roll.
What can you possibly be doing as a product that justifies you releasing 60 times in a working day?
It's certainly better than the old models of releasing on a quarterly cadence with the test team and all that, but it feels like the metric has become the goal when you are releasing that often.
This was the case at a previous job. Someone read some "state of devops report" or similar BS where they wrote that good teams ship often. They took from that it was shipping often that made good teams... And as a result we had some contrived deploy process on Slack with buttons and emojis and what not to deploy. Not a single customer saw any value out of this of course, because shipping something is not the same as shipping something valuable.
So I guess releasing 60 times in a working day proves that you're part of a good team /s
That's why I'm sceptical about the value of ai in software development. Much more can now be done but I don't see costumers paying more for it. The only way it can work is by laying off software engineers and saving money on their salaries.
Tend to agree. AI has failed to produce more business outcome. The only thing left is cost efficiency by reducing headcount. Thus 5x more productivity means needing just 1/5th your current staffing.
and IME the #1 rule for us is work in a profit center, not a cost center. When you're cutting there's a law of diminishing returns, but the floor is so incredibly low you don't want to be near it just to clear the blade.
This post is funny to me, because I feel it falls well short of what a proper, forward-thinking "AI-native" engineering org would actually look like. That is, the author is presenting this as kind of a radical departure from normal engineering, but I just see it as normal engineering with more AI. I don't see that as AI-native.
To me, true "AI-native" engineering will be one where questions regarding how humans will do things like review code are irrelevant. You explicitly throw away any concept of typical software engineering and, instead, differ to the AI. Your job, as a human, is solely to do what the AI can't physically do. That's not the same as closing gaps between what AI is currently capable of doing and what you'd need it to be capable of doing. It's literally enforcing that anything an AI is able to do is treated as correct in itself, and you, as a human, only support it by being a human.
If that sounds outlandish, that's because it is. We don't have such capable AI systems yet. It's possible we never will. But that's beyond the point. The reality is trying to make an "AI-native" engineering org by just layering AI on top of existing processes and systems is futile. If your expectation is that AI will continue to improve in capabilities at an accelerating rate, then these stop-gaps are pointless. You might as well just ignore AI altogether until it is able to completely take over, because that's the inevitable goal, right?
If you're willing to buy that will be possible, then my guess is that it will occur in the form of small scale startups and actual mavericks rather than someone trying to run an existing business who will simply be too risk adverse to go all in when the time comes. Some guy in his basement is more likely to leverage AI "properly" to build an actual "AI-native" engineering org because that person won't have much to lose.
> If you're willing to buy that will be possible, then my guess is that it will occur in the form of small scale startups and actual mavericks rather than someone trying to run an existing business who will simply be too risk adverse to go all in when the time comes. Some guy in his basement is more likely to leverage AI "properly" to build an actual "AI-native" engineering org because that person won't have much to lose.
FWIW I think the company writing this post is one of those small scale startups. But that being said, you're not the only one who thinks this. A decent amount of VCs and funds think that only smaller, newer companies have the potential to be truly AI native because there's too much organizational inertia in an existing business in keeping its existing human processes and so only a newer company can achieve the true "scaling potential" or using AI to scale without human labor.
I think this is all fair, but somewhat just semantics.
Most engineers are working in "normal" engineering orgs, but can feel the ground underneath them shifting, and companies believe that if they are caught flat footed, they will be out competed in the marketplace, so they are all trying to make sure they are adopting it at least in a way that lets them keep pace if not accelerate revenue.
If you are not impacted by economic incentives, you can continue to not care about this transitional state, but most people do.
Are there any companies outside of the "AI shovels" business that make more money because of AI? Every time I ask someone why they went so hard on adopting AI, I hear the same story about the fear of losing the market share or becoming less competitive. Honest question, who is benefitting from all of this? The revenues at majority of companies seem flat at best. There is a growing concern about professionals getting deskilled, the job satisfaction is not improving, wages are stagnant. Mind-boggling that we are actively engaged and often push for this experiment that hurts our own profession. Cheering for our own demise.
Not keen on outing myself by providing evidence, but I think we do generate more revenue by using AI. We launched new products with a small team and also optimized our outcome-pricing based product using AI several times. We also reduced costs by self-hosting more software.
But we're a tiny baby startup, so things like "spin up a new product with 1 engineer and get to $1m in revenue on it" or "reduce our IaaS bill by 300k/yr" is a meaningful contribution for us.
We're also building new AI-enabled software that was not possible before.
The big breakout hits have been AI Infra, but that makes sense from a technological perspective, Cisco made a lot of money well before Google did, but Cisco is not where the money concentrated in the end.
We don't really know what the equivalent of Google as an application business will be in the AI era, but there's a pretty big chance it's not actual Google. But Google still has to do it's best to hold on.
I use AI compatibility as a metric when evaluating new software to add to my stack. If an AI agent isn’t able to do most of what you can do in the app by opening it, I prefer to look for other options.
Does not apply to everything though. E.G: I don’t care about AI support for my non work chats/email, since I’d rather not give it access.
I'm struggling to see what any of this has to do with the parent comment. Maybe I'm having a brain fart.
Reminds me of a few things, like being a meat proxy, as well as books like Company of One or Million Dollar One Person Business, and these were published before AI so right now we will see companies with only one founder and hundreds to thousands of AI agents. There already was that drug delivery company with the one founder but apparently a lot of their revenue was fraudulent. Still, it proves the point that it is possible.
If most of their revenue was fraudulent it doesn’t prove anything. I’m not saying it’s proof against, but it’s definitely not proof for, either.
I was misremembering, they were sued for spam laws, not for their revenue being fraudulent so it seems it's legitimate, making a billion in revenue with one person (and later on his brother who was hired), the rest is all AI.
https://www.nytimes.com/2026/04/02/technology/ai-billion-dol...
https://www.businessinsider.com/medvi-ai-weight-loss-million...
IMO most modern engineering practices are cargo culture. I’m sure in 10 years we look back on today the same way we’d look back on the days before version control was common. We are in the emailing files around stage of development
> differ to the AI
defer to the AI
> I feel it falls well short of what a proper, forward-thinking "AI-native" engineering org would actually look like. That is, the author is presenting this as kind of a radical departure from normal engineering, but I just see it as normal engineering with more AI. I don't see that as AI-native.
A company which proclaims they are "AI-native" isn't actually interested in engineering, they are trying to ride the hype wave. Companies who care about engineering just... do engineering, and don't make a big deal out of whether they are chasing trends.
AI native shouldn't imply idiotic. And designing your execution approach around a strategy based on a technology that doesn't exist is idiotic.
The reality here is even more idiotic: someone criticizing a company for a terrible strategy they aren’t using that wouldn’t work if they did. According to the critic!
If you follow this logic, we get to the true "AI-native" engineer which is ecstatically floating in a bath of dopamine-stimulating liquid, while AI reads their brain pattern and instantly builds everything they didn't even form a full thought about.
"If that sounds outlandish, that's because it is. We don't have such capable AI systems yet."
It this point, a whole lot of teams have been leaning into AI-first, not-entirely-reviewed code since around March this year.
That means we are already 6-8 months in, and not far from having a full year of experience of what that actually looks like, albeit weirdly influenced by the increase in model capabilities during that period.
I'm looking forward to those retrospectives. Did this turn out to work OK for the teams doing it? What were the biggest failings and/or surprises? With hindsight, did this produce good quality software?
I think the biggest shift is in predictability. Will something be done today, next week, or next month - nobody knows. Will we converge on a bug free state? not sure.
So far AI works great at fast moving work which you will probably recode or throw away in a month or two anyway (e.g. a startup) but is weak at maintaining a large code base that you rely on. At least part of this is cultural, as the cost of misalignments between team members can be quickly translated into broken software.
I don't disagree but
> Will we converge on a bug free state?
I have never witnessed this state in my two decades of software work. Why would we think AI will magically achieve this?
Software robustness, at least in terms of "has the app crashed on any client machines today", is achievable. It's just expensive enough to achieve that most people (rightly or not) don't prioritize it, and the more moving parts (like third party libraries or language runtimes) your application has the harder it is to achieve. Maybe AI will lower the cost of achieving that.
When I worked on Guild Wars every server or user crash went to a shared inbox and was bucketed so we could identify and act on new crashes. IIRC we maintained very, very low production crash rates and could prove that many of the crashes were due to hardware faults.
Similarly, Mozilla tries to analyze and act on every crash report they get in the field. If a webpage visited by even a small % of the customer base was causing crashes, the team worked to try and fix the crash to deliver stability. If it was a high-traffic webpage it got treated as a high-priority task - I remember Farmville breaking was basically a 3-alarm fire back when it was extremely popular.
"No bugs at all" is an even higher bar, but it probably is possible for simpler applications with the use of stuff like theorem provers. I'd love to see someone try it in production but I think the return on investment for that is probably super super negative.
I operate on a "no known bugs" basis with quite a bit of effort invested to try to find new, unknown bugs.
The idea that every crash isn't acted on is kind of a crazy one, to me. Not every crash is actually actionable in e.g. a mobile app, but every one should at least get an eyeball in my opinion.
This, to me anyway, is the bare minimum of "caring about software quality". If you do this regularly you'll have very solid code, and it's interesting that I have the perception at least that Guild Wars was a quality product.
There's another layer, where you do that for subthreshold issues like UI inconvenience, etc, but that's bonus, I think.
having spent thousands of hours in gw1 and hundreds in gw2, I don't think I ever had an actual crash, and only very minor gameplay bugs. congrats :D
For all the talk about AI making teams super productive, I don't see much talk about AI improving software quality.
It's a shame, though, because I believe there is a lot of potential there.
I mean, quality is easier with the right tooling (monitoring, linters, property-based tests, theorem prover...). But, given limited resources, it can feel difficult to prioritize investing in quality when business is pressing devs to ship, ship, ship.
But now the cost of writing custom tooling has changed. These custom tools might be sloppy because vibe-coded but we wouldn't have them otherwise! Or we might finally have decided to make that investment after too much time suffering from their absence. Now the cost/benefit changed, so why not invest now?
> But now the cost of writing custom tooling has changed. These custom tools might be sloppy because vibe-coded but we wouldn't have them otherwise
There’s a lot of good tools out there that are not used, either because of ignorance of their existence or lack of expertise. An hour of config can pay real dividend later.
And sometimes you just use some unix utilities and hack a bash script while testing things out.
After 15 years on the field, I still lack a lot of expertise. Many teams do, as you say.
LLM can save hours of doc reading for non-critical tools that I would otherwise not use. Its also pretty good at "hacking a bash script" in order to test things out. Not everyone is confortable writing bash scripts.
Have been at it since Nov 23rd 2025 with a first commit with Opus 4.5 - it was rough at first but after about year well worth it. You can also tell where so many just get stuck in the mud due to lack of direct experience with all the engineering silos (from bare metal sys admin to QA eng silos outside of standard software engineering core) it takes to roll out a new kind of technology company while scaling a new kind of commercialization model, also managed by AI and AI native from the ground up.
Working at an "AI native" company and these questions and stats are far too hand-wavvy for me.
What works: - Empowering engineers. Giving engineers more trust, ownership and product decision-making leads to more engaged engineers vs. micromanaged tickets. Engineers who are treated like owners tend to notice cross-cutting problems and push to get them cleaned up. - Hire for diversity. Not race or gender, but diversity of thought. Different ages, backgrounds, approaches to AI, focus on frontend/backend, etc. Race/gender diversity tends to come along for free. This is a huge edge because every LLM is spinning up the same next token prediction, so approaching problems from different perspectives gives more robust coverage across your codebase. - Flat org structure, low ego. Interns and 20 year vets work side-by-side and teach each other things - hire for openmindedness. - Communicate about product. What should it do? How do we want a feature to work, and how to handle certain edge cases? If the product is well understood, the code follows. In-person/hybrid is valuable. - Architecture matters more than ever, but it is easier than ever to fix. Listen when engineers mention tech debt. Notice when a simple change touches 27 files. Senior engineers shine for this, because they tend to have a better nose for code smell. - Reviewing still matters. Anthropic may throw tokens at reviewing, but it is a great way to share ambient product/code knowledge across the team. 200 engineers throwing code against the wall is chaos and leads to YOLO codebases. You can move fast AND keep a rough mental model of the codebase intact.
Been moving our 17-person engineering team to adopt ai and this comment summarizes all of my findings perfectly.
This specifically summarizes a huge gap I think some people have around ai:
> Architecture matters more than ever, but it is easier than ever to fix.
You can prompt away to your heart’s content, but if you’re not pushing it in the right direction architecturally, you’re bound to face frustration eventually.
The most successful way I’ve found to convert those leaning anti-AI or AI-skeptical on my team is just showing them how much tech debt I can clean up with AI. They get excited to pitch in, and they see that it does actually benefit them in some meaningful ways. That set of endpoints that’s been neglected for years? Let’s stop complaining about them and DO something about them, today. I’ve actually seen life be blown into some devs who were coasting for a while. It’s an interesting time.
This guy knows what's up. Favourite comment.
I was just looking at some user-facing error messages Claude generated for me which contained unnecessary/irrelevant technical details while failed to provide clear instructions or fix.
Earlier, I reviewed code that does way more than specified scope, changes public API and introduces new behavior. I had to almost redo the whole thing.
I thought to myself, no way I am going to ship the code as written without actually reading through it (or as some people put it, only looking at the "intent"). I'll get more complaints from users while having absolutely no idea what's going on with the code or actual behavior.
It's a puzzle to me how these orgs can pull it off.
It really depends on the kind of product you're working on. A lot of developers (and managers and business leaders) assume that their workflows will apply to other kinds of products; and are often very wrong.
For example: A few days ago I saw a quote that over 90% of current Y-Combinator startups vibecode their products. The problem is the author misunderstands context: Startups sacrifice quality for time to market. In other cases, if you're building something that isn't technically challenging, (because it's 80% similar to many other products out there,) you can probably vibecode it.
In contrast, if you're doing something technically challenging, it's to your benefit to be a lot more hands-on. I recently started getting excellent reviews from copilot of my hand-written code. (Or, I should say, the hand-written parts of an AI-assisted pull request.)
I think it also depends on how well defined requirements are. I can have Claude nearly one shot a Terraform provider but it's just taking 2 API contracts and combining them together (the provider contract and the API you're automating)
On the other hand, some things are incredibly product intensive and poorly defined and it seems LLMs will latch on to an arbitrary direction with much lower success rate.
It is also worse in "brown field" where there's lots of tribal knowledge and technical debt. Two systems may both exist that look very similar but oftentimes there's the one on life support that almost no one uses or cares about and the LLM doesn't know it should largely ignore that until it's thoroughly documented (which can be a significant undertaking)
The people / orgs that can actually codify their strong software engineering principles and practices into "a harness"[1] are the people / orgs that find success with these tools.
The rest are just marketers.
[1] Whatever that means to them
Even then, the actual development of a harness for interesting problems is a difficult task. It requires more skill & experience than people realize, and gets harder as you scale things up.
Yes
> no way I am going to ship the code as written without actually reading through it
Interestingly agents can help, not replace, this process
These tools will make us more productive, they will disrupt (have disrupted) our industry, but they will not replace us because they cannot
Spent few minutes reading the infomercial -- regretted it. Engineering principles form the bedrock first, and AI-tools are made to fit in them, not the other way around as this infomercial propounds. Its linkedin vibe is a big turnoff too.
Thanks, this is exactly right. AI is useful insofar it can be made to conform to engineering principles.
Related: this article was very unilkely make it to the Hacker News homepage thanks to the clickbait headline, but I recommend reading it anyway: https://twitter.com/jsnnsa/status/2115623382721126826
It describes a team spending 30x payroll on tokens - $3,000,000 in token spend per month - building a multiplayer game where incidents and bug reports from users are passed directly to Fable 5/5.1 agents who then act on them and deploy the changes to production.
"We consider it an outage if we don't ship every hour"
It's worth reading because it's either a complete disaster or a glimpse of the future, or quite possibly both.
well, i did go check out the company spawn from the article. its certainly interesting how hands off their process is and how much they ship, but holy shit the games were fucking horrible. there was a direct WoW ripoff (right down to the assets) that was unplayable, a racing simulator where my controls stopped working 10 second in, and some sort of RPG that i started and just never loaded. so i will take what they are saying with the heaviest possible grain of salt.
Yes exactly this is my standard for everything now. You can spend an unbelievable amount of tokens doing all kinds of things that are interesting as processes in their own right.
But is the outcome good? Is the game fun?
at the end of it: is anything of value delivered to customers? the company might say "well users make the games with AI, we just serve them and host the multiplayer." okay, but again, what value was created? they brag about being one of anthropic's top customers; congrats you are handing VC money to anthropic. why should i care? theres no answer for why anyone should care, because to them the system and clout is the product, the product doesnt actually matter.
at least thats my read on it. i dont know how else to understand whats going on here.
This is just like when crypto people were pretending their NFT/metaverse games were fun, it's pretty funny.
I guess they didn’t read Simon Willison’s latest post on how to make fun games with AI assistance.
Oh it's Jacob - I used to work w/ him on his acquired startup, Pluto. He was very AI-pilled; I'm not surprised at the results. I remember wading through the AI-generated Python he had written. I think Pluto had something like 500k or 700k lines after 18mo. Everything was duplicated; there was no reuse of anything. Dozens of features, which mostly worked, not all did.
As a product that I would regularly use, Pluto missed the mark for me. As a SWE in 2023, I couldn't meet Jacob's mentality. Today, it's how a lot of us ship software.
I didn't love the product but I have to respect the process: he was able to sell his company. For some, what he did is success.
So as a user I can basically deploy code through bug reports? Sounds like a prompt injection nightmare.
I'm both fascinated and appalled by the metagaming implications. Can I guide fable into subtly rebalancing a game in favor of my preferred play style?
That actually sounds really cool. Assuming it has access to a well-tuned predictive analytics mechanism that allows it to derive that changing the game in this way would increase satisfaction and revenue from you and other players like you, sufficiently to offset development costs and dissatisfaction to others, then it really should.
We are already - as in today - doing this with friends while playing Valheim, the process is semi-automatic with me in the middle as the admin, though.
We make 10+ gameplay changes a day in our custom kitchensink mod that steers everything. 70%+ are bugfixes (we have plenty of mods that conflict/are buggy), the the rest are QoL improvements, small new features and rebalancing. If we see some playstyle is too strong or boring, we spice it up the same or the next day. Mobs too weak/strong? Update the same day. A ton of performance fixes with actual in-game benchmarking. All this while I keep playing.
I built sort of an automatic modding pipeline (including public release and all) with parallel test game clients + servers used by LLMs to develop and test the changes and then merge before releasing a new mod version. We have had very few issues, I'm still shocked how reliable this is. They follow guidance well and test everything in real game clients on wipeable clones of our live world and our actual characters. Josh has a bugged chest at his place? The LLM will reproduce it, develop a patch, yes, but then also log in as him to his actual in-game base and test the fix using game inputs + OCR. At the end I get a .html page report on each change with screenshots and technical details if I ask for them. I don't care if it takes 4h to do a simple fix because it's not my time and game stability is critical. And up to 8 of them can work in parallel, so there isn't a bottleneck caused by excessive testing/re-testing.
I think this will be the future of gaming for many sandbox/easily moddable games. It's a bit too technical at the moment for regular players to handle the updates and models' quirks, but we're only going up from here. Just crazy.
I could definitely see potential in an async co-op game based on this idea. A sim harness, some pbt to avoid broken states, and something like a modified nomic ruleset might get pretty far.
https://en.wikipedia.org/wiki/Nomic
whoa like an ai calvinball. Or, you compete to get the right to one prompt that changes the rules/physics/map etc. Whoever gets to the genie's lamp first …
haha i want to file a critical sev1 "my bank account no longer has a billion dollars in it. please fix asap"
Their argument seems to be that Fable 5/5.1 are smart enough to avoid being subverted in this way. I'm skeptical!
> avoid being subverted in this way
In the instances where they were able to detect it. How much do you think they pour over individual traces nowadays to search for these things?
This is amazing and somewhat insane. It's more like they are building an organism with an immune system than a code base. Interesting that they say only Fable works for them.
I’m building the same but with a malleable SaaS. Basically when you sign up, you get the software deployed and an agent that can control that deployment and code.
But the process is slow. The user wants to add a feature to their app so they ask the AI. The AI has access to a terminal where it can clone, fix, and push. This takes time and isn’t instant.
Once AI can abstract this part, software will change forever. It’s what this post alludes to, does code, PR, etc need to reinvented for this?
Once so, no one would ever use cookie cutter software because you could build your own and maintain it using just an agent.
> because you could build your own and maintain it
One would argue that the effort to build and maintain it is why we have cookie cutter software, agentic built or not.
Yes but now that cookie cutter software can be the starting point. I’m not saying to build it from scratch. There’s probably an open source alternatives to the software you are trying to replace. Use that directly or tell AI to clone it and build it in a stack you understand.
Before AI, the businesses that used Zapier and automation were able to pull ahead of businesses that were still doing things manually. A business that has a custom built software with an agent that can change it on the go will provide even a greater advantage.
This is a complete and utter disaster. They are making spam games. Like miniclips but for spam. A semi functional website to make what is basically youtube playables.
And it's devouring tens of millions. I know what tens of millions can make in the video game industry, it can keep a small team of good devs and designers and artists making something special that's _really_ enjoyed by tens of thousands of people.
We don't want more games, we want more great games. This will never, ever, make great games.
https://www.spawn.co
Look on ye mighty, and despair. I really don't care how this sausage was made because the only thing you need to know is that it shouldn't ever have been made.
As a (light) gamer for two decades who has been reading about the gaming industry for the past few months, I cannot understate how hard it is to create a good game, let alone a great game. So much money was spent/wasted on hopeless endeavors.
These people have no idea what good games mean.
Was thinking of that exact article while I was reading this, we must be following the same folks. Highly recommend folks reading it, even if it feels insane in some regards it's a fascinating look at what the limits of this stuff could be and I found myself with a lot of new ideas to consider after reading it. Interesting times.
Edit - after writing "we must be following the same folks" I realized that you are, in fact, Simon Willison - I saw this article because you posted about it on X, haha.
What's their revenue?
Do players like it?
Will the history of gaming remember this game?
Or it’s a fabrication of some kind. Some times people lie to make themselves look good. Or the omit important details.
This works as we've been doing it since Opus 4.5 (Nov 2025) with every user request, feature and bug. It's the future without a doubt. The feedback loops evolve and get stronger depending on how well you fine-tune or train your system to optimize on its own.
That’s the dumbest thing I’ve ever read. Spaffing millions per month on this slop is the kind of thing people suffering from AI psychosis do. There is no way you need “thousands of agents” to build any product.
The more agents you throw at something, the more they collide. Those collisions generate more slop, which requires further slop processing agents.
so if there's no user reports and fable isn't shipping in the middle of the night, then it's an outage?
> A default that made Savi think longer on every message, landed at 10:22pm and reverted by 10:40pm: that's not right, undo it it's too slow and not jazz.
I giggled on that one. It feels like children playing having a job. If they pay that much for tokens they have to have at least 80k customers paying 50$ a month. Seems huge for something like that. Or they have a lot of investors. Anyway does not feel like something going anywhere.
Are they even making 3M per month?
Isn't there a point where you're shipping too fast that the product hasn't even caught on enough yet to know what to build?
I looked at some of the games and it just looks like a Roblox clone
I think they're making 0/month, spending VC.
This from that tweet stood out to me: "In general, we've seen that OpenAI's models game their own checks"
In the past when working on agentic stuff I've observed models gaming checks aggressively, to the point that when we were validating log files to see what the agent had done, the agent responded by editing the log files to pass validation. I can only imagine how bad that kind of behavior is at scale, and it sounds like they ran into it.
"We consider it an outage if we don't ship every hour" is the dumbest thing I've ever read.
I really mean this. Does it benefit the business? Does it benefit the users? When you optimize for the wrong thing, you inevitably fail.
That’s an uncharitable take. They didn’t say anything about optimizing for it, just that it was a signal for them. Maybe one of the signals.
some things are just dumb on their face
Maybe it's just me, but "How much we ship per engineer" doesn't seem like the correct metric to track.
If that's the benchmark, developers are incentivized to produce over-engineered and bloated code.
Categorically, if you ship more bugs per engineer, you fix more bugs per engineer.
Won't there also be a point were you're shipping so fast that that in itself deteriorates UX? What if a page literally changes every time I reload? What if it changes mid-checkout process and the deal I agreed to 5 seconds ago does not exist anymore?!
You already see this pre-ai with apps asking me to update everytime i open the damn thing…
I have come to realize that code (and I think possibly productivity in general) being expensive was always a feature. I am sure that a few great ideas did get lost to obscurity due to lack of access to resources to build them out. However, we now have an absolute deluge of complete and utter shite.
We didn't need LLMs to come up with the genius idea of making every desktop app a web browser.
Software quality was in terminal decline already, the only hope is that LLMs get good enough to dig outselves out of the several miles deep hole that has been dug.
the root problem isn't the code though; its the business and how its run that affects it, llms will just make it amplified imo
It was trained on that declining quality. Now it's just digging faster.
Agreed, for the individual LLMs can be powerful to let less technical folks build and customize their software - at an org level I reckon we'll start seeing more sinking ships as that bottleneck was actually helpful at limiting the flow of lackluster features to prod.
Yep and there is no real demand for slop engineers are producing anyways.
yeah shipping widgets and shit, not real features
I see a lot of builders (myself included) falling into the trap of focusing on the wrong things. Development velocity doesn't matter per se, what matters are things like growth in profit or traction. It's really easy to conflate automation/systems/process with actual business results. Movement ≠ Progress.
> None of this was free. Through the summer, much of our engineering effort went into tests and tooling, and customer-facing changes per engineer stayed between 17 and 30 a month
This is the reality I think for a lot of orgs and is absorbing a lot of the productivity gains. The gap between vibe coding a green field app from nothing which is inherently "AI native" and implementing AI tooling into complex orgs with large amounts of legacy infrastructure, integrations, ingrained practices etc is enormous. For us, it has been a big effort to enable agentic coding over a significant footprint of our codebase. And all this is happening while the plane keeps flying, with near zero tolerance for defects in critical systems. There is a lot more to it than you will understand if you just look at twitter or Hacker News.
I remember optimizing a huge build that took days. We optimised it to only hours on a cluster - it took years to get this right. We realised at the end of this that the packaging system (handled by another team) was taking about 2x longer than the build because there was one particularly huge zipfile that sat there compressing for ever and ever and ever while the rest of the cluster was idle.
As in this story we blew enormous effort optimizing something that was very difficult and because of our blinkers (and our management's blinkers) we missed that it was only able to achieve a limited result. We didn't look at the whole system. In the end it was never fixed because......siloed thinking...or something.
> What it cost us
> None of this was free. Through the summer, much of our engineering effort went into tests and tooling, and customer-facing changes per engineer stayed between 17 and 30 a month.
What it cost you is that you helped the AI labs become better engineers. At a certain point the Big Flip will happen, and the AI labs will stop their offerings and instead start competing with the engineers.
This assumes that it’s more profitable for these labs to compete with their own customers, rather than increasing the productivity for their customers and - assuming this increase justifies it - increasing their profits.
I don’t think it will ever reach such a point.
This guy knows nothing. A few years experience plus a Claude account...
https://xxcancel.com/sgondala2/status/2115832070274723875
Is there any way to read any of this if I don't have a twitter account?
For the argument lets say that this approach works and results in high quality apps. But will be users happy that app is updated 3 times a week and all updates add major changes? Because code generation is so cheap and fast it means we should even more thing about what to build to not overwhelm our users and actually solve their problems.
In my best impression of the slapchop guy:
First you take your Blendtech and "will it blend" your engineers, then upload the result to the cloud inside what is called an S3 bucket (don't forget to set access permissions on your buckets kids!) to be digested by the clankers, sit back and profit!
Dark humor aside I think disconnecting a bunch of problem solvers from their computing equipment could really enhance the world, putting that analytical power into the world in person could move a lot of cogs, rather then marshalling JSON.
Nice work. How do you handle swapping in a new model version, hot reload or restart? And what happens to a batch that's mid-run when it changes?
I know many (big) places that are already back-paddling. From token leadership boards (more = better) to now saving on AI cost, because the benefit was less than expected. As if this wasn't obvious. Sigh.
Sure MS, Meta, etc, are still laying off engineers to have money for data centers. But I am 99.9% sure that they will thoroughly regret that move in less than 2 years.
We shipped more code! We merged 5x faster! We spent more tokens!
>> Every time we sped up one step, work piled up at the next... we started going through the entire process of shipping software, one step at a time...
I feel like I'm being gaslit here. You can become CTO with this lack of business fundamentals, and a basic misunderstanding of systems engineering?
(in fact it may be a requisite ...)
Can we ban twitter links? I have to use a hack to even read it and it’s always just slop. If you can’t be bothered to write an actual article, why should we bother to read it.
Here we are again, thirty years after I was taught the "waterfall" paradigm at business school.
When asked about "customer satisfaction" we were told, awkwardly, that became "political"
The paradigm did not survive its encounter with reality, and I do not think this one will either
The "only" 175% increase in productivity is very impressive, and seems realistic to me, when you remove the chores of writing code (thank you to coding agents)
What you build is still very dependant on what you need, and you do not know that until you build it
It is a paradox only for the "waterfall" paradigm. We rejected it, in the 1990s for a reason
It did not work
why would you want to do that?
Slop.