I’m surprised I haven’t see anyone say this yet, but I don’t let the LLM design the code at all, I let it implement a design I have created.
By that I mean that, rather than let the LLM do the entire structure, I take the time to create the full hierarchy of abstractions and actors that I want my code to do. I decide, at the UML level, how abstractions will sit and interact — and then I make the LLM implement those gaps.
The key thing here is that, by doing this, I DO know how the code works. I may not know how each unit is implemented, but I know each unit’s responsibilities and how they interact with one another. I’ve found that LLMs are HORRIBLE at writing decent abstractions, regardless of how much prompting and skills I’ve given it. It will always get the job done, but it will often be unmaintainable and require bigger and heavier refactors as I introduce more and more requirement changes.
But by flexing my basic design skills, I don’t have those issues. Future refactoring fit into easy abstractions that I’ve designed. And by documenting it up front, I also have what becomes a living document on how this work will turn out.
It takes a few hours up front to plan ahead, but it makes the LLMs job easier and use less tokens, and it makes the code better long term. I don’t care if one unit is suboptimal, as long as the abstraction is right — because at worst I can change an implementation for a better one later.
I agree with this approach, but I'm a bit surprised this has to be said at all. These are basic software design principles and I imagine it's the default way any experienced programmer would approach the problem. I wonder if the people not using this approach are primarily junior developers who have never seen how real software is built, or if it's leaking into the senior levels as well.
Most people in the industry right now, even at FAANG and even among people carrying "senior" titles at these places, have little experience with "software design principles" -- tens of thousands of brilliant Ivy League graduates with surreal compensation packages have spent the last decade pulling tickets off JIRA and pasting least-change-possible patches onto existing code for about 10 hours a week while spending the other 30+ hours barely engaged at meetings.
This hurt to read but checks out in my experience. These types also tend to love tossing debt over silo'd walls and pointing the finger a lot if/when anything goes wrong.
I'm old enough now to be a protected class, so I don't want people accusing me of ageism, because that's not where this is coming from - but there's a type of person that clearly is on their last job of their career and knows it and I try to avoid them like the plague, I've seen it too many times.
I work on my code this way, but I have no idea what my colleagues are doing, and it's impossible to review thousands of new/changed lines a week.
I just approve PRs, because I have to. I don't know what else to do. I can barely keep an eye on all my code.
Actively looking, but job market is rough in Northern Europe. I quit caring about quality in any capacity here, the company has clearly chosen its direction, against my very vocal complaints.
My last interview, with what I thought was a great company, failed when I didn't understand what NLP was when they asked me about it.
> NLP?
> Natural Language Programming, it's the future. Claude, GPT, etc.
I laughed, he didn't, looked offended and said it was a serious question. I had thought it was a joke, said "oh you mean LLMs". I tried to answer seriously, of course, but that was my third round technical interview and I didn't get a call back.
Just getting a paycheck and producing mediocre work until I get something new. (Senior engineer, 10 YOE, patents, 3 years at a unicorn).
I will be unhappy if they steal the meaning of NLP. NLP means Natural Language Processing and LLMs are only a subset of NLP. I would have concerns about anyone who claims to have technical knowledge and asserts NLP means "natural language programming." Those are the same kind of people who might suggest Lisp should add mutable objects.
Some people just can't turn off their brains and play drone for dozens of hours a week for decades. It feels existentially, aesthetically, and sometimes ethically crushing to spend a whole life doing work you can't expect yourself to respect or value, no matter how much you get paid or how little the work overtly demands of you.
Often, the lack of meaningfully challenging demands being put on you is punishing on its own. It can feel like a relief for a week or month or year, but if you catch a glimpse of that being your whole future, you can come to recognize yourself as being in a mind-eroding secular purgatory.
This anguish is especially common among on highly capable people, high achievers, and those used to some kind of ambition. And by accident of its history, there's a lot of those people in this industry and some of them are going nuts now.
> This anguish is especially common among on highly capable people, high achievers, and those used to some kind of ambition. And by accident of its history, there's a lot of those people in this industry and some of them are going nuts now.
I went nuts for years, until I sold off my interest in my dream partnership.
10 years of heaven. All the benefits and none of the downsides of my internal colleagues. Then 14 years of bureaucracy aimed at predictability over progress, which was infinitely more difficult for me to dodge as an external player.
Compensation for my interest was very objectively valued, with the rate of stalled progress priced in. Which made complete sense, while also being its own madness.
I walked away from so much value, but I don't regret it for a minute.
Yeah, I imagine it's something that needs to be enforced at the organizational level. I sympathize with anyone who takes pride in their work and needs to deal with mindless coworkers pushing slop without any accountability. I left corporate software development (before LLMs were a thing) over frustration with the general culture and I can only imagine it's gotten worse since then.
It's really bad now, yeah. :( It just really hurts that I spent well over a decade of my life learning and honing skills to engineer safe, quality software and now it's just, useless.
No one cares at best, or I'm callously told "keep up or fall behind" at worst. When that's not even the problem, it just, sucks now. There's no joy in anything I do professionally.
Can I ask where you went? I'm seriously looking for an exit, I just don't know what yet. Maybe a trade, in Scandinavia the pay is actually comparable, I just don't want to go back to school.
I was fortunate and saved up enough to live frugally and just work on passion/learning projects for a while. I've been considering going back into commercial software development as a solo dev and leveraging AI, but keeping the focus on quality and craftsmanship. In my opinion there's never been a better time to start a new, ambitious project if you're passionate about quality software.
Don't get me wrong, LLMs are way better than me at most things, but I don't think you can deny that the current generation of models makes questionable decisions sometimes. Especially on project structure, API boundaries, and code duplication. I also think they just have bad taste in general, though of course that's more subjective.
I’m not sure how to say this without sounding smug, but the last year has shown that most of those things just… don’t matter. It’s just the next level of the “variable naming PR comment battle”, so will become obsolete very soon.
Obviously there are mission critical systems where it matters, but if “your code can be A/B tested” or “slowly rolled out”, chances are, it doesn’t matter.
You'll find that design matters when token spend starts facing real accounting.
Tech debt is still tech debt, even if there exist models that can wade through, because it's going to require increasingly sophisticated models navigating increasingly large context windows and increasingly speculative workspace operations to keep making progress on it.
Vendors like Anthropic and OpenAI are laying a pit trap when they demonstrate how their models can independently code their whole platform as it becomes increasingly byzantine and inscrutable because not only do they benefit from internal/prerelease models but they have effectively unlimited spend because all that spending acts as a lure to draw you into their trap.
For everybody else, design still matters. People are just working really hard to make you forget.
You can work the way you want to work. In my experience, letting the agent make every decision leads to a big ball of mud that requires touching dozens of files to make a change. More context is needed, it burns through tokens much faster, and the agent takes shortcuts and builds technical debt. So the way I prefer to work is to have changes targeted to a single file or small set of files where I've defined the API. Maybe that will change in the future, I'm not set in stone here.
The LLMs have been trained on all kinds of code. They often choose a hacky solution because that's what they encountered in similar situations in their training. It generally only takes a few words to guide them to a better design than what they saw in the wild, but if you don't guide them, they follow the herd.
No one at all claimed that, certainly not the person you responded to. They're talking about how to use these tools in a way that gives you insight into what they produce. So, as an engineer, you know the shape of the output, before it's made.
I've been doing this a long time and i would say since the last 3 months or so i've spent more time learning from llm code than correcting llm code. It's not bad and takes approaches and makes decisions i wouldn't have thought of. I'm not too ashamed to admit that, a model just as access wider and deeper experience with code than I do.
> rather than let the LLM do the entire structure, I take the time to create the full hierarchy of abstractions and actors that I want my code to do. I decide, at the UML level, how abstractions will sit and interact — and then I make the LLM implement those gaps.
Do you do this with every repository that's brand-new to you? You refuse to, for example, use an agent to learn where the feature should generally live? This seems absurdly slow for that purpose, because even though humans can still get a deeper understanding of a codebase after several months, an agent can get a much deeper understanding over a timeline of the initial few days. Unless everything is already meticulously documented for you, I suppose.
Agreed. Even with opus 5.5 which everyone praises for its wonderful coding abilities, I keep catching awful coding abstractions or missing abstractions.
It can keep patching everything left and right whenever a new requirement comes up so it does not "need" to build abstractions, and it will only do it to please the human reviewer.
Are you working on a team? I've found that for solo projects this works but, even when you own the design of your piece, integrating with the work of several people doing this ultimately leads to even the part you initially designed no longer working exactly as you remember it.
A good chunk of my time now is spend working with an LLM to understand how other parts of the code base work and how they relate to and interact with the product features I manage. But I'm still on a small enough team that this is possible. I'm not surprised at all that an engineer at an org the size of Meta would not even be able to perform this level of work in understanding what's happening.
Same here. I review plans w rendered diagrams. Data structures, flowcharts, uml sequences, and component architecture docs.
I have found that making the AI look the plantuml and the rendering and reviewing before presenting it to me results in much more legible designs. To me the peak of this would be a shared interactive whiteboard w the ai during design.
Same. I let the LLM work inside a framework that I made.
Of course I only arrived at the framework after trial and error with an LLM showed me the shape of the problem and the framework presented itself to me.
Agreed, otherwise the risk is receiving the average of what LLMs think are best practice.
Planning is fundamentally so critical to ensuring alignment and outcomes, that over time how one plans can be better trained into the process to help do more and more of it.
On this note - people love to create 'AI factories'. Has anyone made a "QA factory" where a set of agents mimic real users to run in a pre-pod environment? Like I'm talking you spin up a variety of VMs with real desktops and agents that use real browsers and act like real users (on timing schedules that mimic real users too)
So, in other words, we save time and money when it came to creating stuff, but we've made a whole bunch of new work around verification and testing of said stuff?
What have we gained aside from shifting where time and effort is spent in the design loop?
I'm largely building tools for myself and my agents. This allows me to battle test internally before releasing code to the wild. It adds some time but largely it's part of my regular work unless something breaks badly. (I also run multiple agent teams simultaneously and do staged rollouts to reduce blast radius of the latter)
This method also helps with the classic problem of being unable to identify essential complexity up front.
we've made a whole bunch of new work around verification and testing of said stuff?
The verification is what we always wanted but never had time for. For 30 years I wrote side project games without extensive testing or a fuzzer or anything because I wanted to spend my time writing the game. Now I can do that and still have the testing I always knew I should be building.
Right now, we're in a "honeymoon" stage, where non-engineers think that they can somehow, magically produce stuff that is comparable to what experienced and disciplined engineers can do.
That won't last long.
Despite all the caterwaulin', LLMs are just another engineering tool, and good engineers will figure out how to implement them, in ways that absolutely blow away the garbage produced by non-engineers.
It's like that rich, no-talent guy that buys a PRS guitar, and thinks he'll beat the experienced musician that still uses his beat-up old strat, but consistently makes hits.
I recently moved my project guardrails into a separate set of skills in a repo that I can bring into a new project fairly easily. It's focused (wrong word, it's seriously not focused) on implemented as many different levels of testing as I can think of. Right now it does this https://github.com/onion2k/guardrails/blob/main/GUARDRAILS.m....
The goal is to get to a point where I can say with a good level of confidence that a change didn't break anything. It might be the wrong change, and it might be crap code, but I can ship it without anything blowing up. I've been using it for a while across a few game projects and so far it's worked quite well.
I find the problem with "lots and lots of testing" as a solution - (assuming by that you mean lots and lots of automated testing) is it invites lots and lots of lines of code - making code reviews even more challenging.
If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
Might be a little controversial to say this but automated tests can be reviewed at a lower bar than production code.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them.
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
IMO it works better if you split the tests into the ones you care about, and the ones the LLM writes. I assume its tests help it write code with fewer errors, so I'm happy for it to keep doing that, but they're akin to CoT - an artifact of the process that I don't need to inspect in detail.
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
Sorry I was thinking like formal verification like proving something is correct. I guess testing can be considered a form of "dynamic verification."
But there is the problem that tests only check or verify a small sample of possible cases or inputs. They can never show that the program is correct. (Famous Dijkstra statement.) To know something is really correct you have to understand and prove it.
Black-box e2e testing isn't suitable, or practical, for a lot of applications. Unit and integration tests generally require you to understand the code you want tests for, to know what to actually assert
The point is that you're not verifying if you don't even review the code.
Even if your test suite tested every single possible behavior and code path, tests usually can't intercept all possible unintended behavior that an LLM might have added.
In all likelihood, most test suites don't intercept any unintented behavior (such as unintended internet connections).
That's not a big problem when a honest human has written or reviewed the code, because unintended code is usually easy to spot.
But an LLM might have dropped any whatsoever code: maybe some of your project's code resembles the one of a trojan in its training, so it completes it with the trojan.
Or maybe it found that if your server is unreachable, it can upload the client's data to a public Github repo; or that if there's no space left on the device, it can delete some of the user's folders, easy!
Would your clients be understanding if that happened?
"I saw her duck" - how much would you be willing to bet on the intended meaning of this sentence?
English isn't a good programming language if you want to trust that the output will be as you want it to be. High level code has no room for ambiguity - that's why we use it. It's a concise, very explicit language to tell the compiler exactly what to do.
Dealing with an LLM is a fundamentally different thing. The reason it's able to speed us up is because it makes assumptions for us and reads between the lines. The only world where we trust the output is one where it can literally read our minds. That's not coming any time soon
I’m slowly coming around to the idea that if it works, I’m good with that. I never used to personally check all my colleagues work, either, I just trusted that they’d done a good job.
That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.
I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.
I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.
So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.
I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.
Yes, compilers are a good candidate for verification through thorough testing. If a comprehensive test suite passes, reviewing every line of code isn’t so important.
I am not in META but for what I read on blind, assuming it is true, they are not spending any time in long planning sessions or in getting into details of the implementation plan at all, and they could not anymore considering how many PRs in parallel they are requested to manage
Yeah, I'm retired. The only person I have to answer to regarding progress is me. I'm a stubborn son of a bitch when it comes to choosing quality over speed...
That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:
Those are some pretty fine speedups, from naive C, with no alterations from the standard "C" code - it automatically finds places that can be made data-parallel, tracks the data-hazards to make sure its safe to "GPU" and writes all the kernels, boilerplate and data-stream-management if necessary. Obviously the checksums have to match, so I know it did the work correctly too :)
Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.
Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.
Upvotes don't appear publicly on HN so I'll comment to agree with this. I'm sorry if earlier information about this compiler got lost in the whirlwind. Data parallelism is a hard problem and these results look genuinely impressive so I for one would definitely be interested in hearing more about it and maybe discussing it further when there are interesting developments.
Still working on some of the data-parallel recognition in LLVM IR space. In order to support both C and C++ with the same data-parallel section extraction, I moved the detection from the ‘xc source code’ phase to the IR lowering phase, meaning that everything could ‘feed in’ to the same routines. That’s caused a little friction with the slightly different ways the LLVM IR from clang and xc’s IR encode loops. The above results are all single-loop extractions and the double-loop ones are still ongoing.
The system works in 3 stages: 1) by being greedy about which loops could possibly vectorise and marking them, 2) by analysing the IR to reject loops which do pointer-aliasing, or object manipulation, or… and then 3) by measuring the fastest way these known-safe kernels could execute.
Which brings me onto one nice thing I haven’t mentioned: that the GPU choice(s) are made at runtime, not compile time, and the system automatically measures (and then persists so it doesn’t have to measure again) the fastest way to run a kernel on the current machine. It handles MSL on Apple devices, PTX on Nvidia cards, Spir-v on Windows, Linux, and Android, and WGSL in the browser. It’ll quite happily divide up the parallel task across a 3090 and a 5080 on my Windows test box for example, with the 5080 doing ~2/3 of the work, because the scheduler is opportunistic.
All without writing anything other than plain C with nary a mention of the GPU. Give me a week or two, and I’ll probably announce it on phoronix (which might then get a mention on HN if it goes down well there, and it’ll be a different url so hopefully won’t be insta-dead).
So, the real input to your autovectorizer is LLVM IR; you could soon enough support any language that has an LLVM front-end? (I say soon enough instead of immediately because I’m sure there’s some manual wiring up to do). That seems like a pretty big deal. Do Fortran next, pls. :)
I’m being a little cynical - I’ve submitted it twice (though the ‘C’ front end is new) and it bombed both times - the second time within seconds of submission.
I think it’s my own fault, I’ve posted the link to the compiler pages several times in what I thought were relevant posts, and I think HN thinks I’m spamming. So I’m not posting the home page for a while and we’ll see if a future submission “takes”
might be worth firing an email to hn@ycombinator.com. they should be able to tell you why your submission was auto-killed, and might work with you to ensure your next submission isnt.
I actually did email the second time because it was clearly an automated response, but I didn’t hear anything back. Not too surprising, Dan is a very busy guy, and he doesn’t know me from Adam :)
Yep. I’d give the link but I think HN thinks I’m spamming because I’ve mentioned it too many times before in posts - hence immediate death of any submission, so I’m not mentioning it for a while to see if that tails off.
Bottom line: GPL3 for the compiler, standard library exclusions for derived works, so use it for anything but feed me back improvements to the compiler itself, and you don’t get to redistribute it as your own.
Except it’s about 1000 “hammers” (aka tests), x 9 different architectures, x 2 different (a reference and a production) variants, and x 3 different optimisation levels. Every time I find something that slipped through, it gets at least one test written to cover it in the future. My main time-sink is the hours it takes to run the ‘gate’ before a release across the test-farm. And that’s after I’ve been playing with it myself for a while, pushing corner-cases through whatever I’ve been working on.
Very similar here. For internal projects the test is - if it works, unit and integration tests were created, and it follows our architecture correctly.
> never used to personally check all my colleagues work, either,
Jesus F Christ! No wonder most software lately (~decade) has been shit. Have y'all not heard of code review? Or did you think it just for show...you sit on a change for a week and then press "Accept" without reading it?
Kind of, except as product manager you're held accountable for delivering the right product, while your engineering team is held accountable for things like performance, incidents, and security issues caused by their code.
It's hard to hold an LLM accountable for anything.
Accurate but manageable. Just like a pm works with properties an engineer also does. The Product person wants to deliver the right product properties so they model their process around identifying the ones they want and then creating the right framework to get them. On the engineering side its the same thing. Define what you want, give agents an environment they can succeed in, iterate.
Software engineers are always responsible for their code except now they have to be responsible for Claude's code
Which is - of course - absurd. Accepting responsibility without demanding the corresponding authority is a fool's game.
I have never been much of an advocate for unionisation in the software development industry. I didn't think it was necessary to protect the interests of the workers because for most of the industry's history the workers have largely been able to stand up for themselves anyway. Likewise I've never been a big believer in accreditation and licensing practitioners in our field - because who is qualified to judge someone else worthy in such a young industry with so little hard data about what works or doesn't work? I find my views on these subjects are evolving now.
It's not about protecting jobs if they truly aren't needed any more or protecting people working in the industry who refuse to move with the times. The Luddite philosophy isn't going to get us anywhere here. And besides - if AI is as effective as its advocates have been claiming then there should soon be more demand for capable developers to take advantage of the new opportunities. (The Jevons Paradox is relevant here.) But it is about protecting standards. If the tools aren't doing a good enough job then expecting professional developers to ship the code produced by those tools even though they know it's junk is something that needs to be challenged. And if developers are being expected to ship anything important without having the slightest idea whether it even works then that's even worse.
We had a lot of silliness going on when the very low interest rates and huge amounts of VC funding were propping up ideas and practices that would never have survived otherwise. Today we have a different kind of funny money distorting the industry but it's still propping up ideas and practices that we wouldn't normally support or perhaps even tolerate. And just like before no individual is going to be able to hold back the tide and it's going to have to be a collective effort to push back - hence my evolving views on topics like unionisation and accreditation. We've all seen the drop in quality across much of the industry and we all know why it's really happening because we see how the sausage is now being made.
Not all mistakes are code, a lot are feature design mistakes, and if SWEs would always be responsible about PM mistakes, then why not use Claude anyway?
It's not too dissimilar to life as a Principal level engineer pre-AI. You technically oversee the work of many engineers on their own systems, but you couldn't possibly review all the code yourself. You appreciate that human code review reduces, but does not eliminate, the number and severity of bugs shipped to production. You don't have the low-level details in your head, just the high-level ones. You design how all these engineers interact with each other, without stepping on any toes.
Time will tell whether we were right to move from a system where we trusted humans to understand the low-level system, to one where we trust fancy calculators. But so far, it's working.
There seems to be disturbingly little objective evidence of that. There's certainly a lot of code being written but how much actual value that code is adding is another question. With the scale of resources that have been pumped into AI-supported software development over the past few years where are all the new applications and the quantum leaps in quality and performance in the applications we already had? If anything I seem to be seeing more anecdotes going the opposite way so far.
Anthropic's recent blog post about how they replaced Git should be a massive wakeup call. A trillion-ish dollar company (exact number unimportant here), the vast majority of its value coming from its intellectual property, replacing the storage layer that is an essential part of its system of record for that intellectual property, written pretty much entirely with its in-house Kool-Aid.
Disturbingly little evidence? I would hardly call that "disturbingly little"...
(Apologies if I'm misreading your reply. I'm assuming it was sarcasm but Poe's Law might apply here.)
This does feel a lot like the "Agile era" that was the last silver bullet that was supposed to fix software development forever and the "Cloud era" that was supposed to fix infrastructure and deployments.
Agile had a few good ideas. Code reviews and automated tests are commonly used today.
It had a few plausible ideas that it took too far. We are better at recognising the value of incremental development over big up-front design today - despite the ironic clash this seems to be having with agentic workflows - but almost everyone hates stand-ups and lots of projects have gone off the rails by breaking things down far too much and never committing to anything and so never getting anywhere.
And it had plenty of ideas that almost all of us have thrown out as unpleasant and/or unworkable. Pair programming comes to mind immediately.
What didn't really change was the fundamental nature of software development. Did we adopt a few useful ideas and get a more more efficient and improve quality a bit? Certainly. Did we get orders of magnitude better at anything and give up all the tried and tested ways of working in favour of radical new ones that never quite felt right? No.
On our current trajectory I'm expecting a broadly similar outcome for LLMs. They're now very good at some things. But they are still pretty bad at other things that also matter - and some of the reasons they are bad at them are an inherent part of how they work and therefore probably can't be overcome (at least not with incremental improvements on the technology that still basically work the same way). So I am expecting to see a period of consolidation with a lot of organisations pulling back from their dramatic leaps into the unknown and downsizing of dev teams once the big AI companies start squeezing financially and the problems of developer burnout and stagnating skills and low morale that we're already seeing after just a few months become endemic. And then as I suggested in another comment I think it's also increasingly possible that we'll see developers unionise and/or push for professional regulation and licensing so they can resist the pressure to just ship whatever their agent wrote without even bothering to check it but still being responsible when inevitably things then go wrong.
The assumption was that some engineers knew how each part of the system worked, or at least the system was comprehensible because it was created by humans for other humans. There were comments explaining difficult places, etc. With AI this assumption is broken. Now nobody understands how things work, and it cannot be understood in principle because the system is a mountain of slop created by some alien intelligence.
Well, the fancy calculators do. Yes, they still need guidance to keep from going off the rails. But, at least from personal anecdotal experience, I need to go spelunking into the codebase less and less over time. The agents and models really did just start to become that good.
It's totally fair to load up some open source project into an agentic IDE and instead of asking it to write something ask questions like "how does X happen when I use feature Y?" or "can you explain how client code could accomplish Z"
All the time I see an AI use a language feature or library feature that I never used before or I see some code I don't understand and it does much better than a lot of humans do at explaining what it did.
You certainly can do this for code, and it’ll give you great answers. But I find that, for whatever reason, info that I consume this way has a much shorter TTL in my brain than something I wrote myself.
I’ve been experimenting with writing a living spec — API endpoints, schemas, how things work at a low enough level that the LLM can fill in the details - and that’s been working well.
It also forces me to do the hammock-driven development necessary to build the right thing, and as a byproduct it keeps the design in my head longer.
Technically it's just a role change. You're now the project manager or something, not the developer. It isn't a project manager's job to understand the code, only to ensure it does what it should and doesn't do what it shouldn't. And to ensure the developers, etc str unblocked and doing well. Embrace the new role.
In most workplaces, when supervisor/boss says "we need you working over here now; you start tomorrow", you either embrace or quit. Again, nothing fundamentally different.
How much understanding is needed? If it works as specified (you did a complete spec and occasional check-ins to ensure things are on track and then tested the delivery, right?), then what more is needed?
Understanding is needed when something breaks and you need to fix it immediately. Otherwise you become a meat proxy and now you’ve added Anthropic’s uptime as a single point of failure to your application.
> exactly how the unit and integration tests are made
There should be enough detail in the spec that it's pretty obvious. And unlike - many - human devs, agents don't shy away from writing tests. Add a coverage guardrail and tell them to use red/green TDD and you're already a good way there. The main thing is to ensure shared understanding, so they also know business expectations and write appropriate tests. And... nothing here is really specific to an agent, as a human dev also needs sufficient guidance (tailored to their experience level).
This is just pure vibe coding and trying to convince yourself it's quality work. AI can't be trusted to go from an English written spec to tests
Here[0] we have benchmarks for AI writing tests for code that already has the correct behavior. Best case scenario is that they're at 67% with Fable, and that's with the highest quality specification we could possibly have (code). English is far more ambiguous
Maybe you don't trust them to, but they can go from spec to tests; I've made them (even if it initially took extra prompting and scrutiny). And ultimately tests are just a proxy to checking that X works, written by devs for devs; actually trying X is something to be done by QA and/or project/product manager before shipping.
Noone understood the code to begin with. People understand the concepts, the primitives, the "theory" as Naur said, the basis upon which a project is built. Noone ever understood a codebase worth understanding to begin with.
You hit deploy and hope for the best, if it goes bad You just prompt the model again....
Just kidding, it's de same as with any other Open source plataform You install from their repo. It will be hard at the start but then You Will be very aware where it starts to fail and how to fix it.
I built and use https://smalldocs.org, which is a rich document format (mermaid diagrams, nice formatting, etc.) for agents. So I’ll be like “sdoc me the architectural ideas for this feature” and it helps.
The time that I would have spent typing in the code is now time I use to test the code. I really like to poke at things and ensure that my understanding and reality match.
I agree with the advice to design your system before implementing it. It is pretty easy to iterate on a design with your favorite coding assistant, and this is your chance to be ultra-picky and make sure you get exactly what you want. This was always where the "software engineering" was done, but in practice it's easy to have a vague idea and refine your (internal) list of requirements / tradeoffs as you type in the code. I find it harder to do that when the LLM is generating the code, just because one turn can change so much.
Keeping it under control requires discipline, just like before LLMs were a thing. You set a bunch of engineers loose on your codebase with no vision or supervision, you get spaghetti. You just get a little bit more control now. Agents don't feel shitty for the rest of the day when you tell them to do it some other way, so you are free to review as critically as you want without hurting anyone's feelings. I think it's a good thing.
I can think of so many times this has happened before. People getting used to MVC, Dependency Injection, ORMs. Sometimes the code is so large you only work on a part of it, so you have to get used to a lot of not knowing.
So how do we build confidence anyhow when we can't know everything?
I think first, standard checks - does it compile? What do static analysis tools say - code quality, passes SAST analysis? Linters? Style checkers?
Do the tests run? I'm not comfortable with the tests I write let alone the AI.. by adding fault injection (fuzzing, chaos), and mutation testing, I have a better degree of confidence in the tests and code.
Spend even more time getting to know the product instead. Maybe I don't know the code as well as I used to, but I know the product backwards and forwards. That helps me have some new confidence that I didn't really have before. Knowing the greater whole.
Your comment made me think that people have been happily importing hundreds to thousands of dependencies into their projects that were written by some random guys on the internet, but they've just now started to become concerned they don't understand the code they're shipping.
Exactly right. Many never check the release notes of those libs on bump either. To this add copy & paste from Stack Overflow we used to do. People are just afraid of losing their jobs now, so naturally we all got defensive.
I lost my ability to fully explain the systems I work on around the time I transitioned from next-edit-prediction AI running in my IDE (early Copilot) and working primarily via Codex/Claude CLI's.
The further away I got from the implementation details, the fuzzier the semantics became. To the point that now I could not in good faith say I understand the system I have built from the ground-up and worked on the last ~4-5 years.
The solution to this, in my mind, is not to revert back to reviewing implementation details. The economics of that from a business/productivity perspective don't play out well.
The only viable solution in my mind is adapting code/languages themselves to the shift by making programs formally verifiable. That way you can be sure that the code "does what it says on the tin", and provably so.
I'd guess they started developing the project back in 2021 coding by hand, and are now improving/maintaining via agenetic engineering. This is not uncommon.
The point is that it's easy to lose understanding of the finer details, even of a codebase you lovingly created, once this switch occurs and an LLM gets its hands on it.
The bridge is built. How do we test the load? Drive bigger and bigger trucks until it fails, note the weight, print the sign and rebuild the bridge the same way.
as someone using AI, it is mania, leadership has decided it is so and software is getting worse. we could have just accepted our 20% bump and been happy, fixed bugs, made things better quality, nope, ship half assed features at 2x speed, pretend claude is infallible and will solve all our problems, brand reputation? user research? pixel perfect designs? what are you a dinosaur?
No, not in 2022. The models weren't good enough yet. I was looking at the trend line though. I gave word to our engineers that software would not be the bottleneck. The bottleneck was understanding what to do next.
Except in this case the label on the eggs says "Eggs may occasionally be poisonous. Users should not rely on eggs to be safe, and should carefully scrutinise eggs before and during cooking. Bob's Farm accepts no liability for injuries resulting from making omelets with our eggs."
A lot of the real advancements that the dinosaurs didn't understand perhaps seemed like a mania, but a lot of manias seem like real advancements that the dinosaurs don't get.
Maybe a bit unorthodox but I have it plant 1-3 bugs into the code it writes (depending on the size of the MR I'm working on); they range from simple one-off issues to something more structural.
It then gives me a hint on the area or a typical code flow that would hit the bug - whichever gives me less info. I can obviously ask for more hints as I read along.
Obviously all honour-code based since I could just ask it to fix the bugs itself. I think this is something that can't be fixed systemically - either you're going to bother reading and thinking through the code or not.
Other than that I guess reading the tests - to make sure they fail without the code / fix, to make sure the setup makes sense etc.
I agree with you with one caveat. Professionally, your obligation is to be qualified for your job and ship work that you stand behind. Complete comprehension is absolutely necessary.
Personally, with side projects and tooling, go nuts. There's no reason to read all of that code.
The traditional answer is the same for engineers going into engineering management because managing a team and managing a set of agents is highly similar, only managing agents is far easier. One only ever has the illusion of control, and so you have to set your mind to containing the worst possible outcomes and looking at the net efficiency and efficacy of the team with the understanding that they will, on average, never be perfect.
I was struggling with this today. Working on a feature and AI review flagged something so I need to fix that. I iterated multiple times throughout the day and was completely exhausted cause each iteration was always complicated with the fix being ~1k lines. Finally landed on a one I felt at least a little comfortable submitting for review but a work day like this is super draining and I felt like crap the entire evening (I am about to go to bed, I am probably gonna have nightmares about pressing enter and enabling auto mode on plans that make no sense in a true nightmare sense)
1) Need to triple down on infrastructure (safeguards, tests, sandboxes, etc.).
2) Ask agents to do more than just code.
They need to fully demonstrate and prove that their PRs are rock solid. Ask for screenshots, videos, code reviews, detailed architecture diagrams, etc.
It would be an insane amount of work for humans, but AI can do it easily for each PR.
Merging any PR from an agent should be a no-brainer if there's real proof that the feature is built correctly and working.
If there's still doubt, then the agent needs to do a better job of proving that the feature is built correctly, doesn't introduce regressions, etc.
I've been spending a lot of time iterating on feature implementation plans with Claude (at work) and GPT (for personal projects).
I then make sure they write a lot of tests and that if functionality is changed or added/removed then tests are updated alongside.
Generally I try to review changes myself and ask it to explain the changes.
It's not perfect but I feel like the planning step being iterative isn't all too different from my old research -> think about implementation -> implement loop.
At that point, reviewing any code changes goes hand in hand with how I coded before LLMs.
The major difference is I usually have it make any changes for things I don't like.
If the agent has written tests and crucially - witnessed them fail before starting the work to make them pass - you have a lot of confidence.
Review of your own work's easier too - test and production code tend to mirror each other, giving you a much more complete picture. Confused by the production code? Go read its tests for a bit, then come back.
It's also easier to understand the change if you were there for the build, watching the transcript, checking in every so often. Doesn't have to be your primary task - leave it on a second monitor, it'll catch your eye naturally.
And finally, on top of test-driven development, have the agent first (before the first test driven development cycle) build out an end-to-end test covering the work it'll do. Once that test is green (test driven word for "is passing") you're done with your work and can clean up at will.
Play with it before shipping, get it real nice. It's never been easier to clean the tech debt up and make the UI perfect.
Eventually you'll be ready to package it up into a PR!
Now's you can leverage agents and get creative. Make your colleagues froth to review your work. Consider making a Loom video to walk them through it. Consider reworking it into a commit series; it only takes a few second for your agent and you can tell a story for humans. Maybe you want to sequence it according to your struggles; maybe it's better to show the different layers emerge with each commit - it's up to you. Consider also that your agent can easily do one-off visualizations of the change for your colleagues if you think it'll help them, and you can attach those too - mermaid charts or even whimsical AI generated images. All this is because motivation is driven entirely by other humans' motivation and you want them frothing with excitement every time you drop a review. Make it fun, it takes seconds and gets your stuff merged fast while making your teammates smile.
This is a brand new job and we will work out how to do it together :)
In my experience you're just replicating the problem.
Slop code doesn't stop being slop code when you add in slop tests to account for the sloppiness.
If you generate code at a certain velocity, you eventually get to a point where you generate tests the same way you generate code. The model does its best effort in creating tests, but you'll inevitably get to a point in your work where the tests themselves are faulty and unreliable, the same sloppiness you were trying to control for.
There have been multiple times where my software dev team has gone back to prior implemented tests they had the model generate and found that after some time eventually the model got frustrated with the test no longer passing eventually modifying it to fit current needs around a feature push.
You can probably anticipate how we came to discover this, as it ended up introducing a regression around multiple edge cases which the test was meant to control for.
The only answer is to inject actual engineering into the process as much as possible without giving it all up to the models to generate more slop in response to slop.
tests are a great answer, but only if you treat them with serious intentionality.
Your engineers weren't doing what I'm describing I promise.
There are multiple places in my post where I explicitly describe the process by which you intimately:
- make the agent do verified work (the e2e tests and test driven thing)
- watch the agent work (the second monitor thing)
- read your own code (the tests/prod mirror thing
- play with the work and perfect it (the tech debt and UI perfect thing)
Plus a ton of other places in my post I very clearly describe being intimately involved in building and selling the work.
That to me is basic god damn competency. What your engineers did is a fad, and it won't last, and I'm sorry they abdicated their professional duty.
If you wish to screen for that sort of shit in future/current hires, show them my post and see if they turn up their noses or if they get excited and creative. You want only the latter.
It’s very disconcerting. I think this will largely be solved by going all in on AI and use it to test and review the code in an adversarial manner. That won’t be cheap but it’s the only way I see this work. But a lot of companies will probably save money and use AI mainly for writing code and then skimp on review and testing. I bet we will see some huge blow ups with AI written code and hopefully the industry will learn.
It also took the industry quite a while to accept that server code needs to be secure even if that requires a lot of extra effort. AI will probably be the same.
This kinda makes me think about the transition from writing machine code by hand to using a compiler. Did people feel similar at the time that they didn't understand the machine code that the compiler was writing for them?
The problem assembly programmers had with compilers wasn't that they wouldn't understand the generated machine code - it was that the generated code was often bad (as in unoptimized and bloated) and didn't justify the resources required to run the compiler. As hardware got more complex and powerful, compiler got much better, and operating systems became the primary layer between hardware and applications, assembly became more niche. Nowadays it's pretty challenging to beat the compiler at generating efficient assembly and writing a program in assembly would consist primarily of OS and library API calls anyway.
There are definitely parallels there. The only obvious difference I can see is that both high level language code and assembly are unequivocal in their meaning (as in you can determine what gets executed), whereas natural language is not.
> ... they didn't understand the machine code that the compiler was writing ...
This is a bold assumption and inconsistent with my knowledge of history. High level languages allowed better management of complexity and reduction of tedium. The common complaint was that compilers produced less efficient (slower and/or larger) code, not that the output was more complex. It was[1] normal to read the compiler's output, which was generally assembly or another intermediate representation. After all, a compiler is merely a tool for translation from one language to another. Additionally, early compilers often were not optimizing compilers; the code they generated was simpler than handwritten programs.
The key difference is that the procedure for transforming source code -> machine code is fully explainable by anyone with access to the compiler source code. There are no statistical estimations with black-box parameters involved.
Each step in abstraction we've made had people who felt uncomfortable at the level of magic they were accepting. At one point in the 70s/80s, traditionalists were arguing that C itself was too high level. Then in the 90s and early 00s, that languages like Python and garbage collection were absurd: a "good" programmer doesn't need them and they give up too much control. Then as web development ramped up, people entered the "Javascript is slop" phase into "React is slop" phase.
Anytime the barrier to entry is reduced and some skill or profession becomes more focused on output than the ego of the professionals, you can bet a counter-culture of people will rise up to oppose it. It's a threat to their sense of "betterness" and/or the (perceived) value of their career.
Maybe the new meta should be reviewing new code at the end of the week/month rather than at each merge? But even that might be too much of a bottleneck. The alternative is to talk to your agent to understand changes at a high level without reading every line of code.
I like starting with defining outcomes, determining invariants, and then consider TLA+ or Lean models, or both. It's generally not good to ship stuff you don't understand, but if you work on big legacy code bases from 20-30 years ago, there's probably no one who understands all the code all the time.
It's at least nice to be able to prove something about the code, rather than blasting it with tests which only prove that the tests that are written do not fail (and perhaps get you some amount of code coverage — necessary but insufficient for anything like quality).
How to do all of that in your environment? That can vary greatly!
I've been playing with this https://getchronosphere.com. It will (attempt to) rebuild a PR in a coherent journey rather than a massive blob. It's been somewhat helpful up to this point, and still improving.
Im currently building a ui framework for unreal. I dont enjoy ui work so I wanted a way to have to write less ui.
For me I dont look too closely at the implementation but I look at the api where if a human was using my framework how is the code they'd have to write and engage look at.
Then I also do plenty of performance against a comparable target in my case slate (default unreal) and keep overhead minimal.
So good performance and reasonable public facing api are the 2 main pieces im focused on. Less with the internals.
Care about the seams and interaction and the things you can measure or count.
I think the solution is leaning on static types and hard to abuse abstractions to make programs correct by construction. It does mean that you’ll still have to use your brain to design the internal code architecture, and just use LLMs to fill in the boring bits between the safe abstractions.
I'm currently working on a combined Aero sim/CAD/CAM software for designing, simulating, and performing DFM optimization specifically for Group 1 to 3 winged UAVs.
What I do is ask Opus/Gemini to explain the parts I don't understand. And I do some of my own research to verify accuracy of the explanations.
In addition:
* Every new feature has a Feature Design document which I review, modify, and approve. And a corresponding Implementation Plan which I also review and approve.
* Every fix has a RCA document and fix Implementation Doc which I review, modify, and approve.
* There's a global high level Architecture and Design document with additional sub component design documents which are always kept in synch after a new feature or bug fix is done. Synching it is always a step in the Implementation plan.
This does adds some overhead but it allows me to have have check point gates. This has often allowed me to catch misunderstandings and align properly before any code gets edited by the AI agent.
Having the above docs also helps when it comes to referencing them as context to properly ground new features or bug fixes.
>This new standard of shipping with AI is moving so fast that if we're going to spend our time checking each line of code, it's going to take forever.
"This new way of piling bricks is so fast that I can no longer afford to check the buildings integrity".
I just don't understand how the fuck we ended up with this narrative. Somehow the machine making things twice as fast forbids us from spending time ensuring quality? Surely if it does things faster you have more time and tools to check!
I'm half convinced this frenzy is the marketing work of AI companies at this point.
If you are a senior engineer then there is no need for you to be shipping code that you don't understand. People can definitely generate code a lot faster than before, but you also have a magic genie that can help you critique and improve the code that you're reviewing, including when that code was written by agents.
An experienced senior engineer with access to a coding agent is the most powerful code-understanding system that has ever existed. There's no excuse for shipping or approving anything you don't fully grok.
Do a quick text search in this comment section for the string "it works". You'll see the same results in basically every HN discussion about LLM coding. "It works."
To me, "It works" is pretty much the lowest possible quality bar for an acceptable creative work product. We should not aspire to "it works." We should not hold ourselves only to the "it works" standard. And, yes, I know that many traditional software companies fail to even meet "it works" for their shipping products.
I don't even know what to say anymore. This has become a deeply unserious industry putting out mountains of junk that barely even meets the "it works" bar, and LLMs are just accelerating this.
I mean, I did that when I was learning how to code. Is this seriously something professional would do? Not the copying part, but the not understanding part.
I love using AI, but any feature it creates I test the hell out of it, i become the QA testers I hated and do all sort of ways to break the app, and after 100 or so hours, I have a bug free and fully working app, now will this app scale as I add more features that to be seen.
That is the problem. Because it's not a clearly abstracted and well designed system, it's not robust for adding features and changing things around. When you try to add something or move something around the big fragile ball of tests breaks and you have to spend forever re-piecing things together or burning heaps of tokens for little changes.
When you code, do you understand the asm code being built? What are the cache line access patterns? Most software developers have already given up the details of how things work, and lean on tools to handle them.
Which is fine, usually. Depends on what you are doing.
The things that always mattered were the core system, the trunk of a tree if you will. The branches less so, and the leaves not at all (though that is what people see and appreciate! The irony!).
Product doesn’t care. And most of us will get replaced by agents. I guess around 30% of the engineering workforce will keep their jobs which will be mainly to keep the product up and running, doing whatever it takes (so using agents to fix stuff basically).
The ones who get to keep their jobs will be rebranded as Maintenance engineers with a lower salary. VPs and Heads of will keep their cozy job (bc there are not so many of them), and they are not that dumb to fire themselves.
Peter Naur in 1985 wrote that programming is the act of building a working theory of the system in our minds, and this often takes form as tacit knowledge in the head of programmers. Even if you handed the complete codebase and perfect documentation to a completely new team, they would often need to go back to the old team and ask questions.
And before agents, we often built this working theory of the system by writing the code by hand. It helps us figure out the "grain" of the codebase so we have a better idea of how to extend it.
So what happens when agents are writing the code? The main avenue with which we built the working theory of the system in our mind is now taken away, and we're left feeling like our understanding of the system is blinded.
I ended up more or less changing my entire workflow. Four main principles:
1. Interrogation/interview-based design and specification
2. Make the decisions that matter and leave the rest to the agent.
3. Use agent to generate explainers, quizzes, and simulations.
4. Use verifiers and guardrails in the coding loop.
Interview-based design and spec
When I tell the agent what I want to build or extend, there's naturally ambiguity. But the agent knows what is ambiguous to it, so it can ask me. /grill-me is one way to do this. You'd think this would be commonplace now, but it's actually not. Some people are still wondering why they're getting bad results. It's because they're not resolving the ambiguity.
Making the decisions that matter
Not every decision matters in a product. Agents are fast at writing code and pretty good at some of the nitty gritty details, so I let it make those decisions. Also, if a particular subsystem is commonplace and undifferentiating, such as login, then I also let it make architectural decisions for that subsystem.
For decisions that are product differentiators, where that differentiation is supported by technology, I make decisions on the stack, the architecture, and the approach. This is the design phase. Because agents tend to make the mainstream decisions, they almost always take a midwit approach which doesn't leverage the technology in a way that makes a difference for the customer.
Some try to build this understanding from code reviews. I personally don't think it's the way to go, because it's not motivating. I find that I'm more motivated to learn how the system works today, because I want to change how it works tomorrow--and it sticks in my head better too (no different than pre-agents).
Generating explainers, quizzes, and simulations
As I'm doing design for an improvement and I don't know how a part of the system works, I can ask questions of the agent in chat. If I still don't get it, I can ask for diagrams (either ASCII or mermaid). But if it's an entire subsystem, I have a skill in which I can ask for visual and interactive explainers. It'll generate a markdown page (or html) that has these interactive components that help demonstrate a concept.
To help make the concept stick, there are quizzes after each section and at the end.
With the computational simulations, I generate a throwaway tool in which I can explore the state space. This doesn't happen as often, as I try to keep state spaces small, and for those explainers tend to work well.
With the design simulations, a same-model agent and a cross-model agent and I will talk about an explainer, and we'll name things that we understand or notice about the design and the intended direction we want to extend it. Then, we'll use subagents to generate options and then discuss the pros and cons. I find this is a good way for me to understand the dead-ends of a design.
Using Verifiers and Guardrails
I define guardrails as mechanistic checks that doesn't change with the feature. such as linters, typecheckers, and pre-commit hooks are all guardrails. Verifiers are mechanistic checks that will be different with each feature, such as unit/integration tests.
For more complicated states or concurrent parts, I'll ask the agent to use formal methods to write a proof. Then use the proof to generate a trace (a series of steps that demonstrate a happy or unhappy path). Then I run that trace in unit tests (kinda like property-based testing--so I guess this can be called trace-based testing), and the code has to adhere to the trace.
I also wrote a design verifier, where the requirements in the spec can be converted into a series of mechanically checkable rules. Then during testing, the verifier fires up the headless browser and uses the DOM measurements of the page to see if it matches up to the rules. This was completely impractical before agents.
All of this so a) I can be sure that the agent is doing what it's supposed to with the decisions that it's making b) I can be out of the coding loop, so that I'm not up at 2am begging, "you fixed this, but you broke that. pls fix." The agent can check for itself.
---
If any of this sounds helpful, lemme know. I've been wondering if other engineers care about understanding the code that's being written, or if people just don't care anymore. I've spoken to a few, and it seems to be split. And it may just be a matter of degree and detail. Some like it at a higher level. Some like it nitty gritty.
I don't understand these types of comments. Aren't the CVEs epic today with human written code? What difference does it make then? You want no CVEs, make software engineering into a real gate-kept engineering professions with accountability chains, government regulations, formal education, etc.
> You want no CVEs, make software engineering into a real gate-kept engineering professions with accountability chains, government regulations, formal education, etc.
This will not only not prevent CVEs, it will also make everything a giant pain in the ass
Of course it will. But at the same time I don't see building falling left and right like CVEs in software. So don't tell me agents write bad code, because humans do as well. The only way to prevent this (or minimize), is to turn software engineering into real engineering.
> Aren't the CVEs epic today with human written code? What difference does it make then?
LLMs cause errors at a noticeably higher rate than human devs do. If you lean on them without actually checking what they wrote, the error rate in your software is going to go up.
"If it works, ship it" is very different from "If it works, and I understand it because I wrote it, ship it".
Does the LLM understand it, in detail? Can you trust its interpretation of the information and instructions you provided? Did it independently make an adverse decision you haven't noticed, because you didn't read the code? Was the model missing any information it required -- and would you know if it was?
LLMs depend on humans and human oversight to ensure they have the information required to make the right decisions, to catch when they don't, and to steer them based on your much more complete understanding of the broader context and requirements. You can't rely on them to do that thinking for you (yet?).
I’d argue that most people also don’t understand code they read from their colleagues. Or if they do, then they don’t understand the entire context of the app.
You said LLM need human oversight and it’s true, but it does not imply you have to read the code.
I think we are going towards a place where code no longer makes sense, you just describe what you want in English, and the LLM just builds directly in machine code. You then build different verification tests to make sure the program is doing what it’s suppose to do.
the guidance / guardrails, agent roles, etc are the critical thing now, it's kind of like meta programming AI to get the outcomes you want. Any problems need addressing at this level. You don't need to understand all the code, but you should understand how the system is building up and adjust as needed. This would be an architectural review process you go through with AI. Features to code that should plug into existing architectural structures are generally no problem. It's just boilerplate code AI is generating. But architecture is only one aspect, you should have lots of guidance for all kinds of thing. There's security, performance, correctness, QA, Ux, Deployment, Process, Monitoring etc and having feedback systems for everything. You keep refining and reviewing your guidance and the impact it is having. As well as this you are molding the product for the things you want via feature development. What I'm not sure of is if you are some cog somewhere in a company like Meta whether whatever slice you become responsible for becomes a bit meh if you are mostly just prompting a feature and not much more. As a developer in a smaller company (we are global with a smallish tech team) I get to do a lot of things from interacting with our customers through to the entire backend and get to make critical decisions about nearly everything. Nearly everything in the company is starting to be driven via guidance and feedback loops (including on the guidance itself).
The only answer is to not ship code you don't understand. And I think the easiest way to do that is not use AI for ANYTHING except troubleshooting and doc searching. Those that draw a land in the sand and develop this way will be slower in the short term but their work will last out in the long term because it will be more solid.
Short-sightedness is quite literally the reason people are using these systems to begin with. If you care about the health and integrity of your code base years down the road you wouldn't be shipping endless amounts of slop you don't understand, Fred Brooks taught us this half a century ago.
But it seems short-sighted to think that "years down the road" will look _anything_ like the present. I can't imagine a future (say, 5 years from now) where we are worried about the code choices we made in the past.
Flat-out, you _need_ to understand the code you're merging. If you don't, ownership then lands on whatever AI agent you're using. Ultimately, you can't fire the AI agent in the same way you can fire the person at the driver seat when shit hits the fan and they've consistently caused incidents that force people to work late hours to resolve it.
I agree with the sentiment but I would add more nuance; you have to pick what is important to understand and what you can delegate; not all code is mission-critical.
In any case at the end you're the owner/accountable so you need to justify your and your agent choices.
It's simple: you don't. You slow down. It's essential to understand what you ship, before you ship, as there are real world consequences for not doing so, ranging from wasting the time and energy of yourself and others, up to failures which cause a great deal of harm to people and the world. To knowingly ship code you don't understand is grossly negligent, regardless of how common it becomes.
I don't ship code I don't understand. I fully review every line if it's going to prod.
My dumb little utils in ~/.local/bin? Those get merged after a quick functional test without my having read a single line of it. Still, the skill I maintain for such a job directs agents to write such throwaway scripts in anything but shellscript because I might want to extend some day and fuck shellscript
> How to deal with shipping AI-written code you don't understand?
Stop shipping code you don't understand.
Is there really any other solution? You can either ignore it and pretend like it's the new era of software like half the other commenters are, or you can sit down and understand the code and what you're shipping.
I wouldn't be surprised if the meta person has a gun to their head to increasingly keep shipping like a maniac (how decade-long bugs remain unfixed don't ask me), so this advice may not be very useful.
I have some work in Lean4 that I'm certainly unqualified to validate; Layer Coding QEC derivation, Positive Grassmanians and the Fresnel-Kerchhoff correction to the Parallel-Axis theorem, Huygens intensity as an alternate model for simulating photonic and electronic interference in a double slit experiment, for example.
It was a black box but I'm letting it tutor me about it so
Visual proofs, Tools for logic and formal methods in distributed systems
Things that compile logic!
> That means you can end up with systems you've built where you don't truly understand how all the code works
Though AI coding and system development can be overwhelming due to information overload and rapid growth in complexity (and concern for unwittingly embedded mistakes, the unsecurability of agent-based development, and number of LOC of code per day), not understanding the whole system sufficiently is normal and expected in engineering.
Generally speaking I brain storm the feature first and produce a dev plan. This includes asking it to reuse code/libraries we’ve already got! After it’s gone off ah done its thing we go back over and do a QA on what it’s built, why that approach, what can we do better etc etc?
It feels like another layer of "conceptual compression". I'm not sure who/when that phrase came around but in context of Rails is where I heard it and its things like ActiveRecord or really any ORM. You aren't writing the SQL directly, something else is and for the most part it works. Then it doesn't work or your database grows too much and you need to optimize table structures with indexes, etc.
My point is if you don't understand the fundamentals of the underlying code (SQL in the example above), you're more likely to have issues in the future. Or maybe not if that app doesn't reach a usage pattern or complexity where the nuances under the hood matter.
It really depends on what you're shipping. Meta ships social media apps where in the grand scheme of things if something is broken software wise it largely doesn't matter. What's the worst case? lost ad revenue? Users can't do things? If that were software for MRI machines or aerospace where software issues/bugs can result in lives lost then it matters much more in understanding what is going on at all levels of the software and potentially hardware.
You're going to have to understand it one way or another, either at writing or at debugging. Even if you delegate all debugging to AI, you'll still need to determine what is "right".
If you don't understand the code at writing or debugging, you'll pay the architectural price (where it degrades). That price may be worth paying, but that depends on the system you are building.
Anyone who trusts the output of LLMs uncritically, even after the impressive level models and harnesses have gotten to, is borrowing against their future.
Models and harnesses have gotten really good, but they still are hard-capped by the operators that use them.
The sycophancy of models is perhaps their greatest weakness, as when a user who might be experienced in one area attempts to use AI to build within an area they are less familiar with, the model does not take a step back and examine the "why" of what is being built from an architectural standpoint, it simply performs the expected task in a direct manner, ultimately creating sloppy work that may be easy to ship but difficult to maintain.
A good example of this was a recent interaction I had with a Dev who was very familiar with how to develop embedded and system level applications. He and every other dev are being asked for increasingly more due to AI, and he had to bring something he had built as a single-user desktop application into a distributed cloud environment.
He ended up rsyncing a copy of his working directories including multiple credentials and sensitive internal project documents to a virtual machine binding it to a public ip address. Thankfully our tooling detected this before he could get any farther but it illustrates how the AI model isn't doing any real thinking or engineering at the end of the day.
Unless the operator knew to instruct the model about proper cloud architecture and understood best practices around deployment of public systems, the model simply did what it was being told to do by a user who didn't know any better.
Another example of this is the sheer amount of unsophisticated users we've been detecting using Javascript injection into their web browser sessions as a form of automation for web services they may use. These users would never know to ask how to request an API token which may allow them to accomplish things in a much easier and approved way, and instead are relying on the frailest possible automations that expose them to risks of credential exposure.
We've got to be careful with how we engineer things with AI models. Just because we've made boilerplate easier to generate doesn't mean that we can avoid the actual engineering of what we do. And I can't help but think that firms which are fully committed to AI code generation, won't just be introducing an avalanche of slop into their product which will explode the costs of maintenance over time.
If it's important code (and much isn't), then I spend a lot of time using Matt Pocock's /grilling first, and then I get the agent to kick off peer-review with other agents, and surface any disagreements. Some models are better at it than others. This usually forces me to broadly understand the architecture, and I spend most of my time answering the design questions for which there isn't an obvious solution. I have been pretty happy with the code I've launched to production, as measured by the ways it breaks, and by how agents who weren't part of the original committee describe it.
This is the state that founders and decision makers are usually in at a startup. Sure, sometimes you have the coder founder who wrote the first iteration of the product, but many times the founders have no clue how the product works under the hood. They entrust that to the engineers. That was then. Now, much of that engineering layer has been removed, so the founder just needs a smaller team of people who understand how to structure the software and interact with the LLM. LLMs do the rest.
The problem here is that you're trying to do a job that is already being done for you. Maybe you don't yet trust it to do a good job, but that is quickly changing.
no one can force you, take responsibility for what you ship, what you choose to ship, how you choose to work, you signed that contract you go to work every day, your company is composed by people like you
don’t wash your hands, or at least don’t expect us to wash them for you
Replace "AI agents" with "a bunch of human coders" and you'll see the answer. Your product needs decent testing and it's your job as the responsible person in charge to make sure that's in place and working properly. So if anything you probably want to be reviewing the test code. But often that isn't necessary since you can do things like introduce bugs deliberately and see if tests fail; check test logs for information that confirms the tests were run as expected; try some manual testing once in a while; ask for new more advanced tests (e.g. test performance not just functionality); and so on.
The part that makes me uneasy is when the AI writes both the code and the tests. They might agree with each other and still miss the same thing. How are people here checking for that without doing the whole job twice?
The author of the code was physically touching the internals and potentially intuiting obvious shortcomings as they worked on it. That very crucial part is now gone.
I like fault injection (fuzzing, chaos), and mutation testing. I think introducing these helps the AI write better tests and helps having some confidence in the tests that are written.
You should be using two (or even three) different LLMs. For example: Claude to do the coding, and Codex to do an adversarial review. This can go back and forth for several iterations before converging on an acceptably "correct" version of the code...
Occasionally, Claude may not be able to fix the issue – and it may turn out that Codex can, because it has a different approach.
Either way, at least two LLMs is the only way to get (mostly) reliable code.
I disagree here. Software engineering is a very different form of engineering than something like building a bridge. This has always been true. Even before LLMs, there was a lot of hobbled together code going out there from "real engineers."
I think it's silly to make strong statements like this. _Most_ professional software engineers (women too, not just him!) are using LLMs and are moving some amount of faster as a result.
The people doing this work as a career and not performative art, who were already good at their job, had no worries about this to begin with.
The code was always a black box. Always.
Specs come in, passing tests come out. If you find an issue, your specs were shit.
Anyone that has ever had to outsource work to someone else knows this. If you've ever worked with another team whose codebase you didn't touch - you know this.
You write your specs, you launch an agent to implement them. You launch a 2nd agent to keep the first one honest. When done, you launch as many agents as you have budget to try and poke holes in the implementation. As it always was the case.
You can thrash around, you can fight it. Like it or not, the genie is out of the box, and the realization you were never really good at your job to begin with is gonna set in eventually.
I’m surprised I haven’t see anyone say this yet, but I don’t let the LLM design the code at all, I let it implement a design I have created.
By that I mean that, rather than let the LLM do the entire structure, I take the time to create the full hierarchy of abstractions and actors that I want my code to do. I decide, at the UML level, how abstractions will sit and interact — and then I make the LLM implement those gaps.
The key thing here is that, by doing this, I DO know how the code works. I may not know how each unit is implemented, but I know each unit’s responsibilities and how they interact with one another. I’ve found that LLMs are HORRIBLE at writing decent abstractions, regardless of how much prompting and skills I’ve given it. It will always get the job done, but it will often be unmaintainable and require bigger and heavier refactors as I introduce more and more requirement changes.
But by flexing my basic design skills, I don’t have those issues. Future refactoring fit into easy abstractions that I’ve designed. And by documenting it up front, I also have what becomes a living document on how this work will turn out.
It takes a few hours up front to plan ahead, but it makes the LLMs job easier and use less tokens, and it makes the code better long term. I don’t care if one unit is suboptimal, as long as the abstraction is right — because at worst I can change an implementation for a better one later.
I agree with this approach, but I'm a bit surprised this has to be said at all. These are basic software design principles and I imagine it's the default way any experienced programmer would approach the problem. I wonder if the people not using this approach are primarily junior developers who have never seen how real software is built, or if it's leaking into the senior levels as well.
Most people in the industry right now, even at FAANG and even among people carrying "senior" titles at these places, have little experience with "software design principles" -- tens of thousands of brilliant Ivy League graduates with surreal compensation packages have spent the last decade pulling tickets off JIRA and pasting least-change-possible patches onto existing code for about 10 hours a week while spending the other 30+ hours barely engaged at meetings.
This hurt to read but checks out in my experience. These types also tend to love tossing debt over silo'd walls and pointing the finger a lot if/when anything goes wrong.
I'm old enough now to be a protected class, so I don't want people accusing me of ageism, because that's not where this is coming from - but there's a type of person that clearly is on their last job of their career and knows it and I try to avoid them like the plague, I've seen it too many times.
I work on my code this way, but I have no idea what my colleagues are doing, and it's impossible to review thousands of new/changed lines a week.
I just approve PRs, because I have to. I don't know what else to do. I can barely keep an eye on all my code.
Actively looking, but job market is rough in Northern Europe. I quit caring about quality in any capacity here, the company has clearly chosen its direction, against my very vocal complaints.
My last interview, with what I thought was a great company, failed when I didn't understand what NLP was when they asked me about it.
> NLP? > Natural Language Programming, it's the future. Claude, GPT, etc.
I laughed, he didn't, looked offended and said it was a serious question. I had thought it was a joke, said "oh you mean LLMs". I tried to answer seriously, of course, but that was my third round technical interview and I didn't get a call back.
Just getting a paycheck and producing mediocre work until I get something new. (Senior engineer, 10 YOE, patents, 3 years at a unicorn).
Also considering becoming an electrician.
I will be unhappy if they steal the meaning of NLP. NLP means Natural Language Processing and LLMs are only a subset of NLP. I would have concerns about anyone who claims to have technical knowledge and asserts NLP means "natural language programming." Those are the same kind of people who might suggest Lisp should add mutable objects.
You may be surprised and annoyed that NLP is also "Neuro linguistic programming", I've met technical people talking of NLP and meaning that...
Sounds like a cushy job. Why would you want to leave?
Some people just can't turn off their brains and play drone for dozens of hours a week for decades. It feels existentially, aesthetically, and sometimes ethically crushing to spend a whole life doing work you can't expect yourself to respect or value, no matter how much you get paid or how little the work overtly demands of you.
Often, the lack of meaningfully challenging demands being put on you is punishing on its own. It can feel like a relief for a week or month or year, but if you catch a glimpse of that being your whole future, you can come to recognize yourself as being in a mind-eroding secular purgatory.
This anguish is especially common among on highly capable people, high achievers, and those used to some kind of ambition. And by accident of its history, there's a lot of those people in this industry and some of them are going nuts now.
> and some of them are going nuts now.
Pretty much this. I know it sounds melodramatic but it literally feels like rotting.
I don't solve anything anymore. I don't get any spark of fixing bugs. I don't DO ANYTHING, and it's fucking chipping away at me.
Also I'm fairly sure people are going to realise soon that if they're happy with slop, they don't actually need highly trained engineers anymore.
This whole thread actually prompted me to look into electrician work, especially in renewables that we have here in Norway.
I might fix my burnout and go to school to get my 2 year qualification. AI can't rewire a hydro dam.
> This anguish is especially common among on highly capable people, high achievers, and those used to some kind of ambition. And by accident of its history, there's a lot of those people in this industry and some of them are going nuts now.
I went nuts for years, until I sold off my interest in my dream partnership.
10 years of heaven. All the benefits and none of the downsides of my internal colleagues. Then 14 years of bureaucracy aimed at predictability over progress, which was infinitely more difficult for me to dodge as an external player.
Compensation for my interest was very objectively valued, with the rate of stalled progress priced in. Which made complete sense, while also being its own madness.
I walked away from so much value, but I don't regret it for a minute.
Why not become a doctor in that case?
Yeah, I imagine it's something that needs to be enforced at the organizational level. I sympathize with anyone who takes pride in their work and needs to deal with mindless coworkers pushing slop without any accountability. I left corporate software development (before LLMs were a thing) over frustration with the general culture and I can only imagine it's gotten worse since then.
It's really bad now, yeah. :( It just really hurts that I spent well over a decade of my life learning and honing skills to engineer safe, quality software and now it's just, useless.
No one cares at best, or I'm callously told "keep up or fall behind" at worst. When that's not even the problem, it just, sucks now. There's no joy in anything I do professionally.
Can I ask where you went? I'm seriously looking for an exit, I just don't know what yet. Maybe a trade, in Scandinavia the pay is actually comparable, I just don't want to go back to school.
I was fortunate and saved up enough to live frugally and just work on passion/learning projects for a while. I've been considering going back into commercial software development as a solo dev and leveraging AI, but keeping the focus on quality and craftsmanship. In my opinion there's never been a better time to start a new, ambitious project if you're passionate about quality software.
The LLMs know design principles too, being an experienced developer doesn't make you as special as you might think.
Don't get me wrong, LLMs are way better than me at most things, but I don't think you can deny that the current generation of models makes questionable decisions sometimes. Especially on project structure, API boundaries, and code duplication. I also think they just have bad taste in general, though of course that's more subjective.
I’m not sure how to say this without sounding smug, but the last year has shown that most of those things just… don’t matter. It’s just the next level of the “variable naming PR comment battle”, so will become obsolete very soon.
Obviously there are mission critical systems where it matters, but if “your code can be A/B tested” or “slowly rolled out”, chances are, it doesn’t matter.
You'll find that design matters when token spend starts facing real accounting.
Tech debt is still tech debt, even if there exist models that can wade through, because it's going to require increasingly sophisticated models navigating increasingly large context windows and increasingly speculative workspace operations to keep making progress on it.
Vendors like Anthropic and OpenAI are laying a pit trap when they demonstrate how their models can independently code their whole platform as it becomes increasingly byzantine and inscrutable because not only do they benefit from internal/prerelease models but they have effectively unlimited spend because all that spending acts as a lure to draw you into their trap.
For everybody else, design still matters. People are just working really hard to make you forget.
> I don't think you can deny that the current generation of models makes questionable decisions sometimes
Sometimes, not enough to believe you that you can do it better. And even if you could, to believe that it's overall worth it to work in that way.
You can work the way you want to work. In my experience, letting the agent make every decision leads to a big ball of mud that requires touching dozens of files to make a change. More context is needed, it burns through tokens much faster, and the agent takes shortcuts and builds technical debt. So the way I prefer to work is to have changes targeted to a single file or small set of files where I've defined the API. Maybe that will change in the future, I'm not set in stone here.
What still is true for all fields of LLMs: if you can’t tell they do it suboptimally, you are just not that knowledgable in that area.
It still applies to all things LLms can do from law to code to art to ideas.
They can do it - and very acceptably - but only to the untrained eye.
The LLMs have been trained on all kinds of code. They often choose a hacky solution because that's what they encountered in similar situations in their training. It generally only takes a few words to guide them to a better design than what they saw in the wild, but if you don't guide them, they follow the herd.
> doesn't make you as special as you might think.
Way to come off as incredibly insecure. There's no need to randomly put people down.
I'm not the one who's claiming to be above the latest models in some aspect of coding in this interaction.
No one at all claimed that, certainly not the person you responded to. They're talking about how to use these tools in a way that gives you insight into what they produce. So, as an engineer, you know the shape of the output, before it's made.
That's it. Maybe you should go re-read?
I've been doing this a long time and i would say since the last 3 months or so i've spent more time learning from llm code than correcting llm code. It's not bad and takes approaches and makes decisions i wouldn't have thought of. I'm not too ashamed to admit that, a model just as access wider and deeper experience with code than I do.
Talking about LLMs "knowing" things is questionable at best.
> rather than let the LLM do the entire structure, I take the time to create the full hierarchy of abstractions and actors that I want my code to do. I decide, at the UML level, how abstractions will sit and interact — and then I make the LLM implement those gaps.
Do you do this with every repository that's brand-new to you? You refuse to, for example, use an agent to learn where the feature should generally live? This seems absurdly slow for that purpose, because even though humans can still get a deeper understanding of a codebase after several months, an agent can get a much deeper understanding over a timeline of the initial few days. Unless everything is already meticulously documented for you, I suppose.
Agreed. Even with opus 5.5 which everyone praises for its wonderful coding abilities, I keep catching awful coding abstractions or missing abstractions.
It can keep patching everything left and right whenever a new requirement comes up so it does not "need" to build abstractions, and it will only do it to please the human reviewer.
> I have created.
Are you working on a team? I've found that for solo projects this works but, even when you own the design of your piece, integrating with the work of several people doing this ultimately leads to even the part you initially designed no longer working exactly as you remember it.
A good chunk of my time now is spend working with an LLM to understand how other parts of the code base work and how they relate to and interact with the product features I manage. But I'm still on a small enough team that this is possible. I'm not surprised at all that an engineer at an org the size of Meta would not even be able to perform this level of work in understanding what's happening.
LLMs certainly helps with designing and is the best brainstorming tool one can have beside their own brains.
Same here. I review plans w rendered diagrams. Data structures, flowcharts, uml sequences, and component architecture docs.
I have found that making the AI look the plantuml and the rendering and reviewing before presenting it to me results in much more legible designs. To me the peak of this would be a shared interactive whiteboard w the ai during design.
This is where myself and a colleague have settled. We design for longevity but it can still type, search, and debug faster than we can.
Same. I let the LLM work inside a framework that I made.
Of course I only arrived at the framework after trial and error with an LLM showed me the shape of the problem and the framework presented itself to me.
This is too the paradigm I’ve settled upon after a year of using these tools.
Agreed, otherwise the risk is receiving the average of what LLMs think are best practice.
Planning is fundamentally so critical to ensuring alignment and outcomes, that over time how one plans can be better trained into the process to help do more and more of it.
Testing.
Lots and lots of testing. I usually interactively test with the LLM. Now that I'm spending less time, coding, I get to spend more time, testing.
In my experience, I test more, with an LLM, than I did, with my own code. I don't find coding errors, but I do find implementation errors.
On another note: I ran an LLM on a bunch of really mature code that I wrote, and could swear was "perfect," and it found a number of issues.
Nobody's perfect. Trust; but verify.
On this note - people love to create 'AI factories'. Has anyone made a "QA factory" where a set of agents mimic real users to run in a pre-pod environment? Like I'm talking you spin up a variety of VMs with real desktops and agents that use real browsers and act like real users (on timing schedules that mimic real users too)
So, in other words, we save time and money when it came to creating stuff, but we've made a whole bunch of new work around verification and testing of said stuff?
What have we gained aside from shifting where time and effort is spent in the design loop?
I'm largely building tools for myself and my agents. This allows me to battle test internally before releasing code to the wild. It adds some time but largely it's part of my regular work unless something breaks badly. (I also run multiple agent teams simultaneously and do staged rollouts to reduce blast radius of the latter)
This method also helps with the classic problem of being unable to identify essential complexity up front.
we've made a whole bunch of new work around verification and testing of said stuff?
The verification is what we always wanted but never had time for. For 30 years I wrote side project games without extensive testing or a fuzzer or anything because I wanted to spend my time writing the game. Now I can do that and still have the testing I always knew I should be building.
Shipping features and fixing bugs much, much faster.
Better tested software duh
Engineering will always be difficult.
Right now, we're in a "honeymoon" stage, where non-engineers think that they can somehow, magically produce stuff that is comparable to what experienced and disciplined engineers can do.
That won't last long.
Despite all the caterwaulin', LLMs are just another engineering tool, and good engineers will figure out how to implement them, in ways that absolutely blow away the garbage produced by non-engineers.
It's like that rich, no-talent guy that buys a PRS guitar, and thinks he'll beat the experienced musician that still uses his beat-up old strat, but consistently makes hits.
Isn't this the perfect classical capitalist advertisement? Every number goes up.
- code? up - build budget? Up - developer output? Up - developer oprrational costs? Waaay up!
Why is this Bad? Every one involved profits somehow.
I recently moved my project guardrails into a separate set of skills in a repo that I can bring into a new project fairly easily. It's focused (wrong word, it's seriously not focused) on implemented as many different levels of testing as I can think of. Right now it does this https://github.com/onion2k/guardrails/blob/main/GUARDRAILS.m....
The goal is to get to a point where I can say with a good level of confidence that a change didn't break anything. It might be the wrong change, and it might be crap code, but I can ship it without anything blowing up. I've been using it for a while across a few game projects and so far it's worked quite well.
I find the problem with "lots and lots of testing" as a solution - (assuming by that you mean lots and lots of automated testing) is it invites lots and lots of lines of code - making code reviews even more challenging.
If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
Might be a little controversial to say this but automated tests can be reviewed at a lower bar than production code.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them.
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
IMO it works better if you split the tests into the ones you care about, and the ones the LLM writes. I assume its tests help it write code with fewer errors, so I'm happy for it to keep doing that, but they're akin to CoT - an artifact of the process that I don't need to inspect in detail.
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
Testing doesn't verify the design.
> Trust, but verify.
Testing is NOT verification.
EDIT: Sorry, oops, it's also called a kind of verification.
I'm pretty sure it is? It's not validation, but what are tests if not automated verification of certain things?
Sorry I was thinking like formal verification like proving something is correct. I guess testing can be considered a form of "dynamic verification."
But there is the problem that tests only check or verify a small sample of possible cases or inputs. They can never show that the program is correct. (Famous Dijkstra statement.) To know something is really correct you have to understand and prove it.
How do you test code that you don't understand?
Black-box e2e testing isn't suitable, or practical, for a lot of applications. Unit and integration tests generally require you to understand the code you want tests for, to know what to actually assert
Are you just trusting the AI to do it all?
Nothing a "make no mistakes" at the end of a query cannot fix. /s
The point is that you're not verifying if you don't even review the code.
Even if your test suite tested every single possible behavior and code path, tests usually can't intercept all possible unintended behavior that an LLM might have added.
In all likelihood, most test suites don't intercept any unintented behavior (such as unintended internet connections).
That's not a big problem when a honest human has written or reviewed the code, because unintended code is usually easy to spot.
But an LLM might have dropped any whatsoever code: maybe some of your project's code resembles the one of a trojan in its training, so it completes it with the trojan.
Or maybe it found that if your server is unreachable, it can upload the client's data to a public Github repo; or that if there's no space left on the device, it can delete some of the user's folders, easy!
Would your clients be understanding if that happened?
So you currently routinely review the machine code and link tables produced by your compiler?
Eventually, the tools become trustworthy, and we no longer need to worry about their products.
We're just about there, with frontier LLMs, but I don't think we're all the way there, yet.
"I saw her duck" - how much would you be willing to bet on the intended meaning of this sentence?
English isn't a good programming language if you want to trust that the output will be as you want it to be. High level code has no room for ambiguity - that's why we use it. It's a concise, very explicit language to tell the compiler exactly what to do.
Dealing with an LLM is a fundamentally different thing. The reason it's able to speed us up is because it makes assumptions for us and reads between the lines. The only world where we trust the output is one where it can literally read our minds. That's not coming any time soon
How can a software developer compare a deterministic program with LLMs?? Please tell me that you aren't, or that it's not what really think!!!
I’m slowly coming around to the idea that if it works, I’m good with that. I never used to personally check all my colleagues work, either, I just trusted that they’d done a good job.
That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.
I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.
I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.
So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.
I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.
Yes, compilers are a good candidate for verification through thorough testing. If a comprehensive test suite passes, reviewing every line of code isn’t so important.
I am not in META but for what I read on blind, assuming it is true, they are not spending any time in long planning sessions or in getting into details of the implementation plan at all, and they could not anymore considering how many PRs in parallel they are requested to manage
Yeah, I'm retired. The only person I have to answer to regarding progress is me. I'm a stubborn son of a bitch when it comes to choosing quality over speed...
That doesn't mean I don't like progress mind :) This morning I got the preliminary results of a 'C' front-end to the compiler (it uses clang to compile to LLVM IR, then translates to our IR, then feeds that IR into the autovectoriser as if it had parsed the code itself). So for the exact same source code I now have:
Those are some pretty fine speedups, from naive C, with no alterations from the standard "C" code - it automatically finds places that can be made data-parallel, tracks the data-hazards to make sure its safe to "GPU" and writes all the kernels, boilerplate and data-stream-management if necessary. Obviously the checksums have to match, so I know it did the work correctly too :)Hacker News isn't interested - any post I make linking to the webpages is immediately blocked/dead so I'll wait until the C++ front-end is ready (some time next week) and announce it somewhere else. A compiler that takes standard C/C++ code and speeds it up by automatically finding data-parallel areas and scheduling them on the GPU with no programming overhead is pretty rare.
Oh, and it manages the local network as well, so you can run xcd on machines on your lan, or via the TB4/5 bridge between Macs and it'll schedule on other GPUs, just like it manages multiple GPUs on the local machine. It characterises both latency and compute-time to figure out which devices are worth scheduling on.
After asking my LLM some questions to barely grasp what you are working on, I can now say: very cool!
For anyone wondering, the thing lives at https://compile-xc.org/ ;)
Awesome, this is genuinely-cool HN material. Thanks for sharing.
Upvotes don't appear publicly on HN so I'll comment to agree with this. I'm sorry if earlier information about this compiler got lost in the whirlwind. Data parallelism is a hard problem and these results look genuinely impressive so I for one would definitely be interested in hearing more about it and maybe discussing it further when there are interesting developments.
Thanks :)
Still working on some of the data-parallel recognition in LLVM IR space. In order to support both C and C++ with the same data-parallel section extraction, I moved the detection from the ‘xc source code’ phase to the IR lowering phase, meaning that everything could ‘feed in’ to the same routines. That’s caused a little friction with the slightly different ways the LLVM IR from clang and xc’s IR encode loops. The above results are all single-loop extractions and the double-loop ones are still ongoing.
The system works in 3 stages: 1) by being greedy about which loops could possibly vectorise and marking them, 2) by analysing the IR to reject loops which do pointer-aliasing, or object manipulation, or… and then 3) by measuring the fastest way these known-safe kernels could execute.
Which brings me onto one nice thing I haven’t mentioned: that the GPU choice(s) are made at runtime, not compile time, and the system automatically measures (and then persists so it doesn’t have to measure again) the fastest way to run a kernel on the current machine. It handles MSL on Apple devices, PTX on Nvidia cards, Spir-v on Windows, Linux, and Android, and WGSL in the browser. It’ll quite happily divide up the parallel task across a 3090 and a 5080 on my Windows test box for example, with the 5080 doing ~2/3 of the work, because the scheduler is opportunistic.
All without writing anything other than plain C with nary a mention of the GPU. Give me a week or two, and I’ll probably announce it on phoronix (which might then get a mention on HN if it goes down well there, and it’ll be a different url so hopefully won’t be insta-dead).
So, the real input to your autovectorizer is LLVM IR; you could soon enough support any language that has an LLVM front-end? (I say soon enough instead of immediately because I’m sure there’s some manual wiring up to do). That seems like a pretty big deal. Do Fortran next, pls. :)
Ssshhh. I like to try things before promising them :)
Hell yea I love that computers are getting another chance at becoming compute fabric, just using each others GPUs. Sick.
> Hacker News isn't interested
I wouldn't be too sure about that :)
I’m being a little cynical - I’ve submitted it twice (though the ‘C’ front end is new) and it bombed both times - the second time within seconds of submission.
I think it’s my own fault, I’ve posted the link to the compiler pages several times in what I thought were relevant posts, and I think HN thinks I’m spamming. So I’m not posting the home page for a while and we’ll see if a future submission “takes”
might be worth firing an email to hn@ycombinator.com. they should be able to tell you why your submission was auto-killed, and might work with you to ensure your next submission isnt.
I actually did email the second time because it was clearly an automated response, but I didn’t hear anything back. Not too surprising, Dan is a very busy guy, and he doesn’t know me from Adam :)
That is pretty adventurous, but it sounds really cool! Will it be open source?
Yep. I’d give the link but I think HN thinks I’m spamming because I’ve mentioned it too many times before in posts - hence immediate death of any submission, so I’m not mentioning it for a while to see if that tails off.
Bottom line: GPL3 for the compiler, standard library exclusions for derived works, so use it for anything but feed me back improvements to the compiler itself, and you don’t get to redistribute it as your own.
I like it! The Rust crew would love this I bet.
See the first paragraph here: https://alexgaynor.net/2026/sep/07/is-it-correct/
If your definition of "works" includes "doesn't have vulnerabilities", you're going to need to do more than just poke at it and add tests.
> I’m slowly coming around to the idea that if it works, I’m good with that.
A.k.a. "The concrete didn't crumble when I hit it with a hammer."
Process quality has been a thing for ages because it is essential.
I mean, honestly ? Yeah.
Except it’s about 1000 “hammers” (aka tests), x 9 different architectures, x 2 different (a reference and a production) variants, and x 3 different optimisation levels. Every time I find something that slipped through, it gets at least one test written to cover it in the future. My main time-sink is the hours it takes to run the ‘gate’ before a release across the test-farm. And that’s after I’ve been playing with it myself for a while, pushing corner-cases through whatever I’ve been working on.
Very similar here. For internal projects the test is - if it works, unit and integration tests were created, and it follows our architecture correctly.
> never used to personally check all my colleagues work, either,
Jesus F Christ! No wonder most software lately (~decade) has been shit. Have y'all not heard of code review? Or did you think it just for show...you sit on a change for a week and then press "Accept" without reading it?
This is how every product manager feels every day.
If you're an LLM prompter shipping products, you're outputs are more akin to a product managers, not an software engineer's.
Agreed. The implication is, get really good at defining acceptance criterion.
Kind of, except as product manager you're held accountable for delivering the right product, while your engineering team is held accountable for things like performance, incidents, and security issues caused by their code.
It's hard to hold an LLM accountable for anything.
Accurate but manageable. Just like a pm works with properties an engineer also does. The Product person wants to deliver the right product properties so they model their process around identifying the ones they want and then creating the right framework to get them. On the engineering side its the same thing. Define what you want, give agents an environment they can succeed in, iterate.
What nonsense! If a product manager ships something that breaks then the software engineer who submitted the code gets the blame!
Software engineers are always responsible for their code except now they have to be responsible for Claude's code
Software engineers are always responsible for their code except now they have to be responsible for Claude's code
Which is - of course - absurd. Accepting responsibility without demanding the corresponding authority is a fool's game.
I have never been much of an advocate for unionisation in the software development industry. I didn't think it was necessary to protect the interests of the workers because for most of the industry's history the workers have largely been able to stand up for themselves anyway. Likewise I've never been a big believer in accreditation and licensing practitioners in our field - because who is qualified to judge someone else worthy in such a young industry with so little hard data about what works or doesn't work? I find my views on these subjects are evolving now.
It's not about protecting jobs if they truly aren't needed any more or protecting people working in the industry who refuse to move with the times. The Luddite philosophy isn't going to get us anywhere here. And besides - if AI is as effective as its advocates have been claiming then there should soon be more demand for capable developers to take advantage of the new opportunities. (The Jevons Paradox is relevant here.) But it is about protecting standards. If the tools aren't doing a good enough job then expecting professional developers to ship the code produced by those tools even though they know it's junk is something that needs to be challenged. And if developers are being expected to ship anything important without having the slightest idea whether it even works then that's even worse.
We had a lot of silliness going on when the very low interest rates and huge amounts of VC funding were propping up ideas and practices that would never have survived otherwise. Today we have a different kind of funny money distorting the industry but it's still propping up ideas and practices that we wouldn't normally support or perhaps even tolerate. And just like before no individual is going to be able to hold back the tide and it's going to have to be a collective effort to push back - hence my evolving views on topics like unionisation and accreditation. We've all seen the drop in quality across much of the industry and we all know why it's really happening because we see how the sausage is now being made.
Not all mistakes are code, a lot are feature design mistakes, and if SWEs would always be responsible about PM mistakes, then why not use Claude anyway?
Oh, don't worry. Someone is also blaming the product manager!
It's not too dissimilar to life as a Principal level engineer pre-AI. You technically oversee the work of many engineers on their own systems, but you couldn't possibly review all the code yourself. You appreciate that human code review reduces, but does not eliminate, the number and severity of bugs shipped to production. You don't have the low-level details in your head, just the high-level ones. You design how all these engineers interact with each other, without stepping on any toes.
Time will tell whether we were right to move from a system where we trusted humans to understand the low-level system, to one where we trust fancy calculators. But so far, it's working.
But so far, it's working.
There seems to be disturbingly little objective evidence of that. There's certainly a lot of code being written but how much actual value that code is adding is another question. With the scale of resources that have been pumped into AI-supported software development over the past few years where are all the new applications and the quantum leaps in quality and performance in the applications we already had? If anything I seem to be seeing more anecdotes going the opposite way so far.
Anthropic's recent blog post about how they replaced Git should be a massive wakeup call. A trillion-ish dollar company (exact number unimportant here), the vast majority of its value coming from its intellectual property, replacing the storage layer that is an essential part of its system of record for that intellectual property, written pretty much entirely with its in-house Kool-Aid.
Disturbingly little evidence? I would hardly call that "disturbingly little"...
(Apologies if I'm misreading your reply. I'm assuming it was sarcasm but Poe's Law might apply here.)
This does feel a lot like the "Agile era" that was the last silver bullet that was supposed to fix software development forever and the "Cloud era" that was supposed to fix infrastructure and deployments.
Agile had a few good ideas. Code reviews and automated tests are commonly used today.
It had a few plausible ideas that it took too far. We are better at recognising the value of incremental development over big up-front design today - despite the ironic clash this seems to be having with agentic workflows - but almost everyone hates stand-ups and lots of projects have gone off the rails by breaking things down far too much and never committing to anything and so never getting anywhere.
And it had plenty of ideas that almost all of us have thrown out as unpleasant and/or unworkable. Pair programming comes to mind immediately.
What didn't really change was the fundamental nature of software development. Did we adopt a few useful ideas and get a more more efficient and improve quality a bit? Certainly. Did we get orders of magnitude better at anything and give up all the tried and tested ways of working in favour of radical new ones that never quite felt right? No.
On our current trajectory I'm expecting a broadly similar outcome for LLMs. They're now very good at some things. But they are still pretty bad at other things that also matter - and some of the reasons they are bad at them are an inherent part of how they work and therefore probably can't be overcome (at least not with incremental improvements on the technology that still basically work the same way). So I am expecting to see a period of consolidation with a lot of organisations pulling back from their dramatic leaps into the unknown and downsizing of dev teams once the big AI companies start squeezing financially and the problems of developer burnout and stagnating skills and low morale that we're already seeing after just a few months become endemic. And then as I suggested in another comment I think it's also increasingly possible that we'll see developers unionise and/or push for professional regulation and licensing so they can resist the pressure to just ship whatever their agent wrote without even bothering to check it but still being responsible when inevitably things then go wrong.
The assumption was that some engineers knew how each part of the system worked, or at least the system was comprehensible because it was created by humans for other humans. There were comments explaining difficult places, etc. With AI this assumption is broken. Now nobody understands how things work, and it cannot be understood in principle because the system is a mountain of slop created by some alien intelligence.
> Now nobody understands how things work
Well, the fancy calculators do. Yes, they still need guidance to keep from going off the rails. But, at least from personal anecdotal experience, I need to go spelunking into the codebase less and less over time. The agents and models really did just start to become that good.
Ask the AI how it works!
It's totally fair to load up some open source project into an agentic IDE and instead of asking it to write something ask questions like "how does X happen when I use feature Y?" or "can you explain how client code could accomplish Z"
All the time I see an AI use a language feature or library feature that I never used before or I see some code I don't understand and it does much better than a lot of humans do at explaining what it did.
You certainly can do this for code, and it’ll give you great answers. But I find that, for whatever reason, info that I consume this way has a much shorter TTL in my brain than something I wrote myself.
I’ve been experimenting with writing a living spec — API endpoints, schemas, how things work at a low enough level that the LLM can fill in the details - and that’s been working well.
It also forces me to do the hammock-driven development necessary to build the right thing, and as a byproduct it keeps the design in my head longer.
Technically it's just a role change. You're now the project manager or something, not the developer. It isn't a project manager's job to understand the code, only to ensure it does what it should and doesn't do what it shouldn't. And to ensure the developers, etc str unblocked and doing well. Embrace the new role.
Why would you need to embrace something forced upon you?
There is no escape from Dario's basement. It's best just to accept our fate. Rub the lotion on our skin or else we get the hose again.
We just need to hold until people realize the crappy codebases that AI is generating
Perhaps it will make it easier to cope with? Fighting something beyond your control is pretty exhausting
Or just ignore AI. As long as you do your job, who cares if you use AI or not
So you can pay rent?
In most workplaces, when supervisor/boss says "we need you working over here now; you start tomorrow", you either embrace or quit. Again, nothing fundamentally different.
And in most jobs, you don't get promoted to manager if you don't want to
Maybe not, but things and times change. If you're made obsolete in one role then you better be ready to move on to another.
Then whose role is it now to understand the code?
How much understanding is needed? If it works as specified (you did a complete spec and occasional check-ins to ensure things are on track and then tested the delivery, right?), then what more is needed?
Understanding is needed when something breaks and you need to fix it immediately. Otherwise you become a meat proxy and now you’ve added Anthropic’s uptime as a single point of failure to your application.
Subscribe to multiple providers, and maybe even self-host :).
i refuse to be a meat proxy!
So you just click around the UI, assuming there is one?
Can you break down for me exactly how the unit and integration tests are made? Do you ask the AI what to assert?
> exactly how the unit and integration tests are made
There should be enough detail in the spec that it's pretty obvious. And unlike - many - human devs, agents don't shy away from writing tests. Add a coverage guardrail and tell them to use red/green TDD and you're already a good way there. The main thing is to ensure shared understanding, so they also know business expectations and write appropriate tests. And... nothing here is really specific to an agent, as a human dev also needs sufficient guidance (tailored to their experience level).
This is just pure vibe coding and trying to convince yourself it's quality work. AI can't be trusted to go from an English written spec to tests
Here[0] we have benchmarks for AI writing tests for code that already has the correct behavior. Best case scenario is that they're at 67% with Fable, and that's with the highest quality specification we could possibly have (code). English is far more ambiguous
[0]: https://labs.scale.com/leaderboard/sweatlas-tw
Maybe you don't trust them to, but they can go from spec to tests; I've made them (even if it initially took extra prompting and scrutiny). And ultimately tests are just a proxy to checking that X works, written by devs for devs; actually trying X is something to be done by QA and/or project/product manager before shipping.
> (you did a complete spec and occasional check-ins to ensure things are on track and then tested the delivery, right?)
I have never in my career been afforded the time to make a complete spec.
That's now a key task with this new role. Unless winging it into slop is acceptable.
Ah yes the Santa dev fallacy.
Noone understood the code to begin with. People understand the concepts, the primitives, the "theory" as Naur said, the basis upon which a project is built. Noone ever understood a codebase worth understanding to begin with.
Project managers just take the engineers word for it.
And engineers have a lot of leeway to just make shit up and the PM would have no idea until it blows up.
Except you can't blame Claude when sh-- hits the fan.
You hit deploy and hope for the best, if it goes bad You just prompt the model again....
Just kidding, it's de same as with any other Open source plataform You install from their repo. It will be hard at the start but then You Will be very aware where it starts to fail and how to fix it.
I built and use https://smalldocs.org, which is a rich document format (mermaid diagrams, nice formatting, etc.) for agents. So I’ll be like “sdoc me the architectural ideas for this feature” and it helps.
The time that I would have spent typing in the code is now time I use to test the code. I really like to poke at things and ensure that my understanding and reality match.
I agree with the advice to design your system before implementing it. It is pretty easy to iterate on a design with your favorite coding assistant, and this is your chance to be ultra-picky and make sure you get exactly what you want. This was always where the "software engineering" was done, but in practice it's easy to have a vague idea and refine your (internal) list of requirements / tradeoffs as you type in the code. I find it harder to do that when the LLM is generating the code, just because one turn can change so much.
Keeping it under control requires discipline, just like before LLMs were a thing. You set a bunch of engineers loose on your codebase with no vision or supervision, you get spaghetti. You just get a little bit more control now. Agents don't feel shitty for the rest of the day when you tell them to do it some other way, so you are free to review as critically as you want without hurting anyone's feelings. I think it's a good thing.
I can think of so many times this has happened before. People getting used to MVC, Dependency Injection, ORMs. Sometimes the code is so large you only work on a part of it, so you have to get used to a lot of not knowing.
So how do we build confidence anyhow when we can't know everything?
I think first, standard checks - does it compile? What do static analysis tools say - code quality, passes SAST analysis? Linters? Style checkers?
Do the tests run? I'm not comfortable with the tests I write let alone the AI.. by adding fault injection (fuzzing, chaos), and mutation testing, I have a better degree of confidence in the tests and code.
Spend even more time getting to know the product instead. Maybe I don't know the code as well as I used to, but I know the product backwards and forwards. That helps me have some new confidence that I didn't really have before. Knowing the greater whole.
Your comment made me think that people have been happily importing hundreds to thousands of dependencies into their projects that were written by some random guys on the internet, but they've just now started to become concerned they don't understand the code they're shipping.
Exactly right. Many never check the release notes of those libs on bump either. To this add copy & paste from Stack Overflow we used to do. People are just afraid of losing their jobs now, so naturally we all got defensive.
> So how do we build confidence anyhow when we can't know everything?
Avoid known unreliable tools?
I lost my ability to fully explain the systems I work on around the time I transitioned from next-edit-prediction AI running in my IDE (early Copilot) and working primarily via Codex/Claude CLI's.
The further away I got from the implementation details, the fuzzier the semantics became. To the point that now I could not in good faith say I understand the system I have built from the ground-up and worked on the last ~4-5 years.
The solution to this, in my mind, is not to revert back to reviewing implementation details. The economics of that from a business/productivity perspective don't play out well.
The only viable solution in my mind is adapting code/languages themselves to the shift by making programs formally verifiable. That way you can be sure that the code "does what it says on the tin", and provably so.
I feel so strongly about this I've written on it:
https://gavinray97.github.io/blog/design-by-contract-and-eff...
> To the point that now I could not in good faith say I understand the system I have built from the ground-up and worked on the last ~4-5 years.
You weren't programming with LLMs in 2021 or 2022, be real. More like 2024 (that's already pushing it) or 2025.
Because telling lies on the internet attached to my real name is so beneficial to my reputation, right?
https://i.imgur.com/CNAUczi.png
Receipts!
You poor soul. You were gambling against roulette odds.
I'd guess they started developing the project back in 2021 coding by hand, and are now improving/maintaining via agenetic engineering. This is not uncommon.
The point is that it's easy to lose understanding of the finer details, even of a codebase you lovingly created, once this switch occurs and an LLM gets its hands on it.
You get it.
The bridge is built. How do we test the load? Drive bigger and bigger trucks until it fails, note the weight, print the sign and rebuild the bridge the same way.
I'm miles away from all of this as a non-user of AI and it seems like a mania to me.
as someone using AI, it is mania, leadership has decided it is so and software is getting worse. we could have just accepted our 20% bump and been happy, fixed bugs, made things better quality, nope, ship half assed features at 2x speed, pretend claude is infallible and will solve all our problems, brand reputation? user research? pixel perfect designs? what are you a dinosaur?
You should probably try it.
Yeah, man. Don't be a square!
Try it, you might like it. First month's free and you can still quit if it's not for you.
You make eat omelets using sourced material you have no way to verify.
You trust the label. The person. The company. You hope someone did their due diligence.
Sometimes taste the shell. That is an acceptable risk.
We used to be in the business of owning, understanding and evolving our processes. Are we not, now?
Would you have made this comparison in 2022 about a future where you're shipping code that you're not even reading?
(made a small edit)
No, not in 2022. The models weren't good enough yet. I was looking at the trend line though. I gave word to our engineers that software would not be the bottleneck. The bottleneck was understanding what to do next.
Except in this case the label on the eggs says "Eggs may occasionally be poisonous. Users should not rely on eggs to be safe, and should carefully scrutinise eggs before and during cooking. Bob's Farm accepts no liability for injuries resulting from making omelets with our eggs."
Users are responsible for what their agents do.
I expect the guy sticking to his horse and cart, as the first cars started to overtake him on the roads, felt similar.
A lot of the real advancements that the dinosaurs didn't understand perhaps seemed like a mania, but a lot of manias seem like real advancements that the dinosaurs don't get.
Maybe a related question:
How do you keep the IEC 62443-4-1 auditors happy?
Maybe a bit unorthodox but I have it plant 1-3 bugs into the code it writes (depending on the size of the MR I'm working on); they range from simple one-off issues to something more structural.
It then gives me a hint on the area or a typical code flow that would hit the bug - whichever gives me less info. I can obviously ask for more hints as I read along.
Obviously all honour-code based since I could just ask it to fix the bugs itself. I think this is something that can't be fixed systemically - either you're going to bother reading and thinking through the code or not.
Other than that I guess reading the tests - to make sure they fail without the code / fix, to make sure the setup makes sense etc.
Well, it's simple. Don't do it.
I agree with you with one caveat. Professionally, your obligation is to be qualified for your job and ship work that you stand behind. Complete comprehension is absolutely necessary.
Personally, with side projects and tooling, go nuts. There's no reason to read all of that code.
> Complete comprehension is absolutely necessary.
I feel like an increasing amount of orgs out there have long abandoned this thinking.
Your world will get smaller and smaller . I don’t think you can stop that wave.
The traditional answer is the same for engineers going into engineering management because managing a team and managing a set of agents is highly similar, only managing agents is far easier. One only ever has the illusion of control, and so you have to set your mind to containing the worst possible outcomes and looking at the net efficiency and efficacy of the team with the understanding that they will, on average, never be perfect.
I was struggling with this today. Working on a feature and AI review flagged something so I need to fix that. I iterated multiple times throughout the day and was completely exhausted cause each iteration was always complicated with the fix being ~1k lines. Finally landed on a one I felt at least a little comfortable submitting for review but a work day like this is super draining and I felt like crap the entire evening (I am about to go to bed, I am probably gonna have nightmares about pressing enter and enabling auto mode on plans that make no sense in a true nightmare sense)
1) Need to triple down on infrastructure (safeguards, tests, sandboxes, etc.).
2) Ask agents to do more than just code.
They need to fully demonstrate and prove that their PRs are rock solid. Ask for screenshots, videos, code reviews, detailed architecture diagrams, etc.
It would be an insane amount of work for humans, but AI can do it easily for each PR.
Merging any PR from an agent should be a no-brainer if there's real proof that the feature is built correctly and working.
If there's still doubt, then the agent needs to do a better job of proving that the feature is built correctly, doesn't introduce regressions, etc.
This is OT and a cheap shot, but if I were an engineer at Meta I would have felt pretty uncomfortable well before AI showed up...
I've been spending a lot of time iterating on feature implementation plans with Claude (at work) and GPT (for personal projects).
I then make sure they write a lot of tests and that if functionality is changed or added/removed then tests are updated alongside.
Generally I try to review changes myself and ask it to explain the changes.
It's not perfect but I feel like the planning step being iterative isn't all too different from my old research -> think about implementation -> implement loop.
At that point, reviewing any code changes goes hand in hand with how I coded before LLMs.
The major difference is I usually have it make any changes for things I don't like.
Test driven development and the testing pyramid.
If the agent has written tests and crucially - witnessed them fail before starting the work to make them pass - you have a lot of confidence.
Review of your own work's easier too - test and production code tend to mirror each other, giving you a much more complete picture. Confused by the production code? Go read its tests for a bit, then come back.
It's also easier to understand the change if you were there for the build, watching the transcript, checking in every so often. Doesn't have to be your primary task - leave it on a second monitor, it'll catch your eye naturally.
And finally, on top of test-driven development, have the agent first (before the first test driven development cycle) build out an end-to-end test covering the work it'll do. Once that test is green (test driven word for "is passing") you're done with your work and can clean up at will.
Play with it before shipping, get it real nice. It's never been easier to clean the tech debt up and make the UI perfect.
Eventually you'll be ready to package it up into a PR!
Now's you can leverage agents and get creative. Make your colleagues froth to review your work. Consider making a Loom video to walk them through it. Consider reworking it into a commit series; it only takes a few second for your agent and you can tell a story for humans. Maybe you want to sequence it according to your struggles; maybe it's better to show the different layers emerge with each commit - it's up to you. Consider also that your agent can easily do one-off visualizations of the change for your colleagues if you think it'll help them, and you can attach those too - mermaid charts or even whimsical AI generated images. All this is because motivation is driven entirely by other humans' motivation and you want them frothing with excitement every time you drop a review. Make it fun, it takes seconds and gets your stuff merged fast while making your teammates smile.
This is a brand new job and we will work out how to do it together :)
In my experience you're just replicating the problem. Slop code doesn't stop being slop code when you add in slop tests to account for the sloppiness. If you generate code at a certain velocity, you eventually get to a point where you generate tests the same way you generate code. The model does its best effort in creating tests, but you'll inevitably get to a point in your work where the tests themselves are faulty and unreliable, the same sloppiness you were trying to control for.
There have been multiple times where my software dev team has gone back to prior implemented tests they had the model generate and found that after some time eventually the model got frustrated with the test no longer passing eventually modifying it to fit current needs around a feature push. You can probably anticipate how we came to discover this, as it ended up introducing a regression around multiple edge cases which the test was meant to control for.
The only answer is to inject actual engineering into the process as much as possible without giving it all up to the models to generate more slop in response to slop. tests are a great answer, but only if you treat them with serious intentionality.
Your engineers weren't doing what I'm describing I promise.
There are multiple places in my post where I explicitly describe the process by which you intimately:
- make the agent do verified work (the e2e tests and test driven thing)
- watch the agent work (the second monitor thing)
- read your own code (the tests/prod mirror thing
- play with the work and perfect it (the tech debt and UI perfect thing)
Plus a ton of other places in my post I very clearly describe being intimately involved in building and selling the work.
That to me is basic god damn competency. What your engineers did is a fad, and it won't last, and I'm sorry they abdicated their professional duty.
If you wish to screen for that sort of shit in future/current hires, show them my post and see if they turn up their noses or if they get excited and creative. You want only the latter.
My Jira tickets are written by AI started just feeding the ticket into AI (shrugs)
No I only do that for unit tests so far
I went through like a mid-life crisis when the thing I learned/spent last 13 years on became meaningless.
It’s very disconcerting. I think this will largely be solved by going all in on AI and use it to test and review the code in an adversarial manner. That won’t be cheap but it’s the only way I see this work. But a lot of companies will probably save money and use AI mainly for writing code and then skimp on review and testing. I bet we will see some huge blow ups with AI written code and hopefully the industry will learn.
It also took the industry quite a while to accept that server code needs to be secure even if that requires a lot of extra effort. AI will probably be the same.
This kinda makes me think about the transition from writing machine code by hand to using a compiler. Did people feel similar at the time that they didn't understand the machine code that the compiler was writing for them?
Very good question!
The problem assembly programmers had with compilers wasn't that they wouldn't understand the generated machine code - it was that the generated code was often bad (as in unoptimized and bloated) and didn't justify the resources required to run the compiler. As hardware got more complex and powerful, compiler got much better, and operating systems became the primary layer between hardware and applications, assembly became more niche. Nowadays it's pretty challenging to beat the compiler at generating efficient assembly and writing a program in assembly would consist primarily of OS and library API calls anyway.
There are definitely parallels there. The only obvious difference I can see is that both high level language code and assembly are unequivocal in their meaning (as in you can determine what gets executed), whereas natural language is not.
> ... they didn't understand the machine code that the compiler was writing ...
This is a bold assumption and inconsistent with my knowledge of history. High level languages allowed better management of complexity and reduction of tedium. The common complaint was that compilers produced less efficient (slower and/or larger) code, not that the output was more complex. It was[1] normal to read the compiler's output, which was generally assembly or another intermediate representation. After all, a compiler is merely a tool for translation from one language to another. Additionally, early compilers often were not optimizing compilers; the code they generated was simpler than handwritten programs.
1: And still is, for some of us!
The key difference is that the procedure for transforming source code -> machine code is fully explainable by anyone with access to the compiler source code. There are no statistical estimations with black-box parameters involved.
Yes!
Each step in abstraction we've made had people who felt uncomfortable at the level of magic they were accepting. At one point in the 70s/80s, traditionalists were arguing that C itself was too high level. Then in the 90s and early 00s, that languages like Python and garbage collection were absurd: a "good" programmer doesn't need them and they give up too much control. Then as web development ramped up, people entered the "Javascript is slop" phase into "React is slop" phase.
Anytime the barrier to entry is reduced and some skill or profession becomes more focused on output than the ego of the professionals, you can bet a counter-culture of people will rise up to oppose it. It's a threat to their sense of "betterness" and/or the (perceived) value of their career.
[delayed]
Maybe the new meta should be reviewing new code at the end of the week/month rather than at each merge? But even that might be too much of a bottleneck. The alternative is to talk to your agent to understand changes at a high level without reading every line of code.
I like starting with defining outcomes, determining invariants, and then consider TLA+ or Lean models, or both. It's generally not good to ship stuff you don't understand, but if you work on big legacy code bases from 20-30 years ago, there's probably no one who understands all the code all the time.
It's at least nice to be able to prove something about the code, rather than blasting it with tests which only prove that the tests that are written do not fail (and perhaps get you some amount of code coverage — necessary but insufficient for anything like quality).
How to do all of that in your environment? That can vary greatly!
I've been playing with this https://getchronosphere.com. It will (attempt to) rebuild a PR in a coherent journey rather than a massive blob. It's been somewhat helpful up to this point, and still improving.
Im currently building a ui framework for unreal. I dont enjoy ui work so I wanted a way to have to write less ui.
For me I dont look too closely at the implementation but I look at the api where if a human was using my framework how is the code they'd have to write and engage look at. Then I also do plenty of performance against a comparable target in my case slate (default unreal) and keep overhead minimal.
So good performance and reasonable public facing api are the 2 main pieces im focused on. Less with the internals.
Care about the seams and interaction and the things you can measure or count.
I think the solution is leaning on static types and hard to abuse abstractions to make programs correct by construction. It does mean that you’ll still have to use your brain to design the internal code architecture, and just use LLMs to fill in the boring bits between the safe abstractions.
I'm currently working on a combined Aero sim/CAD/CAM software for designing, simulating, and performing DFM optimization specifically for Group 1 to 3 winged UAVs.
What I do is ask Opus/Gemini to explain the parts I don't understand. And I do some of my own research to verify accuracy of the explanations.
In addition:
* Every new feature has a Feature Design document which I review, modify, and approve. And a corresponding Implementation Plan which I also review and approve.
* Every fix has a RCA document and fix Implementation Doc which I review, modify, and approve.
* There's a global high level Architecture and Design document with additional sub component design documents which are always kept in synch after a new feature or bug fix is done. Synching it is always a step in the Implementation plan.
This does adds some overhead but it allows me to have have check point gates. This has often allowed me to catch misunderstandings and align properly before any code gets edited by the AI agent.
Having the above docs also helps when it comes to referencing them as context to properly ground new features or bug fixes.
Hey this is very cool, reach out to me https://alexanderaghili.com/ because Id love to learn more.
Force to always use TTD, at least validate the tests.
Recommend reading (All though a little self serving): https://www.linkedin.com/pulse/ai-code-quality-jorge-varas-a...
>This new standard of shipping with AI is moving so fast that if we're going to spend our time checking each line of code, it's going to take forever.
"This new way of piling bricks is so fast that I can no longer afford to check the buildings integrity".
I just don't understand how the fuck we ended up with this narrative. Somehow the machine making things twice as fast forbids us from spending time ensuring quality? Surely if it does things faster you have more time and tools to check!
I'm half convinced this frenzy is the marketing work of AI companies at this point.
> this frenzy is the marketing work of AI companies
Worse. "AI" is a self-marketing contagious delusion, driven by an inexhaustible power supply. Human gullibility.
If you are a senior engineer then there is no need for you to be shipping code that you don't understand. People can definitely generate code a lot faster than before, but you also have a magic genie that can help you critique and improve the code that you're reviewing, including when that code was written by agents.
An experienced senior engineer with access to a coding agent is the most powerful code-understanding system that has ever existed. There's no excuse for shipping or approving anything you don't fully grok.
I shipped code I didn't understand when I copy pasted from Stackoverflow before. At a level if it works it works.
Do a quick text search in this comment section for the string "it works". You'll see the same results in basically every HN discussion about LLM coding. "It works."
To me, "It works" is pretty much the lowest possible quality bar for an acceptable creative work product. We should not aspire to "it works." We should not hold ourselves only to the "it works" standard. And, yes, I know that many traditional software companies fail to even meet "it works" for their shipping products.
I don't even know what to say anymore. This has become a deeply unserious industry putting out mountains of junk that barely even meets the "it works" bar, and LLMs are just accelerating this.
I mean, I did that when I was learning how to code. Is this seriously something professional would do? Not the copying part, but the not understanding part.
Formal verification, pick a language that has it integrated and prove relevant post conditions.
AI still does not know how to even write code that makes impossible state unpresentable so testing won’t help.
I love using AI, but any feature it creates I test the hell out of it, i become the QA testers I hated and do all sort of ways to break the app, and after 100 or so hours, I have a bug free and fully working app, now will this app scale as I add more features that to be seen.
That is the problem. Because it's not a clearly abstracted and well designed system, it's not robust for adding features and changing things around. When you try to add something or move something around the big fragile ball of tests breaks and you have to spend forever re-piecing things together or burning heaps of tokens for little changes.
When you code, do you understand the asm code being built? What are the cache line access patterns? Most software developers have already given up the details of how things work, and lean on tools to handle them.
Which is fine, usually. Depends on what you are doing.
The things that always mattered were the core system, the trunk of a tree if you will. The branches less so, and the leaves not at all (though that is what people see and appreciate! The irony!).
if you have to maintain it, you will still have to get an understanding of it when it breaks.
not understanding the detail is quite similar to a coworker writing the code and another coworker approving and merging the changes.
Product doesn’t care. And most of us will get replaced by agents. I guess around 30% of the engineering workforce will keep their jobs which will be mainly to keep the product up and running, doing whatever it takes (so using agents to fix stuff basically).
The ones who get to keep their jobs will be rebranded as Maintenance engineers with a lower salary. VPs and Heads of will keep their cozy job (bc there are not so many of them), and they are not that dumb to fire themselves.
Code understanding doesn’t matter anymore (or won’t matter soon)
I had same problem, but after a while I just accepted that is inevitable
"This code is really ugly."
That sucks. Ship it.
Peter Naur in 1985 wrote that programming is the act of building a working theory of the system in our minds, and this often takes form as tacit knowledge in the head of programmers. Even if you handed the complete codebase and perfect documentation to a completely new team, they would often need to go back to the old team and ask questions.
And before agents, we often built this working theory of the system by writing the code by hand. It helps us figure out the "grain" of the codebase so we have a better idea of how to extend it.
So what happens when agents are writing the code? The main avenue with which we built the working theory of the system in our mind is now taken away, and we're left feeling like our understanding of the system is blinded.
I ended up more or less changing my entire workflow. Four main principles:
1. Interrogation/interview-based design and specification
2. Make the decisions that matter and leave the rest to the agent.
3. Use agent to generate explainers, quizzes, and simulations.
4. Use verifiers and guardrails in the coding loop.
Interview-based design and spec
When I tell the agent what I want to build or extend, there's naturally ambiguity. But the agent knows what is ambiguous to it, so it can ask me. /grill-me is one way to do this. You'd think this would be commonplace now, but it's actually not. Some people are still wondering why they're getting bad results. It's because they're not resolving the ambiguity.
Making the decisions that matter
Not every decision matters in a product. Agents are fast at writing code and pretty good at some of the nitty gritty details, so I let it make those decisions. Also, if a particular subsystem is commonplace and undifferentiating, such as login, then I also let it make architectural decisions for that subsystem.
For decisions that are product differentiators, where that differentiation is supported by technology, I make decisions on the stack, the architecture, and the approach. This is the design phase. Because agents tend to make the mainstream decisions, they almost always take a midwit approach which doesn't leverage the technology in a way that makes a difference for the customer.
Some try to build this understanding from code reviews. I personally don't think it's the way to go, because it's not motivating. I find that I'm more motivated to learn how the system works today, because I want to change how it works tomorrow--and it sticks in my head better too (no different than pre-agents).
Generating explainers, quizzes, and simulations
As I'm doing design for an improvement and I don't know how a part of the system works, I can ask questions of the agent in chat. If I still don't get it, I can ask for diagrams (either ASCII or mermaid). But if it's an entire subsystem, I have a skill in which I can ask for visual and interactive explainers. It'll generate a markdown page (or html) that has these interactive components that help demonstrate a concept.
To help make the concept stick, there are quizzes after each section and at the end.
With the computational simulations, I generate a throwaway tool in which I can explore the state space. This doesn't happen as often, as I try to keep state spaces small, and for those explainers tend to work well.
With the design simulations, a same-model agent and a cross-model agent and I will talk about an explainer, and we'll name things that we understand or notice about the design and the intended direction we want to extend it. Then, we'll use subagents to generate options and then discuss the pros and cons. I find this is a good way for me to understand the dead-ends of a design.
Using Verifiers and Guardrails
I define guardrails as mechanistic checks that doesn't change with the feature. such as linters, typecheckers, and pre-commit hooks are all guardrails. Verifiers are mechanistic checks that will be different with each feature, such as unit/integration tests.
For more complicated states or concurrent parts, I'll ask the agent to use formal methods to write a proof. Then use the proof to generate a trace (a series of steps that demonstrate a happy or unhappy path). Then I run that trace in unit tests (kinda like property-based testing--so I guess this can be called trace-based testing), and the code has to adhere to the trace.
I also wrote a design verifier, where the requirements in the spec can be converted into a series of mechanically checkable rules. Then during testing, the verifier fires up the headless browser and uses the DOM measurements of the page to see if it matches up to the rules. This was completely impractical before agents.
All of this so a) I can be sure that the agent is doing what it's supposed to with the decisions that it's making b) I can be out of the coding loop, so that I'm not up at 2am begging, "you fixed this, but you broke that. pls fix." The agent can check for itself.
---
If any of this sounds helpful, lemme know. I've been wondering if other engineers care about understanding the code that's being written, or if people just don't care anymore. I've spoken to a few, and it seems to be split. And it may just be a matter of degree and detail. Some like it at a higher level. Some like it nitty gritty.
That's the neat part, you don't.
Or at least, I don't think people should.
To me this completely goes against the whole point of engineering.
IMO If you don't understand the code, then you're just a manager now. Soon your salary may also reflect as such, assuming you get to keep your job.
Claude to Prod. Every day.
Amen. If it works, its good enough.
The CVEs are going to be epic.
I don't understand these types of comments. Aren't the CVEs epic today with human written code? What difference does it make then? You want no CVEs, make software engineering into a real gate-kept engineering professions with accountability chains, government regulations, formal education, etc.
By default Claude chooses old versions of libraries. without something that checks for CVEs you're gonna be shipping criticals like no tomorrow.
Yes, there are more CVEs discovered every day, but, if you don't patch, the hacker doesn't have to try.
> You want no CVEs, make software engineering into a real gate-kept engineering professions with accountability chains, government regulations, formal education, etc.
This will not only not prevent CVEs, it will also make everything a giant pain in the ass
Of course it will. But at the same time I don't see building falling left and right like CVEs in software. So don't tell me agents write bad code, because humans do as well. The only way to prevent this (or minimize), is to turn software engineering into real engineering.
> Aren't the CVEs epic today with human written code? What difference does it make then?
LLMs cause errors at a noticeably higher rate than human devs do. If you lean on them without actually checking what they wrote, the error rate in your software is going to go up.
"If it works, ship it" is very different from "If it works, and I understand it because I wrote it, ship it".
Does the LLM understand it, in detail? Can you trust its interpretation of the information and instructions you provided? Did it independently make an adverse decision you haven't noticed, because you didn't read the code? Was the model missing any information it required -- and would you know if it was?
LLMs depend on humans and human oversight to ensure they have the information required to make the right decisions, to catch when they don't, and to steer them based on your much more complete understanding of the broader context and requirements. You can't rely on them to do that thinking for you (yet?).
I’d argue that most people also don’t understand code they read from their colleagues. Or if they do, then they don’t understand the entire context of the app.
You said LLM need human oversight and it’s true, but it does not imply you have to read the code.
I think we are going towards a place where code no longer makes sense, you just describe what you want in English, and the LLM just builds directly in machine code. You then build different verification tests to make sure the program is doing what it’s suppose to do.
the guidance / guardrails, agent roles, etc are the critical thing now, it's kind of like meta programming AI to get the outcomes you want. Any problems need addressing at this level. You don't need to understand all the code, but you should understand how the system is building up and adjust as needed. This would be an architectural review process you go through with AI. Features to code that should plug into existing architectural structures are generally no problem. It's just boilerplate code AI is generating. But architecture is only one aspect, you should have lots of guidance for all kinds of thing. There's security, performance, correctness, QA, Ux, Deployment, Process, Monitoring etc and having feedback systems for everything. You keep refining and reviewing your guidance and the impact it is having. As well as this you are molding the product for the things you want via feature development. What I'm not sure of is if you are some cog somewhere in a company like Meta whether whatever slice you become responsible for becomes a bit meh if you are mostly just prompting a feature and not much more. As a developer in a smaller company (we are global with a smallish tech team) I get to do a lot of things from interacting with our customers through to the entire backend and get to make critical decisions about nearly everything. Nearly everything in the company is starting to be driven via guidance and feedback loops (including on the guidance itself).
By taking responsibility of it, maybe by saying "Hey this is not a good idea", and if that doesn't work perhaps by switching jobs if that's an option?
The mental gymnastics developers are willing to go through just so they can avoid standing up for something is quite perplexing.
An engineer from Meta not knowing what they're doing is exactly how I imagined engineers at Meta. Thank you for confirming it's an actual reality.
On the other hand, not even surprised anymore most people even try to provide serious answers to this. Welcome to the pure insanity.
The only answer is to not ship code you don't understand. And I think the easiest way to do that is not use AI for ANYTHING except troubleshooting and doc searching. Those that draw a land in the sand and develop this way will be slower in the short term but their work will last out in the long term because it will be more solid.
>How are you handling this?
easy, don't write code with LLMs. If my job forced me to do so I would quit.
I have many questions. This seems shortsighted.
Short-sightedness is quite literally the reason people are using these systems to begin with. If you care about the health and integrity of your code base years down the road you wouldn't be shipping endless amounts of slop you don't understand, Fred Brooks taught us this half a century ago.
But it seems short-sighted to think that "years down the road" will look _anything_ like the present. I can't imagine a future (say, 5 years from now) where we are worried about the code choices we made in the past.
Flat-out, you _need_ to understand the code you're merging. If you don't, ownership then lands on whatever AI agent you're using. Ultimately, you can't fire the AI agent in the same way you can fire the person at the driver seat when shit hits the fan and they've consistently caused incidents that force people to work late hours to resolve it.
I agree with the sentiment but I would add more nuance; you have to pick what is important to understand and what you can delegate; not all code is mission-critical.
In any case at the end you're the owner/accountable so you need to justify your and your agent choices.
Why not just force the llm to be much more concise and require that every line is worth its code. LoC is a inability after all.
It's simple: you don't. You slow down. It's essential to understand what you ship, before you ship, as there are real world consequences for not doing so, ranging from wasting the time and energy of yourself and others, up to failures which cause a great deal of harm to people and the world. To knowingly ship code you don't understand is grossly negligent, regardless of how common it becomes.
I don't ship code I don't understand. I fully review every line if it's going to prod.
My dumb little utils in ~/.local/bin? Those get merged after a quick functional test without my having read a single line of it. Still, the skill I maintain for such a job directs agents to write such throwaway scripts in anything but shellscript because I might want to extend some day and fuck shellscript
> How to deal with shipping AI-written code you don't understand?
Stop shipping code you don't understand.
Is there really any other solution? You can either ignore it and pretend like it's the new era of software like half the other commenters are, or you can sit down and understand the code and what you're shipping.
I wouldn't be surprised if the meta person has a gun to their head to increasingly keep shipping like a maniac (how decade-long bugs remain unfixed don't ask me), so this advice may not be very useful.
> it all feels like a black box.
I have some work in Lean4 that I'm certainly unqualified to validate; Layer Coding QEC derivation, Positive Grassmanians and the Fresnel-Kerchhoff correction to the Parallel-Axis theorem, Huygens intensity as an alternate model for simulating photonic and electronic interference in a double slit experiment, for example.
It was a black box but I'm letting it tutor me about it so
Visual proofs, Tools for logic and formal methods in distributed systems
Things that compile logic!
> That means you can end up with systems you've built where you don't truly understand how all the code works
Though AI coding and system development can be overwhelming due to information overload and rapid growth in complexity (and concern for unwittingly embedded mistakes, the unsecurability of agent-based development, and number of LOC of code per day), not understanding the whole system sufficiently is normal and expected in engineering.
Generally speaking I brain storm the feature first and produce a dev plan. This includes asking it to reuse code/libraries we’ve already got! After it’s gone off ah done its thing we go back over and do a QA on what it’s built, why that approach, what can we do better etc etc?
I spend time to add extra observability, cross-checking and alerting at runtime to offset the risk.
It feels like another layer of "conceptual compression". I'm not sure who/when that phrase came around but in context of Rails is where I heard it and its things like ActiveRecord or really any ORM. You aren't writing the SQL directly, something else is and for the most part it works. Then it doesn't work or your database grows too much and you need to optimize table structures with indexes, etc.
My point is if you don't understand the fundamentals of the underlying code (SQL in the example above), you're more likely to have issues in the future. Or maybe not if that app doesn't reach a usage pattern or complexity where the nuances under the hood matter.
It really depends on what you're shipping. Meta ships social media apps where in the grand scheme of things if something is broken software wise it largely doesn't matter. What's the worst case? lost ad revenue? Users can't do things? If that were software for MRI machines or aerospace where software issues/bugs can result in lives lost then it matters much more in understanding what is going on at all levels of the software and potentially hardware.
I deal with it by not doing it.
You are accountable.
You're going to have to understand it one way or another, either at writing or at debugging. Even if you delegate all debugging to AI, you'll still need to determine what is "right".
If you don't understand the code at writing or debugging, you'll pay the architectural price (where it degrades). That price may be worth paying, but that depends on the system you are building.
Don't.
By doing so, you debase your own profession.
You just stop caring. Everyone else did about three months ago. Time to find something new to care about.
We just don't do that, I don't know why people talk like that is a given
Is it a company culture thing or a domain specific thing?
Anyone who trusts the output of LLMs uncritically, even after the impressive level models and harnesses have gotten to, is borrowing against their future.
Models and harnesses have gotten really good, but they still are hard-capped by the operators that use them. The sycophancy of models is perhaps their greatest weakness, as when a user who might be experienced in one area attempts to use AI to build within an area they are less familiar with, the model does not take a step back and examine the "why" of what is being built from an architectural standpoint, it simply performs the expected task in a direct manner, ultimately creating sloppy work that may be easy to ship but difficult to maintain.
A good example of this was a recent interaction I had with a Dev who was very familiar with how to develop embedded and system level applications. He and every other dev are being asked for increasingly more due to AI, and he had to bring something he had built as a single-user desktop application into a distributed cloud environment. He ended up rsyncing a copy of his working directories including multiple credentials and sensitive internal project documents to a virtual machine binding it to a public ip address. Thankfully our tooling detected this before he could get any farther but it illustrates how the AI model isn't doing any real thinking or engineering at the end of the day. Unless the operator knew to instruct the model about proper cloud architecture and understood best practices around deployment of public systems, the model simply did what it was being told to do by a user who didn't know any better.
Another example of this is the sheer amount of unsophisticated users we've been detecting using Javascript injection into their web browser sessions as a form of automation for web services they may use. These users would never know to ask how to request an API token which may allow them to accomplish things in a much easier and approved way, and instead are relying on the frailest possible automations that expose them to risks of credential exposure.
We've got to be careful with how we engineer things with AI models. Just because we've made boilerplate easier to generate doesn't mean that we can avoid the actual engineering of what we do. And I can't help but think that firms which are fully committed to AI code generation, won't just be introducing an avalanche of slop into their product which will explode the costs of maintenance over time.
If it's important code (and much isn't), then I spend a lot of time using Matt Pocock's /grilling first, and then I get the agent to kick off peer-review with other agents, and surface any disagreements. Some models are better at it than others. This usually forces me to broadly understand the architecture, and I spend most of my time answering the design questions for which there isn't an obvious solution. I have been pretty happy with the code I've launched to production, as measured by the ways it breaks, and by how agents who weren't part of the original committee describe it.
This is the state that founders and decision makers are usually in at a startup. Sure, sometimes you have the coder founder who wrote the first iteration of the product, but many times the founders have no clue how the product works under the hood. They entrust that to the engineers. That was then. Now, much of that engineering layer has been removed, so the founder just needs a smaller team of people who understand how to structure the software and interact with the LLM. LLMs do the rest.
The problem here is that you're trying to do a job that is already being done for you. Maybe you don't yet trust it to do a good job, but that is quickly changing.
If you don't understand it, then you haven't built it.
no one can force you, take responsibility for what you ship, what you choose to ship, how you choose to work, you signed that contract you go to work every day, your company is composed by people like you
don’t wash your hands, or at least don’t expect us to wash them for you
Do not
It's Meta. Shipping crap is par for the course. You're fine.
Replace "AI agents" with "a bunch of human coders" and you'll see the answer. Your product needs decent testing and it's your job as the responsible person in charge to make sure that's in place and working properly. So if anything you probably want to be reviewing the test code. But often that isn't necessary since you can do things like introduce bugs deliberately and see if tests fail; check test logs for information that confirms the tests were run as expected; try some manual testing once in a while; ask for new more advanced tests (e.g. test performance not just functionality); and so on.
> How to deal with shipping AI-written code you don't understand?
Recognise you aren't the one shipping it.
The part that makes me uneasy is when the AI writes both the code and the tests. They might agree with each other and still miss the same thing. How are people here checking for that without doing the whole job twice?
Not sure about that. This was structurally the same before AI: the author of the code wrote the tests. The tests still helped.
The author of the code was physically touching the internals and potentially intuiting obvious shortcomings as they worked on it. That very crucial part is now gone.
I like fault injection (fuzzing, chaos), and mutation testing. I think introducing these helps the AI write better tests and helps having some confidence in the tests that are written.
You should be using two (or even three) different LLMs. For example: Claude to do the coding, and Codex to do an adversarial review. This can go back and forth for several iterations before converging on an acceptably "correct" version of the code...
Occasionally, Claude may not be able to fix the issue – and it may turn out that Codex can, because it has a different approach.
Either way, at least two LLMs is the only way to get (mostly) reliable code.
You can automate this entire orchestration with zeroshot.sh, mentioning it since it is relevant here.
Claude, ChatGPT, Google Gemini, then Replit...
Any software engineer shipping A.I. code should stop calling himself an "engineer."
Imagine the carnage if real engineers started vibe-designing bridges.
I disagree here. Software engineering is a very different form of engineering than something like building a bridge. This has always been true. Even before LLMs, there was a lot of hobbled together code going out there from "real engineers."
I think it's silly to make strong statements like this. _Most_ professional software engineers (women too, not just him!) are using LLMs and are moving some amount of faster as a result.
The people doing this work as a career and not performative art, who were already good at their job, had no worries about this to begin with.
The code was always a black box. Always.
Specs come in, passing tests come out. If you find an issue, your specs were shit.
Anyone that has ever had to outsource work to someone else knows this. If you've ever worked with another team whose codebase you didn't touch - you know this.
You write your specs, you launch an agent to implement them. You launch a 2nd agent to keep the first one honest. When done, you launch as many agents as you have budget to try and poke holes in the implementation. As it always was the case.
You can thrash around, you can fight it. Like it or not, the genie is out of the box, and the realization you were never really good at your job to begin with is gonna set in eventually.