Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment.
Worse, they don't control for breaking the application's tests.
> Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.
Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.
If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.
That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.
> You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.
That's pretty much what we did. We start with a repository, and
- (vibeclean pipeline) make changes to clean analyzer issues (some of them involve moving big code chunks around to reduce cognitive complexity), and tweak the existing test sets when needed.
- (slopify pipeline) inverse, add analyzer issues and increase complexity, add dead code etc.
In both cases, we ensure that the test coverage remains the same, and the tests are passing, before we start our experiments.
To be sure, we had hidden tests that validate whether agent implemented the task appropriately. And this is something we paid a lot of attention to (pass rate, in our paper.)
What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task).
In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.
At best the conclusions are a best case scenario for degraded code quality. That things remain functionally OK with more costly token usage to get work done.
In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs just getting it right at the 1st/2nd attempt.
Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.
You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding.
I wrote down a more detailed post of the various linters I use here:
> Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.
I have legacy endpoints that are no longer used in practice, there for historical reasons, intertwined with existing code etc. They might be marked obsolete, services implementing it are not - agent greps those, builds off of them - produces half legacy garbage.
Linters only handle trivial cases most of us already solved.
Yeah, we have a big struggle with this. We have lots of legacy code that doesn't follow our latest design patterns intermixed with new code. The LLM picks up bad habits depending on what it pulls in to context first. We have AGENTS.md configured with the right way, but old style still slips in. We obviously need to update the old code but on the other hand if it ain't broke why touch it.
I have the agent inject comments that mention that this particular code is legacy and must not be used as a reference, should not be cleaned up, etc. If you have a document that lists all of the reasons not to use or touch some code, the comments can simply be references to it.
Sure, giving agents a deterministic way to run linters is good, but that already implicitly accepts the premise that code hygiene does actually matter. If you agree with that, then you probably also recognize that there are some things that you can't lint for because they're a lot harder to define, and why would you expect an agent to be able to figure that out today when they still need our intervention to even properly run linters?
> You can get the LLM to run a script which checks for all of these
Most of the time. Except for when it forgets to do it.
I think it’s funny that the solution is to use something that is not LLM driven to enforce it.
Also - pre commit hooks aren’t enforced, people will not set them up. You have to run this stuff in CI (which is incredibly annoying given that machines are writing the code in the first place)
This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.
I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…
You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also help agents gain awareness of the overall verification flow including larger things like tests, but that's hard to say for certain.
Putting structural code checks in a precommit hook is arguably better than pulling it into the harness, as it will enforce those constraints no matter whether an agent or human is making the commit.
Writing a custom harness was like 10 bucks of the communist model tokens last month and I ended up with an a actor library, a thing that has a stable identity, writes sutras and knows it wrote the harness itself. Weird.
Adding an auto runner of the unit tests is just... Boring?
If you have a large important financial service to maintain and develop, and such a code basis is part of your backend, I guarantee you that you won't think twice to simply rely on deterministic linters.
The reason is quite obvious if you have dealt with such a huge code basis in production with thousands of developers contributing for decades to it coming from different vendors and countries.
Code has a meaning attached to it. And paradoxically being able to cleanly cut out such dead code raises my suspicion. There is a reason why such code exists in there often times and since almost always stakeholders give a damn about documentation, and developers traditionally have a hard time writing even JavaDocs, JSdocs, whatever and not to mention maintaining them.
In earlier times CPU time was precious and comments were deliberately left off due to space and processing considerations.
So why is this all important?
Because until you cannot find the one guy who uses this code for a good reason, I would never kill it. Good reasons in these cases are almost always so called application owner, an app admin and hosts, who serves according to ITIL specs as deployment and production person.
Dead code can be actually quite lively under the right circumstances. And since sometimes people have to be very creative to serve regulation requirements and compliance, sometimes release pressure or missing tools can make such code an important script or deployment tool or fix for a reboot or whatever.
Believe me, dead code isn't. What you can do is, watch it at least over a period of two years.
Here is why.
Most processes have yearly deadlines. Many fall on the 1.1. of each year, while others somewhere at the end of the year. Some processes need to be served once a year due to compliance to laws.
And why two years then? As I said, human beings. Maybe that one time a guy had an exception running for it or there is a maintainer, who uses a different method - his own dead code so to say - because these scripts are rarely shared and maintainer's best kept secrets. The lesser a company knows, the more important these last line folks feel and they are blackboxes when something is or isn't working. (I hated this, this was not way of working and I changed it. It is not their company and a keeper is a Red Flag for me.)
But rarely the same person will serve the process two times in a row. Vacation times vary as well as positional changes.
Hence the two years period and even then, there are smarter, more easier ways than to use linters.
Hint: it is the frontend first paradigm.
How do I know? Because I invented it. Proof: Huge international Bank. dbCORE.
I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.
It actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing, but the context for agents is taking up the same space and "attention" every time they're run, and they're creations entirely of context, so the quality and examples matter massively.
That's the moment it should take a nap to compress the context. If you are expose the context size to the agent through some diagnostic input it will even do it itself
> I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one.
I guess it depends on what you mean with "better" but almost all the agent-built projects I do with zero regards to code quality, design and architecture ends up with every single agent needing 10+ minutes to do even the easy changes, while the ones where I focused on those things together with the agent, large changes can take 10+ minutes but everything else is solved faster.
I don't have empirical evidence of this yet, I guess I should put together some sort of test to confirm/disconfirm this.
Every time this subject comes up, there are a bunch of takes along the lines of "would you work on a codebase maintained by agents? they'll mess up the code". And I'm asking myself where these people work, because in 20+ years I've yet to see that pristine state of a project that keeps being pristine after the honeymoon greenfield phase, and 50+ people start working on it. Every project devolves in time, old stuff gets patched in a hurry, someone tries to make it better, learns why certain things were done a certain way, hits some undocumented client needs handled by some arcane combination of code + external systems, and so on. If anything, keeping track of what does what in a project is a task where agents can shine, if only in "ask" mode so you can figure out things quicker. Not to mention onboarding and stuff for new team members.
I was reading your comment, agreeing with it but still feeling why this is a bad comment. It just occurred to me that an anecdotal statement like this is the antithesis of scientific discourse. We have a paper here, trying to answer a question, and anecdotal testimonials can only harm the discussion by biasing readers without adding anything of value to let anyone objectively conclude anything on the problem.
The most useful discussion would be if we all read the paper and critique its methodology or results.
I was reading your comment, disagreeing with it but still feeling why this is a good comment. It just occurred to me that this is not science: science must be reproducible and this is just an historical report on artifact that will be unavailable soon.
i mean this is feeling too but im too paranoid and frequently do refactoring and code organization passes and never don't do it, so i cant say i know for sure there is a delta.
though people who complain that llms aren't that great strike me as the type to have messy code bases
One trick I've found that works well is to tell it to refactor, e.g for Python:
Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions.
A variant I've used for Rust code:
The Rust codebase in `/src` has become bloated with several files >1k LoC. Refactor the Rust codebase to fit code organization standards expected of popular open-source Rust code without causing any benchmark performance regressions.
Those types of prompts appear to a) reorganize the code logically and b) do seem to get better performance from the agents because the file names now provide semantic hints to where relevant code resides. For bloated 5k LoC files, the agent has to Read several chunks to find relevant code which is inefficient.
In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.
Exactly. Simply asking agentic coding tools to clean up code bases, to do some targeted refactorings, to enforce things like SOLID principles, and other good practices can result in a lot of easy improvements.
I've noticed a thing where by default, agentic coding tools are reluctant to remove code. Even when you tell them to. It will bend over backwards to keep old code around, to add complexity for allowing that code to still be called, etc. Super annoying if you are basically just prototyping. You basically end up with a lot of dead code, which than confuses things when you try to add to it. But once you know this, you can just ask it to get rid of the legacy stuff.
Keeping the code base clean, actually stimulates AIs to do the right thing. If there are lots of tests, it will add more when creating new functionality. If there's documentation, it will update that without needing to be prompted as well.
As code harnesses improve, a lot of this is probably being built in as well. Which means even less experienced prompters can get decent results.
Asking it to apply the YAGNI principle also sems to work well for trimming codebases down. Generally ask it to review, generate a list of review points, then we go through each one together and I make a decision yes/no on each one (or suggest further modifications).
I can totally see doing this incrementally, but this seems extremely risky to do for the entire codebase in one shot on anything in production. Especially if you don’t have really thoughtful e2e tests of the whole system.
The AGENTS.md also orders the agents to create relevant tests with each added code, so the tests are already present to catch regressions. The benchmarks themselves also serve as de facto tests since they'll typically break if the refactor is done incorrectly.
I think in this context it is actually important to share exact wording that causes the AI to perform well. My favorite is "Do not use your own knowledge."
That's positively incoherent. You can't even interpret an instruction without relying on prior knowledge. Not to mention LLM's are in essence made of knowledge.
That's not what this is. Refactors are necessary because LLMs tend to bloat even with controls against it, but it's ok to make mistakes if they can be easily fixed.
"agent pipelines that [...] clean a messy [repository]"
This feels like a terrible approach, sufficient to condemn the entire study.
Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.
First author here. Please let me offer a clarification. Our notion of "clean" isn't to just ask the agent to write better code. Rather, we give it a list of 50-100s static analyzer rule violations (and code LOC), and ask to remove them. We then check if the rule violations are resolved.
Using LLMs to rewrite code to remove these violations is a rather accepted practice. Sonar's existing one-shot LLM based approach [1] (in production since 1+ year), and a recent agentic approach [2] to do the same work rather well to do this.
I do wonder though, how does metric based factoring compare to expert intuition based factoring? Can the latter be emulated by agents? Are there studies?
Yes, those ones would be at least a somewhat-plausible simulation of a real scenario people care about: a once-clean codebase that was allowed to become messy by a succession of insufficiently-careful vibeslop PRs.
I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a succession of independent agents reimplement each bit of feature work and bugfixes over that period of time), but it's at least a coarse approximation whereas I just don't trust the cleanup methodology to resemble anything real in the first place.
Our initial suggestion was to do something along the lines of your proposal. But we found that when we ask an agent to implement features for `t-n`^th PR, even when we are overly specific makes the code rather divergent, to the point that sometimes the `t-n+1`th task description doesn't really make a lot of sense.
In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1].
I can't imagine how it wouldn't. None of them can fit a real codebase in context and have to browse the code the same way a person would. Doing searches and reading files. If the files are in the places they would be expected to be and things are called what the model or a person would first guess to search then it gets found in the first attempt rather than requiring a deep search and multiple attempts.
An LLM doesnt have to hold the whole codebase in context. Every path, then shift to next paths because you can ignore sections covered. Much like a developer would.
Even if agents can learn to navigate all the stubs and WET crap they leave behind do we really want a code base that no human can follow what's actually going on?
Sure, but that only happens with a clean, concise, human readable codebase. If agents start working directly in binary, it'd be a lot harder to interact and understand what's going on and where the points of failure are..
If anything I've seen them go too far with DRY. Like two small functions have logic that you could separate into a shared helper, but no human programmer would do that because it's an unclean abstraction and breaks next time you want to alter either one at all.
In similar work we're conducting at NJIT, we've found similar results. (We call it "Contextual Quality Contagion") Some of the most interesting comments here are about some of the real-world scenarios that are commonplace in industry: "mixed" quality codebases, as well as codebases with a mix of legacy code patterns with newer "better" patterns, causing the agents to confuse the conventions.
The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.
I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.
This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.
Interesting to see this quantified. Clean structure seems to lower the cognitive load for both humans and agents, which probably explains why naming and modularization matter more than we think.
The way we should define code quality is arguably how easy it is to affect correct changes to the code, that's hard to quantify, but ultimately the thing any code quality metric is trying to capture.
Based on that, it should be a pretty unsurprising conclusion as long as the code quality metrics you are using are reasonable; as long as the quality metric is good (within the context of coding agents), then this is the result we'd expect to see.
In my experience this has an obvious answer (yes). LLM’s yes-and whatever patterns are present in the code base no matter how insane. If you don’t tend to your code garden it will be overcome with weeds in an instant.
At work we had largely moved off of some old hardware but kept partial support for it in the code because normally that would be zero cost. But the constant overhead of bullying, bribing, and threatening the LLMs to leave that code alone was way more work than just excising it and reimplementing it if needed.
A lot of those tokens are traversal - either searching for code or following call-sites. Basically building enough context to be able to work on the task.
You can reduce a lot of the token use for traversal by giving your agent access to some form of LSP in addition to hierarchical direction with your AGENTS.md (or equivalent) for monorepos - but a spread-out codebase is always going to end up requiring some form of traversal to solve each task.
And that traversal isn't just token use - its repeated round trip latency (LLM (queue time -> prefill -> decode -> output) -> Agent (parsing -> tool call -> tool response) -> back to LLM) for EACH step (well, some can be done in parallel, but in practice its mostly sequential) - slowing down the task considerably.
Locality and structure are key when it comes to efficient use of agents. The context window is always bounded and attention across it is inconsistent.
In my experience everything that affects engineers affects an agent. Good abstractions, reasonably sized methods, good names, principled (intra & inter) service architecture, unit tests, etc.
All of these things have historically been the job of engineers, because it helps other people contribute to the code.
Now it helps other people and other agents contribute to the code.
It has taken LLMs to encourage companies to prioritise a clean codebase, tickets with unambiguous context and examples of what is right and wrong when onboarding new team members.
Did the degraded variants also end up with weaker test coverage, or was that controlled separately? Because the fix for "agent burns more tokens understanding the code" is very different from "agent gets confused by dead code" — the first needs better tests and docs, the second just needs a linter pass.
If the models are quantifiably different on cleaner codebases, surely this could be leveraged into a measure of code cleanliness.
I'm not sure if you would call it an objective or subjective measure, a fixed model would be consistent which would provide an objective base for comparison, but other models would be different, so it would be subjective in relation to the model itself.
In AI labs, they can run a task repeatedly for extended periods, eventually marking it as completed in benchmarks. But in the real world, completing tasks quickly and well remains important and valuable.
Another consideration: written by hand, the trade-off of development velocity vs well organised code means that it can be worth taking some tech debt now in order to deliver some value now. (Especially when prototyping etc.).
With coding agents, agents can produce code quicker. The same trade-off still applies.. but, the time it takes an LLM coding agent to write well organised code is still going to be quicker than the time it takes me to write scrappy code.
Yes this is one reason IMO I think of AI code as instant legacy code.
You take on a lot of tech debt. Then you need to do the same work you would do with any legacy app: finding where the brittle points are, what needs better testing, which leads to breaking apart the big ball of mud into cleaner components.
I would assume- yes, if your tokens transport inexperience a inexperienced answer is reflected back to you? Of course self assured as if absolutly right.
Agents struggle with this: DRY means they create helper functions and classes for a testing scenario and maybe it's called two times. When you ask for a refactor of your code and tests to follow, these helper functions are also ignored and then, dead code starts piling up. Specific cleanup sessions always seem to leave residues behind and doing Ralph loops in the first place seems to help with this, but I'm just not satisfied with the overall performance. Any ideas?
Use code quality tools. For TypeScript projects I use Fallow gated with commit hooks. Fallow supports checking dead code, complexity hotspots, duplicate lines etc. Works pretty well
As pure slop coder, after telling agent to clean up my messy projects from earlier models, i.e. ts, 400 line max, and most of my tasks have gone down to consuming fraction of quota as previous and tasks takes significantly less time to finish. I also just tell it to evaluate tasks on compute/token efficiency, i.e. whether to reuse context, or smart new convo/switch models, and now my quota stretch much further. Like I assume agents should be smart enough to route tasks to proper model capabilities in background, but then I ask myself... what economic incentive to they have but to make users token max.
agents are pretty good at cleaning up a codebase, finding dead code, fixing bad abstractions, etc. You just have to spend some focusing the agents on that goal.
If we don't write the codebase, we don't write the tests, and the agents are modifying it, what even considered a messy codebase? Too many files? Too little files? Related code spread accros many files? What is considered a messy codebase?
Have you even seen one ever?
Also, not checking if it breaks unrelated tests is wild, good software is written by modifying the "just right" amount of code to get your result.
If you break _unrelated_ tests than you most likely didn't changed the right amount of code.
Paraphrasing: if I don't write the codebase [but someone else does], I don't write the tests, and other team members are modifying it, what even is considered a messy codebase?
I actually don't see a connection between the mechanism used to create the code and the code messiness. Things like code repetition, incorrect level of abstractions, tests testing only tests themselves, using too smart optimizations for things that don't matter, .. These all can happen in both person- and machine -authored code.
I would be surprised if a professional software developer has never seen at least some aspects of messy codebase in most any large project. The difference can be whether this messiness ever managed, or just piled on.
"Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate. However, it substantially alters the agent's operational footprint: agents working on cleaner code use 7 to 8% fewer tokens and reduce file revisitations by 34%. Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development [...]"
Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment.
Worse, they don't control for breaking the application's tests.
> Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.
Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.
If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.
That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.
> You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.
That's pretty much what we did. We start with a repository, and
- (vibeclean pipeline) make changes to clean analyzer issues (some of them involve moving big code chunks around to reduce cognitive complexity), and tweak the existing test sets when needed.
- (slopify pipeline) inverse, add analyzer issues and increase complexity, add dead code etc.
In both cases, we ensure that the test coverage remains the same, and the tests are passing, before we start our experiments.
> they don't control for breaking the application's tests.
Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
To be sure, we had hidden tests that validate whether agent implemented the task appropriately. And this is something we paid a lot of attention to (pass rate, in our paper.)
What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.
At best the conclusions are a best case scenario for degraded code quality. That things remain functionally OK with more costly token usage to get work done.
Also controlling input
Input matching distribution of a specific one-shotted LLM... Nice.
[dead]
In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs just getting it right at the 1st/2nd attempt.
Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.
You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding.
I wrote down a more detailed post of the various linters I use here:
https://www.balajeerc.info/Use-Deterministic-Guardrails-for-...
> Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.
I have legacy endpoints that are no longer used in practice, there for historical reasons, intertwined with existing code etc. They might be marked obsolete, services implementing it are not - agent greps those, builds off of them - produces half legacy garbage.
Linters only handle trivial cases most of us already solved.
Yeah, we have a big struggle with this. We have lots of legacy code that doesn't follow our latest design patterns intermixed with new code. The LLM picks up bad habits depending on what it pulls in to context first. We have AGENTS.md configured with the right way, but old style still slips in. We obviously need to update the old code but on the other hand if it ain't broke why touch it.
I have the agent inject comments that mention that this particular code is legacy and must not be used as a reference, should not be cleaned up, etc. If you have a document that lists all of the reasons not to use or touch some code, the comments can simply be references to it.
Thanks for the tip. I’ll try that out.
Sure, giving agents a deterministic way to run linters is good, but that already implicitly accepts the premise that code hygiene does actually matter. If you agree with that, then you probably also recognize that there are some things that you can't lint for because they're a lot harder to define, and why would you expect an agent to be able to figure that out today when they still need our intervention to even properly run linters?
> You can get the LLM to run a script which checks for all of these
Most of the time. Except for when it forgets to do it.
I think it’s funny that the solution is to use something that is not LLM driven to enforce it.
Also - pre commit hooks aren’t enforced, people will not set them up. You have to run this stuff in CI (which is incredibly annoying given that machines are writing the code in the first place)
This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.
I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…
You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also help agents gain awareness of the overall verification flow including larger things like tests, but that's hard to say for certain.
Putting structural code checks in a precommit hook is arguably better than pulling it into the harness, as it will enforce those constraints no matter whether an agent or human is making the commit.
> , as it will enforce those constraints no matter whether an agent or human is making the commit.
Unless the agent, or the human don't enable the precommit hooks in the first place.
You don't need the pre commit hooks if you can make the harness hooks.
Belt and suspenders. What if you make a manual code edit? What if your harness hook malfunctions?
What if you don’t enable pre commit hooks?
> what if your harness hook malfunctions?
That’s a bug in the harness and should be enforced
Writing a custom harness was like 10 bucks of the communist model tokens last month and I ended up with an a actor library, a thing that has a stable identity, writes sutras and knows it wrote the harness itself. Weird.
Adding an auto runner of the unit tests is just... Boring?
Why not just fix the code before you make a new feature? Or is that what you mean? Only you automatically fix the code on every prompt?
If you have a large important financial service to maintain and develop, and such a code basis is part of your backend, I guarantee you that you won't think twice to simply rely on deterministic linters.
The reason is quite obvious if you have dealt with such a huge code basis in production with thousands of developers contributing for decades to it coming from different vendors and countries.
Code has a meaning attached to it. And paradoxically being able to cleanly cut out such dead code raises my suspicion. There is a reason why such code exists in there often times and since almost always stakeholders give a damn about documentation, and developers traditionally have a hard time writing even JavaDocs, JSdocs, whatever and not to mention maintaining them.
In earlier times CPU time was precious and comments were deliberately left off due to space and processing considerations.
So why is this all important?
Because until you cannot find the one guy who uses this code for a good reason, I would never kill it. Good reasons in these cases are almost always so called application owner, an app admin and hosts, who serves according to ITIL specs as deployment and production person.
Dead code can be actually quite lively under the right circumstances. And since sometimes people have to be very creative to serve regulation requirements and compliance, sometimes release pressure or missing tools can make such code an important script or deployment tool or fix for a reboot or whatever.
Believe me, dead code isn't. What you can do is, watch it at least over a period of two years.
Here is why.
Most processes have yearly deadlines. Many fall on the 1.1. of each year, while others somewhere at the end of the year. Some processes need to be served once a year due to compliance to laws.
And why two years then? As I said, human beings. Maybe that one time a guy had an exception running for it or there is a maintainer, who uses a different method - his own dead code so to say - because these scripts are rarely shared and maintainer's best kept secrets. The lesser a company knows, the more important these last line folks feel and they are blackboxes when something is or isn't working. (I hated this, this was not way of working and I changed it. It is not their company and a keeper is a Red Flag for me.)
But rarely the same person will serve the process two times in a row. Vacation times vary as well as positional changes.
Hence the two years period and even then, there are smarter, more easier ways than to use linters.
Hint: it is the frontend first paradigm.
How do I know? Because I invented it. Proof: Huge international Bank. dbCORE.
I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.
It actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing, but the context for agents is taking up the same space and "attention" every time they're run, and they're creations entirely of context, so the quality and examples matter massively.
> actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing,
You’ve never had an agent completely lose the plot and forget/confuse its instructions due to the context filling up?
That's the moment it should take a nap to compress the context. If you are expose the context size to the agent through some diagnostic input it will even do it itself
> I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one.
I guess it depends on what you mean with "better" but almost all the agent-built projects I do with zero regards to code quality, design and architecture ends up with every single agent needing 10+ minutes to do even the easy changes, while the ones where I focused on those things together with the agent, large changes can take 10+ minutes but everything else is solved faster.
I don't have empirical evidence of this yet, I guess I should put together some sort of test to confirm/disconfirm this.
Feel the same way myself when working in messy codebases… At some point, the horrible patterns start to rub off…
Every time this subject comes up, there are a bunch of takes along the lines of "would you work on a codebase maintained by agents? they'll mess up the code". And I'm asking myself where these people work, because in 20+ years I've yet to see that pristine state of a project that keeps being pristine after the honeymoon greenfield phase, and 50+ people start working on it. Every project devolves in time, old stuff gets patched in a hurry, someone tries to make it better, learns why certain things were done a certain way, hits some undocumented client needs handled by some arcane combination of code + external systems, and so on. If anything, keeping track of what does what in a project is a task where agents can shine, if only in "ask" mode so you can figure out things quicker. Not to mention onboarding and stuff for new team members.
Entropy is real and with offshore contractors you don't need agents to blame for it
I was reading your comment, agreeing with it but still feeling why this is a bad comment. It just occurred to me that an anecdotal statement like this is the antithesis of scientific discourse. We have a paper here, trying to answer a question, and anecdotal testimonials can only harm the discussion by biasing readers without adding anything of value to let anyone objectively conclude anything on the problem.
The most useful discussion would be if we all read the paper and critique its methodology or results.
I was reading your comment, disagreeing with it but still feeling why this is a good comment. It just occurred to me that this is not science: science must be reproducible and this is just an historical report on artifact that will be unavailable soon.
i mean this is feeling too but im too paranoid and frequently do refactoring and code organization passes and never don't do it, so i cant say i know for sure there is a delta.
though people who complain that llms aren't that great strike me as the type to have messy code bases
One trick I've found that works well is to tell it to refactor, e.g for Python:
A variant I've used for Rust code: Those types of prompts appear to a) reorganize the code logically and b) do seem to get better performance from the agents because the file names now provide semantic hints to where relevant code resides. For bloated 5k LoC files, the agent has to Read several chunks to find relevant code which is inefficient.In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.
Exactly. Simply asking agentic coding tools to clean up code bases, to do some targeted refactorings, to enforce things like SOLID principles, and other good practices can result in a lot of easy improvements.
I've noticed a thing where by default, agentic coding tools are reluctant to remove code. Even when you tell them to. It will bend over backwards to keep old code around, to add complexity for allowing that code to still be called, etc. Super annoying if you are basically just prototyping. You basically end up with a lot of dead code, which than confuses things when you try to add to it. But once you know this, you can just ask it to get rid of the legacy stuff.
Keeping the code base clean, actually stimulates AIs to do the right thing. If there are lots of tests, it will add more when creating new functionality. If there's documentation, it will update that without needing to be prompted as well.
As code harnesses improve, a lot of this is probably being built in as well. Which means even less experienced prompters can get decent results.
Asking it to apply the YAGNI principle also sems to work well for trimming codebases down. Generally ask it to review, generate a list of review points, then we go through each one together and I make a decision yes/no on each one (or suggest further modifications).
I can totally see doing this incrementally, but this seems extremely risky to do for the entire codebase in one shot on anything in production. Especially if you don’t have really thoughtful e2e tests of the whole system.
The AGENTS.md also orders the agents to create relevant tests with each added code, so the tests are already present to catch regressions. The benchmarks themselves also serve as de facto tests since they'll typically break if the refactor is done incorrectly.
I just say "refactor the codebase" and that also works pretty good!
I have my code styling rules in my CLAUDE.md already anyway
> I have my code styling rules in my CLAUDE.md already anyway
Well... that is quite different than just saying "refactor the codebase".
the word your looking for is idiomatic
Yes and Pythonic is the Pythonic word for idiomatic
Pythonic is a specific type of idiomatic; it never hurts to be overly specific with modern agents.
Also, as evident from the different tones in these two prompts, my prompts are often freeform.
The word you are looking for is "you're".
(Can we not play language police? It's boring and doesn't lead to interesting discussion.)
I think in this context it is actually important to share exact wording that causes the AI to perform well. My favorite is "Do not use your own knowledge."
That's positively incoherent. You can't even interpret an instruction without relying on prior knowledge. Not to mention LLM's are in essence made of knowledge.
No, it's a specific direction to look information up in docs or online rather than relying on model training.
It works though.
yeah, me too. I usually ask it to do a code review using SOLID standards and it usually does a good job, if not a little overkill sometimes.
Have you tried telling it:
“Write perfect code, make no mistakes”
I use this one in my Ralph Harness all the time, it’s a classic!
It’s not that it can’t do that, it’s just that you haven’t told it to!
That's not what this is. Refactors are necessary because LLMs tend to bloat even with controls against it, but it's ok to make mistakes if they can be easily fixed.
So wait you’ve told it not to make any mistakes or you haven’t? Seems like you’re missing this one easy trick!
It’s not about making mistakes, it’s about telling it not to make mistakes!
"agent pipelines that [...] clean a messy [repository]"
This feels like a terrible approach, sufficient to condemn the entire study.
Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.
First author here. Please let me offer a clarification. Our notion of "clean" isn't to just ask the agent to write better code. Rather, we give it a list of 50-100s static analyzer rule violations (and code LOC), and ask to remove them. We then check if the rule violations are resolved.
Using LLMs to rewrite code to remove these violations is a rather accepted practice. Sonar's existing one-shot LLM based approach [1] (in production since 1+ year), and a recent agentic approach [2] to do the same work rather well to do this.
[1] https://www.sonarsource.com/solutions/ai/ai-codefix/
[2] https://www.sonarsource.com/products/sonarqube/remediation-a...
I do wonder though, how does metric based factoring compare to expert intuition based factoring? Can the latter be emulated by agents? Are there studies?
Would you trust clean repos that are messed up by AI?
Yes, those ones would be at least a somewhat-plausible simulation of a real scenario people care about: a once-clean codebase that was allowed to become messy by a succession of insufficiently-careful vibeslop PRs.
I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a succession of independent agents reimplement each bit of feature work and bugfixes over that period of time), but it's at least a coarse approximation whereas I just don't trust the cleanup methodology to resemble anything real in the first place.
Our initial suggestion was to do something along the lines of your proposal. But we found that when we ask an agent to implement features for `t-n`^th PR, even when we are overly specific makes the code rather divergent, to the point that sometimes the `t-n+1`th task description doesn't really make a lot of sense.
In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1].
[1] https://arxiv.org/html/2603.24755
No, because the outputs will still be "in distribution" so to speak.
I can't imagine how it wouldn't. None of them can fit a real codebase in context and have to browse the code the same way a person would. Doing searches and reading files. If the files are in the places they would be expected to be and things are called what the model or a person would first guess to search then it gets found in the first attempt rather than requiring a deep search and multiple attempts.
An LLM doesnt have to hold the whole codebase in context. Every path, then shift to next paths because you can ignore sections covered. Much like a developer would.
Even if agents can learn to navigate all the stubs and WET crap they leave behind do we really want a code base that no human can follow what's actually going on?
Even if the agent does everything, English is an imprecise description of what the code does.
So I personally at a minimum will want to talk “in code” about what code does.
Sure, but that only happens with a clean, concise, human readable codebase. If agents start working directly in binary, it'd be a lot harder to interact and understand what's going on and where the points of failure are..
If anything I've seen them go too far with DRY. Like two small functions have logic that you could separate into a shared helper, but no human programmer would do that because it's an unclean abstraction and breaks next time you want to alter either one at all.
In similar work we're conducting at NJIT, we've found similar results. (We call it "Contextual Quality Contagion") Some of the most interesting comments here are about some of the real-world scenarios that are commonplace in industry: "mixed" quality codebases, as well as codebases with a mix of legacy code patterns with newer "better" patterns, causing the agents to confuse the conventions.
The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.
I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.
This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.
Interesting to see this quantified. Clean structure seems to lower the cognitive load for both humans and agents, which probably explains why naming and modularization matter more than we think.
The way we should define code quality is arguably how easy it is to affect correct changes to the code, that's hard to quantify, but ultimately the thing any code quality metric is trying to capture.
Based on that, it should be a pretty unsurprising conclusion as long as the code quality metrics you are using are reasonable; as long as the quality metric is good (within the context of coding agents), then this is the result we'd expect to see.
In my experience this has an obvious answer (yes). LLM’s yes-and whatever patterns are present in the code base no matter how insane. If you don’t tend to your code garden it will be overcome with weeds in an instant.
At work we had largely moved off of some old hardware but kept partial support for it in the code because normally that would be zero cost. But the constant overhead of bullying, bribing, and threatening the LLMs to leave that code alone was way more work than just excising it and reimplementing it if needed.
A lot of those tokens are traversal - either searching for code or following call-sites. Basically building enough context to be able to work on the task.
You can reduce a lot of the token use for traversal by giving your agent access to some form of LSP in addition to hierarchical direction with your AGENTS.md (or equivalent) for monorepos - but a spread-out codebase is always going to end up requiring some form of traversal to solve each task.
And that traversal isn't just token use - its repeated round trip latency (LLM (queue time -> prefill -> decode -> output) -> Agent (parsing -> tool call -> tool response) -> back to LLM) for EACH step (well, some can be done in parallel, but in practice its mostly sequential) - slowing down the task considerably.
Locality and structure are key when it comes to efficient use of agents. The context window is always bounded and attention across it is inconsistent.
In my experience everything that affects engineers affects an agent. Good abstractions, reasonably sized methods, good names, principled (intra & inter) service architecture, unit tests, etc.
All of these things have historically been the job of engineers, because it helps other people contribute to the code.
Now it helps other people and other agents contribute to the code.
It has taken LLMs to encourage companies to prioritise a clean codebase, tickets with unambiguous context and examples of what is right and wrong when onboarding new team members.
If only we cared so much for junior developers.
imo yes, based on my experience with 4+ repos in production built using AI.
1. clean structure means the AI can predictably work because i want deterministic output.
2. basic things like following SRP leads naturally to layered code which makes testing new features and fixing bugs super simple.
3. reviewing code is simpler, because i see ai agents as "servants" i command to do something and thus an overseer is still required
Did the degraded variants also end up with weaker test coverage, or was that controlled separately? Because the fix for "agent burns more tokens understanding the code" is very different from "agent gets confused by dead code" — the first needs better tests and docs, the second just needs a linter pass.
If the models are quantifiably different on cleaner codebases, surely this could be leveraged into a measure of code cleanliness.
I'm not sure if you would call it an objective or subjective measure, a fixed model would be consistent which would provide an objective base for comparison, but other models would be different, so it would be subjective in relation to the model itself.
I wrote this, where possible I use non llm tooling to assess and llm just writes up the findings.
https://github.com/bjcoombs/ai-native-toolkit/tree/main#what...
In the real world, clean code still has value.
In AI labs, they can run a task repeatedly for extended periods, eventually marking it as completed in benchmarks. But in the real world, completing tasks quickly and well remains important and valuable.
Another consideration: written by hand, the trade-off of development velocity vs well organised code means that it can be worth taking some tech debt now in order to deliver some value now. (Especially when prototyping etc.).
With coding agents, agents can produce code quicker. The same trade-off still applies.. but, the time it takes an LLM coding agent to write well organised code is still going to be quicker than the time it takes me to write scrappy code.
Yes this is one reason IMO I think of AI code as instant legacy code.
You take on a lot of tech debt. Then you need to do the same work you would do with any legacy app: finding where the brittle points are, what needs better testing, which leads to breaking apart the big ball of mud into cleaner components.
I would assume- yes, if your tokens transport inexperience a inexperienced answer is reflected back to you? Of course self assured as if absolutly right.
Research done by SonarSource, a company that sells static analysis products. This explains the focus on static analysis, among other things.
Agents struggle with this: DRY means they create helper functions and classes for a testing scenario and maybe it's called two times. When you ask for a refactor of your code and tests to follow, these helper functions are also ignored and then, dead code starts piling up. Specific cleanup sessions always seem to leave residues behind and doing Ralph loops in the first place seems to help with this, but I'm just not satisfied with the overall performance. Any ideas?
Use code quality tools. For TypeScript projects I use Fallow gated with commit hooks. Fallow supports checking dead code, complexity hotspots, duplicate lines etc. Works pretty well
As pure slop coder, after telling agent to clean up my messy projects from earlier models, i.e. ts, 400 line max, and most of my tasks have gone down to consuming fraction of quota as previous and tasks takes significantly less time to finish. I also just tell it to evaluate tasks on compute/token efficiency, i.e. whether to reuse context, or smart new convo/switch models, and now my quota stretch much further. Like I assume agents should be smart enough to route tasks to proper model capabilities in background, but then I ask myself... what economic incentive to they have but to make users token max.
agents are pretty good at cleaning up a codebase, finding dead code, fixing bad abstractions, etc. You just have to spend some focusing the agents on that goal.
the study found clean code helps agents. my takeaway: the agent is better at reading my code than I am, and that is genuinely humiliating.
If we don't write the codebase, we don't write the tests, and the agents are modifying it, what even considered a messy codebase? Too many files? Too little files? Related code spread accros many files? What is considered a messy codebase?
Have you even seen one ever?
Also, not checking if it breaks unrelated tests is wild, good software is written by modifying the "just right" amount of code to get your result.
If you break _unrelated_ tests than you most likely didn't changed the right amount of code.
Idk I call bs.
Paraphrasing: if I don't write the codebase [but someone else does], I don't write the tests, and other team members are modifying it, what even is considered a messy codebase?
I actually don't see a connection between the mechanism used to create the code and the code messiness. Things like code repetition, incorrect level of abstractions, tests testing only tests themselves, using too smart optimizations for things that don't matter, .. These all can happen in both person- and machine -authored code.
I would be surprised if a professional software developer has never seen at least some aspects of messy codebase in most any large project. The difference can be whether this messiness ever managed, or just piled on.
[dead]
[flagged]
[flagged]
[flagged]
"Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate. However, it substantially alters the agent's operational footprint: agents working on cleaner code use 7 to 8% fewer tokens and reduce file revisitations by 34%. Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development [...]"
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[dead]