This is significantly more thorough than any testing I've done, and in a totally different domain, but my anecdotal experience getting agents to use Hypothesis was quite poor.
The agent really, really struggled to bridge the gap between the code and the actual business rules it was meant to be modelling. It also struggled to work out which functions at which layer were appropriate to write tests for. So, its tests tended to be very brittle to changes to domain logic.
Essentially, agents have always seemed to struggle with modularity and problem decomposition. Good testing is about finding the right things to test, which means working out how to subdivide the input state into an appropriate product state, and checking each behaviour independently. IMO, this is the most difficult and complicated thing about programming, so I won't say it struggled _more_ than a human would --- but humans have the advantage of being able to sleep on it?
One minor thing I observed was that it tended to get really hung up on floating point edge cases (NaNs, infinities). Maybe floating point edge cases are over-represented in the property-testing training data, but it's essentially irrelevant for my usecase, at least as far as the business rules go.
In my experience, agents often think of more edge cases than humans when writing unit tests. But under the guidance of certain skills, they can become mechanical and lose sight of the business logic.
For example, when I use the Superpowers skill set, the agent proactively adopts TDD for every new feature. But its understanding of testing often stays superficial: if the user asks for a screen with a “Send” button, it first writes a test checking whether the button exists. The test fails, so it adds the button to make it pass.
As a result, the test suite fills up with low-value cases that check whether a property exists or a string matches exactly. The agent follows the “write a failing test, then implement the feature” workflow, but never really tests the business behavior: When should sending be allowed? What should happen after success or failure? How should duplicate submissions be handled?
The problem isn’t that agents can’t write tests. It’s that they seem prone to reducing TDD to a rigid sequence of steps, struggling to independently derive meaningful test cases from business requirements and use them to drive development.
Probably because thats how most companies do it because most corporate workers are lazy box-tickers who are long fed up with the processes. It really feels like we're forcing human processes onto AI.
What actually is the point of TDD?
- If its to force you to think about edge cases early before you've started building the feature then that sounds like a human trait
- If its to be living documentation then that sounds like a human trait
We're going into weird rabbit holes where we've mismatched the tool that is AI which produces extremely cheap code very quickly - with the processes that we've built for slow and expensive to write human-generated code.
It is still early, but I find that this experiment makes little to no sense and it is barely useful.
The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
80%+ of effective testing is not in the testing framework but in the code architecture.
The author doesn't mention how the code is being architected and managed.
For what it is worth, I found that forcing agents on DI/hexagonal architecture and forcing a trivial coverage check is quite useful and produce overall good enough code with relative little effort
> The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure. Coupling to implementation details is unavoidable but should be managed, and ideally minimized, to keep the test suite robust and maintainable.
Given the choice between a test suite that monkey patches out some dependency (redis, say) on a per-test basis, and one which substitutes a fake for this dependency in some centralized composition root, which one do you prefer, and why?
I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.
You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.
If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed?
The article claims that the agents didn’t actually use TDD or mutation testing. While it’s possible for an agent to ignore the TDD procedure even when instructed to follow it, it can’t ignore a build failure.
This surprised me too. I suspect the robots only got a "use mutation testing" prompt and independently decided that it must mean manual mutation testing, or possibly that they ran in a sandbox where an automated mutation testing tool was not installed.
But I'm surprised Dan Luu didn't make a remark about this. Surely he must know the difference!
Results seem somewhat reasonable given that the amount of verus/TLA/Creusot/Lean code out there is tiny compared to all the other non-formal code.
So it's understandable that the agents wont be able
to go beyond proving trival things, given how much more difficult it is to write such code.
A (more) interesting experiment (to me) would be to write a high level spec manually for a non-trivial system (liveness etc.) and see if the agent can produce an implementation using guided refinements that satisfies this specification.
Yeah, the interesting thing to me with formal methods is where you write some (partial) specs to tell the LLM what you want. It'll do the usual stuff, plus extra proof work to make sure your intent was actually realized.
Throwing tools haphazardly at the LLM and hoping they increase the correctness of its output is expectedly pretty ineffective. Good to see this borne out in the article.
I don't know what I'm most impressed by: (a) the testing expertise, (b) the time spent looking into the reasoning mistakes generated by LLMs, or (c) the insane amounts of money this must have cost!
Timely. I'm also looking at this now, actually! We tend to throw benchmark after benchmark at systems, but miss that models are one part of the system. Harnesses are more than models and need tuning too, and in doing so there can be gains or loss of prior tested function as well.
I haven't finished reading and had to make a detour into another article by the same author, so, probably, I will have to extend the comment, but so far this:
> Perhaps the limiting factor is just that knowledge of effective test techniques isn't very widespread
But of course... the problem of testing is a lot harder than performance optimization... I'm surprised this comes as a surprise. Performance optimization has plenty of evaluation metrics by its very nature. Testing? -- I wish there was anything tangible at all... Because we have metrics for optimization, we have theories of optimization, i.e. we have a way of explaining how or what optimization should do. With testing? -- we are nowhere close to this point.
Another aspect of this disparity is that we also know how to sell performance optimizations. It's easy to write into an ad pamphlet that the version 2.0 of gobbledygook does 185% more gobbledygook than the 1.0! (The number faithfully copied from my cereal box!) With testing? -- How can you even tell the customer that the product was tested better? Swear on your life and cross your heart (twice, as opposed to the last time when you only did it once?)
In general, in the field, I've only have so far met with extreme pessimism about feasibility of "theory of testing" existing. Even though the need for testing goes without saying, the actual testing task is reserved for the least competent and there's little no no effort made to improve anything in this department as it's perceived to be a black hole in the budget: no matter how much you could spend on testing, the effect is likely to be the same.
Detail in TFA looks impressive and I promise I'll do a close reading later. But..
The whole premise of the question is hilarious. They change text in an existing one-line comment and the best models in the world think, gee, maybe I'll lint everything AND run 4000 units. Maybe 5000 integration tests too, just to ensure we collide with any other work in progress. So you write the obligatory but often-ignored obvious things into agent memory or project steering markdown or periodic nudges: You must have a hypothesis when you run expensive tests, you must spot check changes first, then start with the most relevant tests only, then move outwards only as necessary to broader labels and only then suites and only then ALL suites in a widening gyre.
But like a falcon ignoring the falconer, the models want to run the everything for anything. So you sigh, you get the model to write a deterministic hook to catch the wrong invocation of the test suite, and you spend weeks refining the rules every time you hit a edge-case, and so it goes. At least you don't have to write the involved regexes by hand, and maybe one day it will be finished..
This is answered by the large plot early in TFA. All tests were made using the same model, and adding no special instructions at all provided performance above the average.
This is significantly more thorough than any testing I've done, and in a totally different domain, but my anecdotal experience getting agents to use Hypothesis was quite poor.
The agent really, really struggled to bridge the gap between the code and the actual business rules it was meant to be modelling. It also struggled to work out which functions at which layer were appropriate to write tests for. So, its tests tended to be very brittle to changes to domain logic.
Essentially, agents have always seemed to struggle with modularity and problem decomposition. Good testing is about finding the right things to test, which means working out how to subdivide the input state into an appropriate product state, and checking each behaviour independently. IMO, this is the most difficult and complicated thing about programming, so I won't say it struggled _more_ than a human would --- but humans have the advantage of being able to sleep on it?
One minor thing I observed was that it tended to get really hung up on floating point edge cases (NaNs, infinities). Maybe floating point edge cases are over-represented in the property-testing training data, but it's essentially irrelevant for my usecase, at least as far as the business rules go.
In my experience, agents often think of more edge cases than humans when writing unit tests. But under the guidance of certain skills, they can become mechanical and lose sight of the business logic.
For example, when I use the Superpowers skill set, the agent proactively adopts TDD for every new feature. But its understanding of testing often stays superficial: if the user asks for a screen with a “Send” button, it first writes a test checking whether the button exists. The test fails, so it adds the button to make it pass.
As a result, the test suite fills up with low-value cases that check whether a property exists or a string matches exactly. The agent follows the “write a failing test, then implement the feature” workflow, but never really tests the business behavior: When should sending be allowed? What should happen after success or failure? How should duplicate submissions be handled?
The problem isn’t that agents can’t write tests. It’s that they seem prone to reducing TDD to a rigid sequence of steps, struggling to independently derive meaningful test cases from business requirements and use them to drive development.
Probably because thats how most companies do it because most corporate workers are lazy box-tickers who are long fed up with the processes. It really feels like we're forcing human processes onto AI.
What actually is the point of TDD? - If its to force you to think about edge cases early before you've started building the feature then that sounds like a human trait - If its to be living documentation then that sounds like a human trait
We're going into weird rabbit holes where we've mismatched the tool that is AI which produces extremely cheap code very quickly - with the processes that we've built for slow and expensive to write human-generated code.
It is still early, but I find that this experiment makes little to no sense and it is barely useful.
The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
80%+ of effective testing is not in the testing framework but in the code architecture.
The author doesn't mention how the code is being architected and managed.
For what it is worth, I found that forcing agents on DI/hexagonal architecture and forcing a trivial coverage check is quite useful and produce overall good enough code with relative little effort
> The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure. Coupling to implementation details is unavoidable but should be managed, and ideally minimized, to keep the test suite robust and maintainable.
Given the choice between a test suite that monkey patches out some dependency (redis, say) on a per-test basis, and one which substitutes a fake for this dependency in some centralized composition root, which one do you prefer, and why?
I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.
I wonder how well this fares
In my experience, agents know how to do it. They just don't if it's not the default style of the language.Go is an amazing AI language for this reason.
The tests that get created also tend to be higher quality than say the slop I see in python.
Though I do suspect my codebases are doing heavy lifting in terms of steering towards quality outcomes.
You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
Most developers are not good at testing. That's why the role of test engineer exists.
I've seen in multiple projects things like assertTrue(true).
I'm sure the agent is better in testing than average enterprise developer.
big tech is moving / has moved away from the role.
I don't think these results show that.
I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.
I thought so too but it's probably because I never review the tests. I made an exception recently and found tons of
or tests for keywords in prompts:If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed?
The article claims that the agents didn’t actually use TDD or mutation testing. While it’s possible for an agent to ignore the TDD procedure even when instructed to follow it, it can’t ignore a build failure.
This surprised me too. I suspect the robots only got a "use mutation testing" prompt and independently decided that it must mean manual mutation testing, or possibly that they ran in a sandbox where an automated mutation testing tool was not installed.
But I'm surprised Dan Luu didn't make a remark about this. Surely he must know the difference!
Would automated mutation testing actually make a big difference for agents, or would they just find new ways to game the metric?
I believe it would. Automated mutation testing is the test coverage metric that is nearly impossible to cheat.
Results seem somewhat reasonable given that the amount of verus/TLA/Creusot/Lean code out there is tiny compared to all the other non-formal code.
So it's understandable that the agents wont be able to go beyond proving trival things, given how much more difficult it is to write such code.
A (more) interesting experiment (to me) would be to write a high level spec manually for a non-trivial system (liveness etc.) and see if the agent can produce an implementation using guided refinements that satisfies this specification.
Yeah, the interesting thing to me with formal methods is where you write some (partial) specs to tell the LLM what you want. It'll do the usual stuff, plus extra proof work to make sure your intent was actually realized.
Throwing tools haphazardly at the LLM and hoping they increase the correctness of its output is expectedly pretty ineffective. Good to see this borne out in the article.
I don't know what I'm most impressed by: (a) the testing expertise, (b) the time spent looking into the reasoning mistakes generated by LLMs, or (c) the insane amounts of money this must have cost!
Timely. I'm also looking at this now, actually! We tend to throw benchmark after benchmark at systems, but miss that models are one part of the system. Harnesses are more than models and need tuning too, and in doing so there can be gains or loss of prior tested function as well.
I haven't finished reading and had to make a detour into another article by the same author, so, probably, I will have to extend the comment, but so far this:
> Perhaps the limiting factor is just that knowledge of effective test techniques isn't very widespread
That linked to: https://danluu.com/testing/
But of course... the problem of testing is a lot harder than performance optimization... I'm surprised this comes as a surprise. Performance optimization has plenty of evaluation metrics by its very nature. Testing? -- I wish there was anything tangible at all... Because we have metrics for optimization, we have theories of optimization, i.e. we have a way of explaining how or what optimization should do. With testing? -- we are nowhere close to this point.
Another aspect of this disparity is that we also know how to sell performance optimizations. It's easy to write into an ad pamphlet that the version 2.0 of gobbledygook does 185% more gobbledygook than the 1.0! (The number faithfully copied from my cereal box!) With testing? -- How can you even tell the customer that the product was tested better? Swear on your life and cross your heart (twice, as opposed to the last time when you only did it once?)
In general, in the field, I've only have so far met with extreme pessimism about feasibility of "theory of testing" existing. Even though the need for testing goes without saying, the actual testing task is reserved for the least competent and there's little no no effort made to improve anything in this department as it's perceived to be a black hole in the budget: no matter how much you could spend on testing, the effect is likely to be the same.
Detail in TFA looks impressive and I promise I'll do a close reading later. But..
The whole premise of the question is hilarious. They change text in an existing one-line comment and the best models in the world think, gee, maybe I'll lint everything AND run 4000 units. Maybe 5000 integration tests too, just to ensure we collide with any other work in progress. So you write the obligatory but often-ignored obvious things into agent memory or project steering markdown or periodic nudges: You must have a hypothesis when you run expensive tests, you must spot check changes first, then start with the most relevant tests only, then move outwards only as necessary to broader labels and only then suites and only then ALL suites in a widening gyre.
But like a falcon ignoring the falconer, the models want to run the everything for anything. So you sigh, you get the model to write a deterministic hook to catch the wrong invocation of the test suite, and you spend weeks refining the rules every time you hit a edge-case, and so it goes. At least you don't have to write the involved regexes by hand, and maybe one day it will be finished..
nice setup for this. did any single technique clearly stand out, or did it mostly come down to the base model?
This is answered by the large plot early in TFA. All tests were made using the same model, and adding no special instructions at all provided performance above the average.