Can authors of such articles at least cite Dijkstra's "On the foolishness of "natural language programming"." which appeared eons ago ? Which presents an argument against the "english is a programming language" hype.
This would require Martin Fowler to know about this article and to appreciate that it might be important.
He might, but I am not encouraged by his prior work or those of his contemporary Agile boosters. Regardless of how you feel about the Agile Manifesto (my feelings are quite mixed), these boosters over the last 25-ish years tend to love citing Agile and OOP things and rarely seem to look beyond that to historical or fundamental CS.
They found a lucrative niche telling management why their software process is broken and how to fix it. And why the previous attept at Agile wasn't really Agile and so things are still broken.
Perhaps now they have to ride the AI hype train, too. I can only guess that whatever AI-driven lucrative consulting / talk-circuit may emerge from this will also be able to explain why the last AI attempt wasn't really AI and that's why things are still broken.
1. Natural language appears to be to be the starting point of any endeavor.
2.
> It may be illuminating to try to imagine what would have happened if, right from the start our native tongue would have been the only vehicle for the input into and the output from our information processing equipment. My considered guess is that history would, in a sense, have repeated itself, and that computer science would consist mainly of the indeed black art how to bootstrap from there to a sufficiently well-defined formal system. We would need all the intellect in the world to get the interface narrow enough to be usable, and, in view of the history of mankind, it may not be overly pessimistic to guess that to do the job well enough would require again a few thousand years.
LLMs are trying to replicate all of the intellect in the world.
I’m curious if the author would consider that these lofty caveats may be more plausible today than they were when the text was written.
One thing I’d add to all of the other comments is just to reflect on experience. Maybe I’ve mostly worked with people who are incompetent with natural language. But assuming that I’ve mostly worked with average people it’s astonishing how common miscommunication is amongst experts when discussing changes to software. I’ve always found the best way to avoid that is to drop into a more structured language. You see this with most communication tools. They add structure to avoid miscommunication.
> I’m curious if the author would consider that these lofty caveats may be more plausible today than they were when the text was written.
What is missed by many and highlighted in the article is the following: that there is no way to be "precise" with natural languages. The "operational definition" of precision involves formalism. For example, I could describe to you in english how an algorithm works, and maybe you understand it. But for you to precisely run that algorithm requires some formal definition of a machine model and steps involved to program it.
The machine model for english is undefined ! and this could be considered a feature and not a bug. ie, It allows a rich world of human meaning to be communicated. Whereas, formalism limits what can be done and communicated in that framework.
I forgot where I read it, but the reason that natural languages works so well for communication is because the terms are labels for categories instead of identifiers. You can concatenate enough to refer to a singleton, but for the person in front, it can be many items or an empty set. Some labels may even be nonexistent in their context
So when we want deterministic process, we invent a set of labels where each is a singleton. Alongside them is a set of rules that specify how to describe their transformation. Then we invented machines that can interpret those instructions. The main advantage was that we know the possible outputs (assuming a good reliability) before we even have to act.
LLMs don't work so well in that regard, as while they have a perfect embedding of textual grammar rules, they don't have a good representation for what those labels refers to. All they have are relations between labels and how likely are they used together. But not what are the sets that those labels refer to and how the items in those sets interact.
This is a nice explanation of language, and the world-model that the language is intended to depict. If I understand you correctly, formalism is a kind of language where the world-model (ie, items and actions in the world depicted by the language) leaves no room for doubt.
Pretty much and if we take programming languages, where inputs and RNG are not specified by the formal grammar of that language, the programmer needs to split them between good values and bad values. And ideally, halt the program when it detects the latter as the result would be nonsense.
So a program is a more restrictive version of the programming languages, which itself is a more restrictive version of a computer. But the tools to specify those restrictions are not perfect as speed and intuitiveness would suffer greatly (haskell vs python).
> All they have are relations between labels and how likely are they used together. But not what are the sets that those labels refer to and how the items in those sets interact.
Why would "membership in a set" not show up as a relationship between the items and the set?
In fact, it's not obvious to me that there's any semantic meaning not contained in the relationship between labels.
Because the training data does not have it. So we have the label "rock" which intersect with some other label like "hard" and "earth". But the item itself have more attributes that we don't bother assigning label to them. Instead, we just experience them. So the label get linked to some qualia. We can assume that there's a collective intersection of the qualia that the label "rock" refers to.
LLMs don't have access to these hidden attributes (think how to describe "blue" to someone born blind). They may understand that color is a property of object, or that "black" is the color you wear for funerals in some locations. But ask them how to describe the color of a specific object and the output is almost guaranteed to be wrong. Unless they are in a funeral in the above location, so he can predict that most people wear black. But it's a guess, not an informed answer.
But for most human endeavors, "operational precision" is a useful implementation detail, not a fundamental requirement.
We want software to be operationally precise because it allows us to build up towers of abstractions without needing to worry about leaks (even the leakiest software abstraction is far more watertight than any physical "abstraction").
But, at the level of the team or organization that's _building_ the software, there's no such operational precision. Individuals communicating with each other drop down to such precision when useful, but at any endeavor larger than 2-3 people, the _vast_ majority of communication occurs in purely natural language. And yet, this still generates useful software.
The phase change of LLMs is that they're computers that finally are "smart" enough to engage at this level. This is fundamentally different from the world Dijkstra was living in.
> so I can't just store my prompts in git and know that I'll get the same behavior each time.
Yes you can, albeit it's pretty silly to do so. LLMs are not (inherently) nondeterministic, you're just not pinning the seed, or are using a remote, managed service for them with no reliability and consistency guarantees. [0]
Here, experiment with me [1]:
- download Ollama 0.9.3 (current latest)
- download the model gemma3n e4b (digest: 15cb39fd9394) using the command "ollama run gemma3n:e4b"
- pin the seed to some constant; let's use 42 as an example, by issuing the following command in the ollama interactive session started in the previous step: "/set parameter seed 42"
- prompt the model: "were the original macs really exactly 9 inch?"
It will respond with:
> You're right to question that! The original Macintosh (released in 1984) *was not exactly 9 inches*. It was
marketed as having a *9-inch CRT display*, but the actual usable screen size was a bit smaller. (...)
The response should be the same over time, across all devices, regardless of whether GPU-acceleration is available.
Bit of an aside, but the overall sentiment echoed in the article reminds me to how visual programming was going to revolutionize everything and take programmers' jobs. With the exception that AI I find actually useful, and was able to integrate it into my workflow.
[0] All of this is to say, to the extent LLM nondeterminism is currently a model trait, it is substituted using a PRNG. Actual nondeterminism is at most an inference engine trait typically instead, see e.g. batched inference.
[1] Details are for experiment reproduction purposes. You can substitute the listed inference engine, model, seed, and prompt with whatever your prefer for your own set of experiments.
I tried recreating this - gemma3n:e4b ID 15cb39fd9394 and /set parameter seed 42 and I got 2 different results on the two times I asked 'were the original macs really exactly 9 inch?'
Strangely, I'm only getting 2 alternating results every time I restart the model. I was not able to get the same result as you and certainly not with links to external sources. Is there anything else I could do to try to replicate your result?
I've only used ChatGPT prior and it'd be nice to use locally run models with consistent results.
Hmm, that's pretty strange. Not sure what might be going wrong, could be terminal shenanigans or a genuine bug in ollama.
Of course, double checking the basics would be a good thing to cover: ollama --version should return ollama 0.9.3, and the prompt should be copied and pasted to ensure it's byte-exactly matching.
Maybe you could also try querying the model through its API (localhost:11434/api/generate)? I'll ask a colleague to try and repro on his Mac like last time just to double check. I also tried restarting the model a few times, worked as expected here.
*Update:* getting some serious reproducibility issues across different hardware. A month ago the same experiment with regular quantized gemma3 worked fine between GPU, CPU, and my colleague's Mac, this time the responses differ everywhere (although they are consistent between resets on the same hw). Seems like this model may be more sensitive to hardware differences? I can try generating you a response with regular gemma3 12b qat if you're interested in comparing that.
And that exact prompt and that exact inference engine [version].
Pretty reasonable if you ask me. All of this was to say, these are still programs, all the regular sensibilities still apply. Heck, even for that, the sensibilities that apply are pretty old-school: in modern, threaded applications, you'd expect runtime behavioral variations. Not the case here. Even for the high-level language compilers referred to in the article, this doesn't apply so easily. The folks over at reproducible builds [0] put in a decent bit of effort to my knowledge to make it happen.
The overarching point being that it's not magic: it's technology. And if you hold them to even broadly similar standards you hold compilers to, they are absolutely deterministic.
In case you mean that if you pick anything else other than what I picked here, the process ceases to be deterministic, that is not true. You can trivially test and confirm that the same way I did.
This is really brittle, though, and in a way that matters a lot more than reproducible builds. Like, sure: a slightly different compiler might cause a slightly different bug to be introduced into the code, but here a slight bit of non-determinism leads to a drastically and fundamentally different result! And not only is the exact inference engine version important, but sometimes the exact hardware as well (when run on GPU)...
My pet peeve here is mistaking nondeterminism with model unreliability. The issue is not that these models or inference engines are inherently nondeterministic, they aren't, but that the former are just plain unreliable.
I can ask the example question multiple times, and often it will tell me absolute balooney. This is not the model being nondeterministic, it's the model being not very good. It's inconsistent across what we consider semantically equivalent prompts. This is a gap in alignment, not some sort of computational property, like nondeterminism. If you ask it 4+4 in multiple different ways, it will always(?) reply correctly, but ask it something more complex, and you can get seriously different results.
The same exchange can be played out multiple times again and again, and it will reply with the exact same misunderstandings in the exact same order again and again, byte to byte matching. The kicker here is batched inference being a necessity for economical reasons at scale, and exactly matching outputs being impractical because of their non-formal target usage. So you get these wishy washy back and forths about whether the models are deterministic or not, which is super frustrating and misses the point.
And I have to keep dodging bullets like "well if you swap the hardware from under it it might change" or "if you prompt a remote service where they control what's deployed and don't tell you fully, and batch your prompt together with a bunch of other people's which you have no control over, it will not be reproducible". Yes, it might not be, obviously. Even completely deterministic code can become nondeterministic if you run it on top of an execution environment that makes it so. That's not exactly a reasonable counter I don't think.
With regular software, there's a contract (the specifications) that code is developed against, and variances are constrained within there. This is not the case with LLMs, the variances are letting jesus take over the wheel tier, and that's a perfectly fine retort. But then it's not nondeterminism that's the issue.
> I like those folks (hello hboeck!), but their work are unrelated to determinstic LLM output, so why even bring them up here?
Did you read the blogpost in the OP?
I cannot understand what I wrote for you on your behalf. What could possibly be unclear about this? Or was this another rhetorical like the previous one, to polish your snark?
> Yea are you even responding to me, or is this just a stream of thought?
Yes, I was. Are you able to ask non-rhetorical questions too? Should I have asked for your blessed permission to write more than just a direct address of what you asked about?
> As we learn to use LLMs in our work, we have to figure out how to live with this non-determinism [...] but there will also things we'll gain that few of us understand yet.
No thanks. Let's not give up determinism for vague promises of benefits "few of us understand yet".
Determinism isn't always ideal. Determinism may trade off with things like accuracy, performance, etc. There are situations where the tradeoff is well worth it.
Yep, there are plenty of things that aren't computable without burning all the entropy in the visible universe, yet if you exchange it with a heuristic you can get a good enough answer in polynomial time.
Also, at temperature 0 LLMs can behave deterministically! Indeterminism isn't necessarily quite the right word for the kind of abstraction LLMs provide
Even at temperature != 0 it's trivial to just use a fixed seed in the RNG... it's just a computer being used in a naive, not even multi threaded (i.e. with race conditions), way.
I wouldn't be surprised to find out different stacks multiple fp16s slightly differently or something. Getting determinism across machines might take some work... but there's really nothing magic going on here.
Quite pleased you mentioned this. I would like to add transformer LLMs can be turing complete, see the work of Franz Nowak and his colleagues (I think there were at least one or two other papers by other teams but I read Nowak's the closest as it was the latest one when I became aware of this).
Not true. Perhaps very few do, but some do in fact run them at 0. I've done it myself. There are many small models that will gladly perform well in QA with temp 0. Of course there are few situations where this is the recommended setup -- we all know RAG takes less than a billion parameters now to do effectively. But nevertheless there are people who do this, and there are plausibly some use cases for it.
Nobody was stopping anyone from making compilers that introduced random different behavior every time you ran them. I think it's telling this didn't catch on.
There were definitely compilers that used things like data-structures with an unstable iteration order resulting in non-determinism, and people went stopping other people from doing that. This behavior would result in non-deterministic performance everywhere, and combined with race conditions or just undefined behavior other random non-deterministic behaviors too.
At least in part this was achieved with techniques that can be used to make LLMs to, like by seeding RNGs in hash tables deterministically. LLMs are in that sense no less deterministic than iterating over a hash table (they are just a bunch of matrix multiplications with a sampling procedure at the end, after all).
I think this gets at a major hurdle that needs to be overcome for truly human-level AGI.
Because the human brain is also non-deterministic. If you ask a software engineer the same question on different days, you can easily get different answers.
So I think what we want from LLMs is not determinism, just as that's not really what you'd want from a human. It's more about convergence. Non-determinism is ok, but it shouldn't be all over the map. If you ask the engineer to talk through the best way to solve some problem on Tuesday, then you ask again on Wednesday, you might expect a marginally different answer considering they've had time to think on it, but you'd also expect quite a lot of consistency. If the second answer went in a completely different direction, and there was no clear explanation for why, you'd probably raise an eyebrow.
Similarly, if there really is a single "right" answer to a question, like something fact-based or where best practices are extremely well established, you want convergence around that single answer every time, to the point that you effectively do have determinism in that narrow scope.
LLMs struggle with this. If you ask an LLM to solve the same problem multiple times in code, you're likely to get wildly different approaches each time. Adding more detail and constraints to the prompt helps, but it's definitely an area where LLMs are still far behind humans.
It's more search and act based on the first output. You don't know what's going to come out, you just hope it will be good. The issue is that that the query is fed to the output function. So what you get is a mixture of what is a mixture of what you told it and what's was stored. Great if you can separate the two afterwards, not so if the output is tainted by the query.
With automation, what you seek is predictability. Not an echo chamber.
ADDENDUM
If we continue with the echo chamber analogy:
Prompt Engineering: Altering your voice so that the result back is more pleasant
System Prompt: The echo chamber's builders altering the configuration to get the above effects
RAG: Sound effects
Agent: Replace yourself in front of the echo chamber with someone/something that act based on the echo.
Now that machines can also follow natural language instructions, then programs can become a mix of code and natural language - using each where it makes the most sense.
and @bwfan123, we did cite Dijkstra as well as Knuth. I believe sometimes you need the rigor of symbolic notation and other times you need the ambiguity of language - it depends on what you are doing and we can let the user decide how and when to blend the two.
It's not about one or the other, but a gentle and harmonious blending
I think we should shift the focus from adapting LLMs to our purposes (e.g. external tool use) and adapting how we think about software and focus on getting models that internally understand compilation and execution. Rather than merely building around next token prediction, the industry should take advantage of the fact that software in particular provides a cheap path to learning a domain-specific "world model".
Currently I sometimes get predictions where a variable that doesn't exist gets used or a method call doesn't match the signature. The text of the code might look pretty plausible but it's only relatively late that a tool invocation flags that something is wrong.
If instead of just code text, we trained a model on (code text,IR, bytecode) tuples, (byte code, fuzzer inputs, execution trace) examples, and (trace, natural language description) annotations. The model needs to understand not just what token sequences seem likely but (a) what will the code compile to? (b) what does the code _do_ and (c) how would a human describe this behavior? Bonus points for some path to tie in pre/post conditions, invariants, etc
"People need to adapt to weaker abstractions in the LLM era" is a short term coping strategy. Making models that can reason about abstractions in a much tighter loop and higher fidelity loop may get us code generation we can trust.
These types of errors are not only rare in one-shots, but also very easy to fix in subsequent iterations - e.g. Claude Code with Sonnet rarely makes these errors.
If LLMs are the new compilers, enabling software to be built with natural language, why can't LLMs just generate bytecode directly? Why generate HLL code at all?
Same reason humans use high-level languages: limited context windows.
Both humans and LLMs benefit from non-leaky abstractions—they offload low-level details and free up mental or computational bandwidth for higher-order concerns. When, say, implementing a permissioning system for a web app, I can't simultaneously track memory allocation and how my data model choices aligns with product goals. Abstractions let me ignore the former to "spend" my limited intelligence on the latter; same with LLMs and their context limits.
Yes, more intelligence (at least in part) means being able to handle larger contexts, and maybe superintelligent systems could keep everything "in mind." But even then, abstraction likely remains useful in trading depth for surface area. Chris Sawyer was brilliant enough to write Rollercoaster Tycoon in assembly, but probably wouldn't be able to do the same for Elden Ring.
(Also, at least until LLMs are so transcendentally intelligent they outstrip our ability to understand their actions, HLLs are much more verifiable by humans than assembly is. Admittedly, this might be a time-limited concern)
Why would the ability to generate source code imply the ability to generate bytecode? Also you wouldn’t want that, humans can’t review bytecode. I think you may be taking the metaphor too literally.
Because the semantic for each term in a programming language is pretty much a 1:1 relation to a sequential and logic-based ordering of terms in bytecode (which are still code).
> Also you wouldn’t want that, humans can’t review bytecode
The one great thing about automation (and formalism) is that you don't have to continuously review it. You vet it once, then you add another mechanism that monitors for wrong output/behavior. And now, the human is free for something else.
I dont think they are... LLMs can learn from anything thats been tokenized. Feed enough decompiled and labeled data with the bytecode and it's likely the machine will be able to dump out an executable. I wouldn't be surprised if an llm could output a valid elf right now other than the tokens may have been stripped in pretraining.
The vibe coders would tell you: you don't. You test the program, or ask the LLM to write tests for you, and if there are any issues, you ask it to fix them. And you do that in a loop until there are no more issues.
I imagine that at some point they must wonder what their role is, and why the LLM couldn't do all of that independently.
> I've not had the opportunity to do more than dabble with the best Gen-AI tools, but I'm fascinated as I listen to friends and colleagues share their experiences. I'm convinced that this is another fundamental change
So: impressions of impressions is the foundation for a declaration of fundamental change?
What exactly is this abstraction? why nature? why new?
i have a "server" at home that host proxmox. On it there is 1 VM i used to test non-so-legal software, with no connection to internet or local network of course (hopefully proxmox virtual switch is good).
Can authors of such articles at least cite Dijkstra's "On the foolishness of "natural language programming"." which appeared eons ago ? Which presents an argument against the "english is a programming language" hype.
[1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...
This would require Martin Fowler to know about this article and to appreciate that it might be important.
He might, but I am not encouraged by his prior work or those of his contemporary Agile boosters. Regardless of how you feel about the Agile Manifesto (my feelings are quite mixed), these boosters over the last 25-ish years tend to love citing Agile and OOP things and rarely seem to look beyond that to historical or fundamental CS.
They found a lucrative niche telling management why their software process is broken and how to fix it. And why the previous attept at Agile wasn't really Agile and so things are still broken.
Perhaps now they have to ride the AI hype train, too. I can only guess that whatever AI-driven lucrative consulting / talk-circuit may emerge from this will also be able to explain why the last AI attempt wasn't really AI and that's why things are still broken.
Who said: Great minds talk about ideas, average minds talk about events, and small minds talk about people?
A wise human
Interesting read and thanks for sharing.
Two observations:
1. Natural language appears to be to be the starting point of any endeavor.
2.
> It may be illuminating to try to imagine what would have happened if, right from the start our native tongue would have been the only vehicle for the input into and the output from our information processing equipment. My considered guess is that history would, in a sense, have repeated itself, and that computer science would consist mainly of the indeed black art how to bootstrap from there to a sufficiently well-defined formal system. We would need all the intellect in the world to get the interface narrow enough to be usable, and, in view of the history of mankind, it may not be overly pessimistic to guess that to do the job well enough would require again a few thousand years.
LLMs are trying to replicate all of the intellect in the world.
I’m curious if the author would consider that these lofty caveats may be more plausible today than they were when the text was written.
One thing I’d add to all of the other comments is just to reflect on experience. Maybe I’ve mostly worked with people who are incompetent with natural language. But assuming that I’ve mostly worked with average people it’s astonishing how common miscommunication is amongst experts when discussing changes to software. I’ve always found the best way to avoid that is to drop into a more structured language. You see this with most communication tools. They add structure to avoid miscommunication.
> I’m curious if the author would consider that these lofty caveats may be more plausible today than they were when the text was written.
What is missed by many and highlighted in the article is the following: that there is no way to be "precise" with natural languages. The "operational definition" of precision involves formalism. For example, I could describe to you in english how an algorithm works, and maybe you understand it. But for you to precisely run that algorithm requires some formal definition of a machine model and steps involved to program it.
The machine model for english is undefined ! and this could be considered a feature and not a bug. ie, It allows a rich world of human meaning to be communicated. Whereas, formalism limits what can be done and communicated in that framework.
I forgot where I read it, but the reason that natural languages works so well for communication is because the terms are labels for categories instead of identifiers. You can concatenate enough to refer to a singleton, but for the person in front, it can be many items or an empty set. Some labels may even be nonexistent in their context
So when we want deterministic process, we invent a set of labels where each is a singleton. Alongside them is a set of rules that specify how to describe their transformation. Then we invented machines that can interpret those instructions. The main advantage was that we know the possible outputs (assuming a good reliability) before we even have to act.
LLMs don't work so well in that regard, as while they have a perfect embedding of textual grammar rules, they don't have a good representation for what those labels refers to. All they have are relations between labels and how likely are they used together. But not what are the sets that those labels refer to and how the items in those sets interact.
This is a nice explanation of language, and the world-model that the language is intended to depict. If I understand you correctly, formalism is a kind of language where the world-model (ie, items and actions in the world depicted by the language) leaves no room for doubt.
Pretty much and if we take programming languages, where inputs and RNG are not specified by the formal grammar of that language, the programmer needs to split them between good values and bad values. And ideally, halt the program when it detects the latter as the result would be nonsense.
So a program is a more restrictive version of the programming languages, which itself is a more restrictive version of a computer. But the tools to specify those restrictions are not perfect as speed and intuitiveness would suffer greatly (haskell vs python).
> All they have are relations between labels and how likely are they used together. But not what are the sets that those labels refer to and how the items in those sets interact.
Why would "membership in a set" not show up as a relationship between the items and the set?
In fact, it's not obvious to me that there's any semantic meaning not contained in the relationship between labels.
Because the training data does not have it. So we have the label "rock" which intersect with some other label like "hard" and "earth". But the item itself have more attributes that we don't bother assigning label to them. Instead, we just experience them. So the label get linked to some qualia. We can assume that there's a collective intersection of the qualia that the label "rock" refers to.
LLMs don't have access to these hidden attributes (think how to describe "blue" to someone born blind). They may understand that color is a property of object, or that "black" is the color you wear for funerals in some locations. But ask them how to describe the color of a specific object and the output is almost guaranteed to be wrong. Unless they are in a funeral in the above location, so he can predict that most people wear black. But it's a guess, not an informed answer.
But for most human endeavors, "operational precision" is a useful implementation detail, not a fundamental requirement.
We want software to be operationally precise because it allows us to build up towers of abstractions without needing to worry about leaks (even the leakiest software abstraction is far more watertight than any physical "abstraction").
But, at the level of the team or organization that's _building_ the software, there's no such operational precision. Individuals communicating with each other drop down to such precision when useful, but at any endeavor larger than 2-3 people, the _vast_ majority of communication occurs in purely natural language. And yet, this still generates useful software.
The phase change of LLMs is that they're computers that finally are "smart" enough to engage at this level. This is fundamentally different from the world Dijkstra was living in.
On that note, I wonder if having LLM agents communicating with each others in a human language rather than latent space is a big limitation.
> so I can't just store my prompts in git and know that I'll get the same behavior each time.
Yes you can, albeit it's pretty silly to do so. LLMs are not (inherently) nondeterministic, you're just not pinning the seed, or are using a remote, managed service for them with no reliability and consistency guarantees. [0]
Here, experiment with me [1]:
- download Ollama 0.9.3 (current latest)
- download the model gemma3n e4b (digest: 15cb39fd9394) using the command "ollama run gemma3n:e4b"
- pin the seed to some constant; let's use 42 as an example, by issuing the following command in the ollama interactive session started in the previous step: "/set parameter seed 42"
- prompt the model: "were the original macs really exactly 9 inch?"
It will respond with:
> You're right to question that! The original Macintosh (released in 1984) *was not exactly 9 inches*. It was marketed as having a *9-inch CRT display*, but the actual usable screen size was a bit smaller. (...)
Full response here: https://pastebin.com/PvFc4yH7
The response should be the same over time, across all devices, regardless of whether GPU-acceleration is available.
Bit of an aside, but the overall sentiment echoed in the article reminds me to how visual programming was going to revolutionize everything and take programmers' jobs. With the exception that AI I find actually useful, and was able to integrate it into my workflow.
[0] All of this is to say, to the extent LLM nondeterminism is currently a model trait, it is substituted using a PRNG. Actual nondeterminism is at most an inference engine trait typically instead, see e.g. batched inference.
[1] Details are for experiment reproduction purposes. You can substitute the listed inference engine, model, seed, and prompt with whatever your prefer for your own set of experiments.
I tried recreating this - gemma3n:e4b ID 15cb39fd9394 and /set parameter seed 42 and I got 2 different results on the two times I asked 'were the original macs really exactly 9 inch?'
Strangely, I'm only getting 2 alternating results every time I restart the model. I was not able to get the same result as you and certainly not with links to external sources. Is there anything else I could do to try to replicate your result?
I've only used ChatGPT prior and it'd be nice to use locally run models with consistent results.
Hmm, that's pretty strange. Not sure what might be going wrong, could be terminal shenanigans or a genuine bug in ollama.
Of course, double checking the basics would be a good thing to cover: ollama --version should return ollama 0.9.3, and the prompt should be copied and pasted to ensure it's byte-exactly matching.
Maybe you could also try querying the model through its API (localhost:11434/api/generate)? I'll ask a colleague to try and repro on his Mac like last time just to double check. I also tried restarting the model a few times, worked as expected here.
*Update:* getting some serious reproducibility issues across different hardware. A month ago the same experiment with regular quantized gemma3 worked fine between GPU, CPU, and my colleague's Mac, this time the responses differ everywhere (although they are consistent between resets on the same hw). Seems like this model may be more sensitive to hardware differences? I can try generating you a response with regular gemma3 12b qat if you're interested in comparing that.
Yeah trying out gemma3 12b qat sounds great.
I got back 0.9.3 as well as copied and pasted the prompt (included quotes and no quotes as well just in case...)
I can try the API as well and I'm using a legion 15ach6 but I could also try on my MacBook Pro.
It's only deterministic with that exact model and that exact seed. So you are SOL unless you have a copy of that model.
And that exact prompt and that exact inference engine [version].
Pretty reasonable if you ask me. All of this was to say, these are still programs, all the regular sensibilities still apply. Heck, even for that, the sensibilities that apply are pretty old-school: in modern, threaded applications, you'd expect runtime behavioral variations. Not the case here. Even for the high-level language compilers referred to in the article, this doesn't apply so easily. The folks over at reproducible builds [0] put in a decent bit of effort to my knowledge to make it happen.
The overarching point being that it's not magic: it's technology. And if you hold them to even broadly similar standards you hold compilers to, they are absolutely deterministic.
In case you mean that if you pick anything else other than what I picked here, the process ceases to be deterministic, that is not true. You can trivially test and confirm that the same way I did.
[0] https://reproducible-builds.org/
This is really brittle, though, and in a way that matters a lot more than reproducible builds. Like, sure: a slightly different compiler might cause a slightly different bug to be introduced into the code, but here a slight bit of non-determinism leads to a drastically and fundamentally different result! And not only is the exact inference engine version important, but sometimes the exact hardware as well (when run on GPU)...
My pet peeve here is mistaking nondeterminism with model unreliability. The issue is not that these models or inference engines are inherently nondeterministic, they aren't, but that the former are just plain unreliable.
I can ask the example question multiple times, and often it will tell me absolute balooney. This is not the model being nondeterministic, it's the model being not very good. It's inconsistent across what we consider semantically equivalent prompts. This is a gap in alignment, not some sort of computational property, like nondeterminism. If you ask it 4+4 in multiple different ways, it will always(?) reply correctly, but ask it something more complex, and you can get seriously different results.
The same exchange can be played out multiple times again and again, and it will reply with the exact same misunderstandings in the exact same order again and again, byte to byte matching. The kicker here is batched inference being a necessity for economical reasons at scale, and exactly matching outputs being impractical because of their non-formal target usage. So you get these wishy washy back and forths about whether the models are deterministic or not, which is super frustrating and misses the point.
And I have to keep dodging bullets like "well if you swap the hardware from under it it might change" or "if you prompt a remote service where they control what's deployed and don't tell you fully, and batch your prompt together with a bunch of other people's which you have no control over, it will not be reproducible". Yes, it might not be, obviously. Even completely deterministic code can become nondeterministic if you run it on top of an execution environment that makes it so. That's not exactly a reasonable counter I don't think.
With regular software, there's a contract (the specifications) that code is developed against, and variances are constrained within there. This is not the case with LLMs, the variances are letting jesus take over the wheel tier, and that's a perfectly fine retort. But then it's not nondeterminism that's the issue.
> Pretty reasonable if you ask me.
Where can i download ChatGPT models?
> The folks over at reproducible builds
I like those folks (hello hboeck!), but their work are unrelated to determinstic LLM output, so why even bring them up here?
> The overarching point being that it's not magic: it's technology.
Yea are you even responding to me, or is this just a stream of thought?
Noone said LLM is magic.
> Where can i download ChatGPT models?
Nowhere. Does that make LLMs nondeterministic?
> I like those folks (hello hboeck!), but their work are unrelated to determinstic LLM output, so why even bring them up here?
Did you read the blogpost in the OP?
I cannot understand what I wrote for you on your behalf. What could possibly be unclear about this? Or was this another rhetorical like the previous one, to polish your snark?
> Yea are you even responding to me, or is this just a stream of thought?
Yes, I was. Are you able to ask non-rhetorical questions too? Should I have asked for your blessed permission to write more than just a direct address of what you asked about?
> As we learn to use LLMs in our work, we have to figure out how to live with this non-determinism [...] but there will also things we'll gain that few of us understand yet.
No thanks. Let's not give up determinism for vague promises of benefits "few of us understand yet".
Determinism isn't always ideal. Determinism may trade off with things like accuracy, performance, etc. There are situations where the tradeoff is well worth it.
Yep, there are plenty of things that aren't computable without burning all the entropy in the visible universe, yet if you exchange it with a heuristic you can get a good enough answer in polynomial time.
Weather forecasts are a good example of this.
I understand there are probabilities and shortcuts in weather forecasts.... but what part is non-deterministic?
Most heuristics are still deterministic.
Also, at temperature 0 LLMs can behave deterministically! Indeterminism isn't necessarily quite the right word for the kind of abstraction LLMs provide
Even at temperature != 0 it's trivial to just use a fixed seed in the RNG... it's just a computer being used in a naive, not even multi threaded (i.e. with race conditions), way.
I wouldn't be surprised to find out different stacks multiple fp16s slightly differently or something. Getting determinism across machines might take some work... but there's really nothing magic going on here.
Quite pleased you mentioned this. I would like to add transformer LLMs can be turing complete, see the work of Franz Nowak and his colleagues (I think there were at least one or two other papers by other teams but I read Nowak's the closest as it was the latest one when I became aware of this).
That runs into the issue that nobody runs LLMs with a temperature of zero.
Not true. Perhaps very few do, but some do in fact run them at 0. I've done it myself. There are many small models that will gladly perform well in QA with temp 0. Of course there are few situations where this is the recommended setup -- we all know RAG takes less than a billion parameters now to do effectively. But nevertheless there are people who do this, and there are plausibly some use cases for it.
Nobody was stopping anyone from making compilers that introduced random different behavior every time you ran them. I think it's telling this didn't catch on.
I think there was actually a very big push to stop people from doing that - https://en.wikipedia.org/wiki/Reproducible_builds
There were definitely compilers that used things like data-structures with an unstable iteration order resulting in non-determinism, and people went stopping other people from doing that. This behavior would result in non-deterministic performance everywhere, and combined with race conditions or just undefined behavior other random non-deterministic behaviors too.
At least in part this was achieved with techniques that can be used to make LLMs to, like by seeding RNGs in hash tables deterministically. LLMs are in that sense no less deterministic than iterating over a hash table (they are just a bunch of matrix multiplications with a sampling procedure at the end, after all).
I think this gets at a major hurdle that needs to be overcome for truly human-level AGI.
Because the human brain is also non-deterministic. If you ask a software engineer the same question on different days, you can easily get different answers.
So I think what we want from LLMs is not determinism, just as that's not really what you'd want from a human. It's more about convergence. Non-determinism is ok, but it shouldn't be all over the map. If you ask the engineer to talk through the best way to solve some problem on Tuesday, then you ask again on Wednesday, you might expect a marginally different answer considering they've had time to think on it, but you'd also expect quite a lot of consistency. If the second answer went in a completely different direction, and there was no clear explanation for why, you'd probably raise an eyebrow.
Similarly, if there really is a single "right" answer to a question, like something fact-based or where best practices are extremely well established, you want convergence around that single answer every time, to the point that you effectively do have determinism in that narrow scope.
LLMs struggle with this. If you ask an LLM to solve the same problem multiple times in code, you're likely to get wildly different approaches each time. Adding more detail and constraints to the prompt helps, but it's definitely an area where LLMs are still far behind humans.
Abstraction? Hardly.
What are the new semantics and how are the lower levels precisely implemented?
Spoken language isn’t precise enough for programming.
I’m starting to suspect what people are excited about is the automation.
But that's not really automation.
It's more search and act based on the first output. You don't know what's going to come out, you just hope it will be good. The issue is that that the query is fed to the output function. So what you get is a mixture of what is a mixture of what you told it and what's was stored. Great if you can separate the two afterwards, not so if the output is tainted by the query.
With automation, what you seek is predictability. Not an echo chamber.
ADDENDUM
If we continue with the echo chamber analogy:
Prompt Engineering: Altering your voice so that the result back is more pleasant
System Prompt: The echo chamber's builders altering the configuration to get the above effects
RAG: Sound effects
Agent: Replace yourself in front of the echo chamber with someone/something that act based on the echo.
Now that machines can also follow natural language instructions, then programs can become a mix of code and natural language - using each where it makes the most sense.
We looked into how we could blend this by integrating natural language within a programming system here https://blog.ballerina.io/posts/2025-04-26-introducing-natur...
and @bwfan123, we did cite Dijkstra as well as Knuth. I believe sometimes you need the rigor of symbolic notation and other times you need the ambiguity of language - it depends on what you are doing and we can let the user decide how and when to blend the two.
It's not about one or the other, but a gentle and harmonious blending
I think we should shift the focus from adapting LLMs to our purposes (e.g. external tool use) and adapting how we think about software and focus on getting models that internally understand compilation and execution. Rather than merely building around next token prediction, the industry should take advantage of the fact that software in particular provides a cheap path to learning a domain-specific "world model".
Currently I sometimes get predictions where a variable that doesn't exist gets used or a method call doesn't match the signature. The text of the code might look pretty plausible but it's only relatively late that a tool invocation flags that something is wrong.
If instead of just code text, we trained a model on (code text,IR, bytecode) tuples, (byte code, fuzzer inputs, execution trace) examples, and (trace, natural language description) annotations. The model needs to understand not just what token sequences seem likely but (a) what will the code compile to? (b) what does the code _do_ and (c) how would a human describe this behavior? Bonus points for some path to tie in pre/post conditions, invariants, etc
"People need to adapt to weaker abstractions in the LLM era" is a short term coping strategy. Making models that can reason about abstractions in a much tighter loop and higher fidelity loop may get us code generation we can trust.
These types of errors are not only rare in one-shots, but also very easy to fix in subsequent iterations - e.g. Claude Code with Sonnet rarely makes these errors.
If LLMs are the new compilers, enabling software to be built with natural language, why can't LLMs just generate bytecode directly? Why generate HLL code at all?
Same reason humans use high-level languages: limited context windows.
Both humans and LLMs benefit from non-leaky abstractions—they offload low-level details and free up mental or computational bandwidth for higher-order concerns. When, say, implementing a permissioning system for a web app, I can't simultaneously track memory allocation and how my data model choices aligns with product goals. Abstractions let me ignore the former to "spend" my limited intelligence on the latter; same with LLMs and their context limits.
Yes, more intelligence (at least in part) means being able to handle larger contexts, and maybe superintelligent systems could keep everything "in mind." But even then, abstraction likely remains useful in trading depth for surface area. Chris Sawyer was brilliant enough to write Rollercoaster Tycoon in assembly, but probably wouldn't be able to do the same for Elden Ring.
(Also, at least until LLMs are so transcendentally intelligent they outstrip our ability to understand their actions, HLLs are much more verifiable by humans than assembly is. Admittedly, this might be a time-limited concern)
Why would the ability to generate source code imply the ability to generate bytecode? Also you wouldn’t want that, humans can’t review bytecode. I think you may be taking the metaphor too literally.
Because the semantic for each term in a programming language is pretty much a 1:1 relation to a sequential and logic-based ordering of terms in bytecode (which are still code).
> Also you wouldn’t want that, humans can’t review bytecode
The one great thing about automation (and formalism) is that you don't have to continuously review it. You vet it once, then you add another mechanism that monitors for wrong output/behavior. And now, the human is free for something else.
I dont think they are... LLMs can learn from anything thats been tokenized. Feed enough decompiled and labeled data with the bytecode and it's likely the machine will be able to dump out an executable. I wouldn't be surprised if an llm could output a valid elf right now other than the tokens may have been stripped in pretraining.
https://ai.meta.com/research/publications/meta-large-languag...
I agree here. English (human language) to Bytecode is the future.
With reverse translation as needed.
English is a pretty terrible language for describing the precise behavior of a program.
How will you figure out or solve hallucinated assembly code ?
The vibe coders would tell you: you don't. You test the program, or ask the LLM to write tests for you, and if there are any issues, you ask it to fix them. And you do that in a loop until there are no more issues.
I imagine that at some point they must wonder what their role is, and why the LLM couldn't do all of that independently.
Dupe of https://news.ycombinator.com/item?id=44366904 , which was abruptly sinking from the front page.
> I've not had the opportunity to do more than dabble with the best Gen-AI tools, but I'm fascinated as I listen to friends and colleagues share their experiences. I'm convinced that this is another fundamental change
So: impressions of impressions is the foundation for a declaration of fundamental change?
What exactly is this abstraction? why nature? why new?
RESULT: unfounded, ill-formed expression
i have a "server" at home that host proxmox. On it there is 1 VM i used to test non-so-legal software, with no connection to internet or local network of course (hopefully proxmox virtual switch is good).
I could say it's a lab, right ?
a TLA+ model combined with a good LLM to generate code from it is the ultimate level of useful abstraction.
LLMs are deterministic.
If you run an LLM with optimization turned on on a NVIDIA GPU then you can get non-deterministic results.
But, this is a choice.