The amount of cognitive dissonance here is interesting.
I compiled c to asm. Title says the llm did this.
it works! But it's broken.
It generated a bunch of other files! But I only need one.
It couldn't target z80 so I was a human in the loop.
You have to trust it and understand how the Black box works to get n factor gains. But no one knows how these tools actually work and general advice is NOT to trust LLM outputs and the author didn't trust them either... And even the final result has the incorrect tax rates...
I'm not denying LLMs can sort of rewrite small chunks of code in other languages, add comments to code, etc. but the way people talk about them is so snake oily.
Going by any of the major bullet points I would say that the title is wrong, and misleading at best.
The OP seems to be on an LLM spree. They ask LLM to produce code. The code is invariable always broken thanks to hallucinations but they go ahead and post it on HN anyway with a misleading title.
Ah well maybe the blogs are written by an LLM to try to gain "agency" or test how compelling their poopy code is against people on the internet for free... I cannot imagine what would motivate an actual person to be that publicly wrong and frankly hostile about it while maintaining any sense of reputation...
I've tried to get LLMs rewrite a bunch of stuff in Rust, from different languages (JavaScript, Python, C++, C) and I can definitely relate: LLMs cannot be trusted rewriting anything significant without a lot of supervision (realistically, the only thing I've gained was not to have to type boring boilerplate, but that's pretty much it).
And before you say “oh but it's because Rust is too hard”, SOTA LLMs don't have much problem writing Rust code nowadays, and I suspect Rust is actually a better candidate than most language as a target, because the compiler catch so many things and the errors are very explicit, which helps the LLM a lot when doing multi-turn rewriting sessions.
The point isn't and never has been that it's a flawless tool. The point is that you can work the tool and get a working POC of something you'd never attempt to do before in literally a couple hours.
It isn't a compiler, it isn't a logician, it's a lossily-compressed image of the whole internet with English as a query language. Use it within the operational envelope, which is what the author did, with some interesting results and possibly pointing to a large implications on vibe-coding solutions you'd previously pay for.
For this piece of code, one can rewrite it correctly and much more performant in much less time. If the tool only drags you down, you'd better drop it.
Just a note that - 'one can' does not mean everyone can. And this can be applied between any two languages or tasks. An AI will simply have better broad knowledge than practically anyone at a task they are unfamiliar with so they can massively reduce the friction of getting started.
You can. I can't. I don't want to. I don't care if it's slow, I can easily tell if it's correct and I can make the LLM fix incorrectness (in this simple case, anyway).
The point is this project probably wouldn't have happened without an LLM.
Getting the right answer with the wrong process is still a failure.
So even if the result happens to be correct for the example you gave it, this process may not be able to produce correct results for other cases. Or even this case again.
And if you can't understand the process, then you won't know when that happens.
Now, if you're just trying to sell the result. I guess, no harm, no foul. When the process breaks, you'll be the one affected. That's fair play.
But if you're trying to sell the process, you should be able to verify the process. Because otherwise you are selling a broken tool.
I'm reminded of the infamous hn comment when Dropbox was announced
> 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
I'm going to go against the grain and say that AI could feasibly replace a very large percentage of most development teams, even today.
Not because the AI itself could do all of the coding with no developers at all in the room, but the developers who do know how to use it effectively could output so much more than those who don't that they would more than make up for their lost productivity.
Lots of companies have not yet fully embraced AI, and their developers are actually held back by not having access to it as a tool. But as someone who was recently given full access to sophisticated AI tooling, it makes a massive difference in my productivity.
Lots of people don't like to hear this, but if you are not using AI today in some capacity, or your company is lagging in its adoption your career is at risk, especially if you are still relatively young. This said as somebody who originally was a massive AI skeptic, but decided to give it a shot.
Yes, you still need to know how to code. That is not going away. But there will come a time when you yourself write an order of magnitude less code than you do today because you will become more of a reviewer than a developer yourself. Software development as a role will still exist, because in essence our job is to solve problems and build software, it just happens to be we write a lot of code to do that nowadays. But we will reach a breaking point where we don't write much of the code ourselves at all, maybe just some edits, and we review orders of magnitude than we do today.
The fact is that for a lot of projects, you don't write a lot of code. And for the occasion that you do, the issues lies mainly in integrations and requirements/design cycle. I learned Vim and Emacs, not to write code faster, but to edit it faster. I've never been in a position where I said: I wish I could write more code.
In fact, most of my coding happens in an unfocused state. It's when I'm reading code (when there's a gap in the docs), learning a new system (libraries, platform, language), or designing a system (architecture, integration,...) that I give my full attention. The actual writing is mostly Edit/(Compile|Lint|Test)/Fix cycle that actually goes pretty fast in term of iteration and don't use that much mental energy.
> Lots of people don't like to hear this, but if you are not using AI today in some capacity, or your company is lagging in its adoption your career is at risk, especially if you are still relatively young. This said as somebody who originally was a massive AI skeptic, but decided to give it a shot.
> Yes, you still need to know how to code. That is not going away. But there will come a time when you yourself write an order of magnitude less code than you do today because you will become more of a reviewer than a developer yourself. Software development as a role will still exist, because in essence our job is to solve problems and build software, it just happens to be we write a lot of code to do that nowadays. But we will reach a breaking point where we don't write much of the code ourselves at all, maybe just some edits, and we review orders of magnitude than we do today.
spot on - key reason behind authoring this https://ghuntley.com/multi-boxing/ - I see a future where exactly that. SWE's spend more time reviewing code than artisanally crafting it by hand. Instead of the IDE we use today being the tool, it will become something only used exceptional circumstances. Instead, our primary tool will be a code review tool (that doesn't suck) that drives agents.
There are areas (IME - React internal tooling frontends) in which the only missing piece is the 'review UI' and it isn't even that important - vibe coding is good enough. My point is I've been living in the future you speak of for the past couple months and I love it.
Give it a try. You'll be surprised at what can be achieved when the LLM is driven via /specs (business requirements) + /stdlib (steering LLM technical outcomes). The end result, when driven by a good eval loop (property-based tests + compiler that provides soundness such as Haskell or Rust), is code outputted at brrrrrrrr speeds which is high quality.
Having seen and helped fix colleague and stranger generated LLM generated rust and typescript code I would rather not rewrite code all day to make it hold water. Doing that for Haskell would probably give me an aneurysm.
Every test I've ever tried with an LLM to get it to generate code has produced a complete unworkable mess. I have no idea what people are generating with them, but its always far less work to write something that I understand, rather than spend the time trying to fix up a complete disaster that barely even begins to touch the problem I asked it to solve
I'm not saying this is you but I have to ask as someone who has had success using LLM's but originally had this same mentality - what is the most recent model and tool you used to try to generate workable code?
If you tried it even just 3-6 months ago, in just this small amount of time tooling has had such a massive improvement that maybe you haven't tried it recently. I had your same experience when I tried before, but I have readily been able to get LLM's generate thousands of lines of actually useful and readable code for my job.
I tried generating code before and dismissed it because I had similarly bad experiences. But having generated myself now entire apps where I barely wrote any code that were actually usable and productive, including internal tooling, personal apps, and code that is client facing in production today, I don't think this really applies anymore. LLM's are more than capable of producing lots of high quality code given the right tools.
Interesting. Surprisingly, it decided to encode the multiplication and division as addition/subtraction loops, which is incredibly inefficient - multiplying e.g. 32,000 by 32,000 (ignoring the overflow ...) will take 1,024,000,000 iterations, so thousands of seconds on the speccy's humble 4Mhz Z80 (8 instructions, each taking at least 2 T-states ...)
Here is the multiplication loop (division is similar but in reverse, subtracting instead of adding).
mult_loop:
; Check if BC is zero
ld a, b
or c
jr z, mult_done
; Add HL to result
ex de, hl ; DE = multiplier, HL = result
add hl, de ; Add multiplier to result
ex de, hl ; DE = result, HL = multiplier
; Decrement counter
dec bc
jr mult_loop
oh interesting. One thing about the specy, is it's incredibly hard to hook in the eval technique (see middle/bottom of https://ghuntley.com/specs/ ) as deployment is manual/human. So I had to drive it all by hand and just accept it as afaik there's no testing framework. If this was another programming language I would have taken the approach of creating a `cargo bench` over the application and then looping _that_ back into the LLM to identify and resolve the performance issues. I've done it; it works well. Just not on the speccy :P
I've done three major conversions of assembly to C:
1. My Empire game was written in assembler on the PDP-11. To port it to the IBM PC, I rewrote it line by line into C.
2. FutureNet's schematic editor for the IBM PC was a very large assembler program. It was written in assembler because there was no other way to make it fit on the PC. I converted it line by line into C so it would run on a Sun. It took a couple months.
3. I converted some of the Optlink linker, a 32 bit Windows linker, into C. The main difficulty, though, was there was no test suite and it was very very hard to get right. I got maybe 10% of it converted. I eventually abandoned the project because 32 bit Windows code is now obsolete. The other hard part of the project was it was not conventionally divided up into functions. Code would jump into the middle of other functions. Another problem with it was its use of macros and conditional assembly was so hard to decipher that I would assemble the code, run it through my disassembler, and use the latter as my starting point. (Macros are Spawn of the Devil.)
As someone who is generally very excited about the potential of LLMs to improve developer productivity I find this article a bit frustrating.
This isn't a productive way to use an LLM. The example was so trivial you could easily rewrite it from scratch in less time.
That would be irrelevant if the process scaled to more complex applications, but the blog shows it fails to understand or implement the simple example repeatedly. This left me with little hope that the technique scales.
I worry that creating so much hype will lead to some kind of backlash.
At some point the food-tax dropped from 10% to 5% which I guess is good for people who need to eat!
I still do a fair bit of z80 programming for myself, and I'm very familiar with the Spectrum so this was a nice article to see, but it's a bit underwhelming how well this seemed to go.
Yeah, this was pointed out to me after publishing. Specifically, when the ASM was converted over /specs, it incorrectly wrote the amount of sales tax to the specs. Thus, the z80 implementation took that incorrect /spec and made it happen. At this stage, I'm sure if I had put some more care/effort into transpiling something from the source (i.e., connecting two or more LLMs as a check on that process) to specs, this problem could have been avoided.
It could have been avoided, but you didn't notice the problem and therefore couldn't avoid it.
To me it looks like your comparison to Tesla FSD is much more apt than you realize. If you want the "magic" to work you have to trust it, and accept that it will run you over the cliff the first chance it gets.
If you're a true fan you'll then head to hackernews to explain that "if only i had stopped it from running me over the cliff, the technology would have been amazing".
…and it changed the default tax rate from 25% to 10%.
It’s still impressive, but it’s basically advanced autocomplete. You still have to read and check every line and make sure it is doing what you expect it to do.
but yes, indeed you still need to watch it like a hawk when using it outside of a tight eval loop (ie. make all | make property-tests) and/or implement some other form eval loop on the original /spec generation.
Are there any demos showing how to manage an existing codebase? Everyone loves to demo how AI can create new programs with a snap, but the elephant in the room is how well AI works with existing codebases and manages things like naming conventions, APIs from other apps, etc.
That's an interesting question, and I spend plenty of time exploring this at my current client. The best tip I can give you right now is really - it depends.
For example .NET/Java have this horrendous convention of splitting files into separate locations by the hundreds onto the filesystem.
ie. com/yada/repository | com/yada/models | com/yada/controllers | com/yada/services | com/yada/dtos et al
See https://www.youtube.com/watch?t=1507&v=J1-W9O3n7j8&feature=y... for an excellent discussion from folks sharing learnings that this is an anti-pattern for the current generation of AI assistants. By splitting the tests + everything related to the code being modified into separate files the LLM does worse.
Depending on the 'uniqueness' of the codebase and how it has been 'structured for humans' (vs being structured for LLMs - see above) then one will need to do some funky stuff with building custom MCP tools that teach the AI assistants how to work with the codebase...
Can you also share your opinion if you compare the "code to spec" vs "spec to code" phases?
I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense.
PS: I learned BASIC on a friend's ZX Spectrum, and your post made me remember some forgotten childhood memories :) Extra thanks.
> I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense.
It would have been possible to go direct from intel asm to z80 asm without /specs.
> and the other as a human language
There's some research here from a couple years ago over at https://githubnext.com/projects/speclang/ which is all about /specs as the source of truth for creating an app.
I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation becomes dire.
> We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. [0]
I copied all the code that I could from TFA and pasted it into OpenAI's tokenizer. It counted ~15k tokens. Many other tokens were generated in TFA's chat, some of which are not visible to the user. I think it's fair to assume that the entire chat was at least 25k tokens, right? Therefore, I believe that by the end of that chat, 4o's performance was significantly degraded.
I think a major skill to develop for LLM supported coding is to compress a chat after just few thousand tokens into something like a step1.md file. Then start a new chat with "read step1.md" as the first prompt, and so on..
It is sound! The real context window is indeed smaller than the advertised marketing number. This gets interesting in verbose languages such as Java, where I've seen +16k tokens wasted on the classpath/import stanzas within a class. Yick.
The current series of prompts I'm using is somewhat 'manually manage/allocate memory' of the LLM context window.
It starts with a PROMPT.md with this content.
```
create or update implementation_status.md with the implementation status
study /specs and implement what has not been implemented yet in /src
create property-based tests inline in the source file that is being implemented
run "make" to verify the implementation after each change
```
If it goes off the rails (and it does) - restart the chat and use this.
```
@prompt
continue
```
None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.
> None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.
The LLM coding tool I have the most experience with is Windsurf IDE + Sonnet 3.5. In Windsurf you can define both global and project rules in .md files. I have found that managing those files very closely is key to success. They just tried to automate that with auto-generated "Memories" but they are generally slop, and I delete them. Managing the project rule file will save you so much pain. That is where I define the frameworks and APIs to use.
Getting Supabase MCP running recently was another "oh shit!" moment for me.
Prompting to check the schema is great, but prompting against live data in the DB is amazing! For example:
> When logged in as user me@domain.com, I can't see the Edit button on src/pages/projects/show.tsx. Please use MCP to check if I have the correct permissions in organization_roles. If I do, look at the UI code to see if we are checking it correctly."
Describing an annotated ASM program is something I would have thought LLMs should be perfect for, but it failed immediately. The author really published a whole blog post without even mentioning that core business logic wasn't working.
I admittedly don't really understand the purpose behind any of the intermediate LLM steps here. I'd have assumed an ASM to ASM translation would make the most sense and be the least error prone.
There is no comment or string containing "Food", "Hygiene" or etc. in the asm, because rodata is not dumped. If he inputted the listed asm in a separated session, how could LLM predicted them?
It was more about applying /specs, to generate specs. Then applying /stdlib + /specs to generate an application for another platform without using the original reference application/disassembler. Proving a point that the technique works to re-create any software from /specs via LLM...
The amount of cognitive dissonance here is interesting.
I compiled c to asm. Title says the llm did this. it works! But it's broken. It generated a bunch of other files! But I only need one. It couldn't target z80 so I was a human in the loop. You have to trust it and understand how the Black box works to get n factor gains. But no one knows how these tools actually work and general advice is NOT to trust LLM outputs and the author didn't trust them either... And even the final result has the incorrect tax rates...
I'm not denying LLMs can sort of rewrite small chunks of code in other languages, add comments to code, etc. but the way people talk about them is so snake oily.
Going by any of the major bullet points I would say that the title is wrong, and misleading at best.
> Going by any of the major bullet points I would say that the title is wrong, and misleading at best.
And it wouldn't be their first time.
Check https://news.ycombinator.com/item?id=43217357 by the same author. That post got flagged for being misleading too.
Read https://news.ycombinator.com/item?id=43220639 for same type of criticism.
The OP seems to be on an LLM spree. They ask LLM to produce code. The code is invariable always broken thanks to hallucinations but they go ahead and post it on HN anyway with a misleading title.
https://ghuntley.com/about/
Ah well maybe the blogs are written by an LLM to try to gain "agency" or test how compelling their poopy code is against people on the internet for free... I cannot imagine what would motivate an actual person to be that publicly wrong and frankly hostile about it while maintaining any sense of reputation...
I've tried to get LLMs rewrite a bunch of stuff in Rust, from different languages (JavaScript, Python, C++, C) and I can definitely relate: LLMs cannot be trusted rewriting anything significant without a lot of supervision (realistically, the only thing I've gained was not to have to type boring boilerplate, but that's pretty much it).
And before you say “oh but it's because Rust is too hard”, SOTA LLMs don't have much problem writing Rust code nowadays, and I suspect Rust is actually a better candidate than most language as a target, because the compiler catch so many things and the errors are very explicit, which helps the LLM a lot when doing multi-turn rewriting sessions.
You're missing the point.
The point isn't and never has been that it's a flawless tool. The point is that you can work the tool and get a working POC of something you'd never attempt to do before in literally a couple hours.
It isn't a compiler, it isn't a logician, it's a lossily-compressed image of the whole internet with English as a query language. Use it within the operational envelope, which is what the author did, with some interesting results and possibly pointing to a large implications on vibe-coding solutions you'd previously pay for.
For this piece of code, one can rewrite it correctly and much more performant in much less time. If the tool only drags you down, you'd better drop it.
Just a note that - 'one can' does not mean everyone can. And this can be applied between any two languages or tasks. An AI will simply have better broad knowledge than practically anyone at a task they are unfamiliar with so they can massively reduce the friction of getting started.
If you cannot exam the result, you better not touch anything LLM generated.
The estimated run time of this code provided by another poster would have compelled me to seek alternatives as quickly as possible that's for sure...
Again, missing the point.
You can. I can't. I don't want to. I don't care if it's slow, I can easily tell if it's correct and I can make the LLM fix incorrectness (in this simple case, anyway).
The point is this project probably wouldn't have happened without an LLM.
This "project" is a toy application that exists for no other reason than vibe coding evangelism.
> You can. I can't.
> I can easily tell if it's correct and I can make the LLM fix incorrectness (in this simple case, anyway).
Pick one.
I'm pretty sure black box testing is not a dark art around here.
Getting the right answer with the wrong process is still a failure.
So even if the result happens to be correct for the example you gave it, this process may not be able to produce correct results for other cases. Or even this case again.
And if you can't understand the process, then you won't know when that happens.
Now, if you're just trying to sell the result. I guess, no harm, no foul. When the process breaks, you'll be the one affected. That's fair play.
But if you're trying to sell the process, you should be able to verify the process. Because otherwise you are selling a broken tool.
I'm reminded of the infamous hn comment when Dropbox was announced
> 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
https://news.ycombinator.com/item?id=9224
The sounds like exactly the point OP is making. The way LLMs are spoken about implies much more than actually demonstrated.
> The point isn't and never has been that it's a flawless tool
It's a contradiction with all the managers and vibe-coding developers that have been saying for months that it can replace 90% of a development team.
I'm going to go against the grain and say that AI could feasibly replace a very large percentage of most development teams, even today.
Not because the AI itself could do all of the coding with no developers at all in the room, but the developers who do know how to use it effectively could output so much more than those who don't that they would more than make up for their lost productivity.
Lots of companies have not yet fully embraced AI, and their developers are actually held back by not having access to it as a tool. But as someone who was recently given full access to sophisticated AI tooling, it makes a massive difference in my productivity.
Lots of people don't like to hear this, but if you are not using AI today in some capacity, or your company is lagging in its adoption your career is at risk, especially if you are still relatively young. This said as somebody who originally was a massive AI skeptic, but decided to give it a shot.
Yes, you still need to know how to code. That is not going away. But there will come a time when you yourself write an order of magnitude less code than you do today because you will become more of a reviewer than a developer yourself. Software development as a role will still exist, because in essence our job is to solve problems and build software, it just happens to be we write a lot of code to do that nowadays. But we will reach a breaking point where we don't write much of the code ourselves at all, maybe just some edits, and we review orders of magnitude than we do today.
The fact is that for a lot of projects, you don't write a lot of code. And for the occasion that you do, the issues lies mainly in integrations and requirements/design cycle. I learned Vim and Emacs, not to write code faster, but to edit it faster. I've never been in a position where I said: I wish I could write more code.
In fact, most of my coding happens in an unfocused state. It's when I'm reading code (when there's a gap in the docs), learning a new system (libraries, platform, language), or designing a system (architecture, integration,...) that I give my full attention. The actual writing is mostly Edit/(Compile|Lint|Test)/Fix cycle that actually goes pretty fast in term of iteration and don't use that much mental energy.
> Lots of people don't like to hear this, but if you are not using AI today in some capacity, or your company is lagging in its adoption your career is at risk, especially if you are still relatively young. This said as somebody who originally was a massive AI skeptic, but decided to give it a shot.
spot on - key reason behind authoring https://ghuntley.com/ngmi/
> Yes, you still need to know how to code. That is not going away. But there will come a time when you yourself write an order of magnitude less code than you do today because you will become more of a reviewer than a developer yourself. Software development as a role will still exist, because in essence our job is to solve problems and build software, it just happens to be we write a lot of code to do that nowadays. But we will reach a breaking point where we don't write much of the code ourselves at all, maybe just some edits, and we review orders of magnitude than we do today.
spot on - key reason behind authoring this https://ghuntley.com/multi-boxing/ - I see a future where exactly that. SWE's spend more time reviewing code than artisanally crafting it by hand. Instead of the IDE we use today being the tool, it will become something only used exceptional circumstances. Instead, our primary tool will be a code review tool (that doesn't suck) that drives agents.
There are areas (IME - React internal tooling frontends) in which the only missing piece is the 'review UI' and it isn't even that important - vibe coding is good enough. My point is I've been living in the future you speak of for the past couple months and I love it.
Give it a try. You'll be surprised at what can be achieved when the LLM is driven via /specs (business requirements) + /stdlib (steering LLM technical outcomes). The end result, when driven by a good eval loop (property-based tests + compiler that provides soundness such as Haskell or Rust), is code outputted at brrrrrrrr speeds which is high quality.
Having seen and helped fix colleague and stranger generated LLM generated rust and typescript code I would rather not rewrite code all day to make it hold water. Doing that for Haskell would probably give me an aneurysm.
Every test I've ever tried with an LLM to get it to generate code has produced a complete unworkable mess. I have no idea what people are generating with them, but its always far less work to write something that I understand, rather than spend the time trying to fix up a complete disaster that barely even begins to touch the problem I asked it to solve
I'm not saying this is you but I have to ask as someone who has had success using LLM's but originally had this same mentality - what is the most recent model and tool you used to try to generate workable code?
If you tried it even just 3-6 months ago, in just this small amount of time tooling has had such a massive improvement that maybe you haven't tried it recently. I had your same experience when I tried before, but I have readily been able to get LLM's generate thousands of lines of actually useful and readable code for my job.
I tried generating code before and dismissed it because I had similarly bad experiences. But having generated myself now entire apps where I barely wrote any code that were actually usable and productive, including internal tooling, personal apps, and code that is client facing in production today, I don't think this really applies anymore. LLM's are more than capable of producing lots of high quality code given the right tools.
You’re digging your own hole by using “brrrrrrr speeds” as a marketing term. It doesn’t help your overall argument.
Interesting. Surprisingly, it decided to encode the multiplication and division as addition/subtraction loops, which is incredibly inefficient - multiplying e.g. 32,000 by 32,000 (ignoring the overflow ...) will take 1,024,000,000 iterations, so thousands of seconds on the speccy's humble 4Mhz Z80 (8 instructions, each taking at least 2 T-states ...)
Here is the multiplication loop (division is similar but in reverse, subtracting instead of adding).
Well, can you really expect an LLM to regurgitate e.g. the quarter-of-square method [0] without specifically prompting it for it?
[0] https://www.nature.com/articles/040573c0
oh interesting. One thing about the specy, is it's incredibly hard to hook in the eval technique (see middle/bottom of https://ghuntley.com/specs/ ) as deployment is manual/human. So I had to drive it all by hand and just accept it as afaik there's no testing framework. If this was another programming language I would have taken the approach of creating a `cargo bench` over the application and then looping _that_ back into the LLM to identify and resolve the performance issues. I've done it; it works well. Just not on the speccy :P
I've done three major conversions of assembly to C:
1. My Empire game was written in assembler on the PDP-11. To port it to the IBM PC, I rewrote it line by line into C.
2. FutureNet's schematic editor for the IBM PC was a very large assembler program. It was written in assembler because there was no other way to make it fit on the PC. I converted it line by line into C so it would run on a Sun. It took a couple months.
3. I converted some of the Optlink linker, a 32 bit Windows linker, into C. The main difficulty, though, was there was no test suite and it was very very hard to get right. I got maybe 10% of it converted. I eventually abandoned the project because 32 bit Windows code is now obsolete. The other hard part of the project was it was not conventionally divided up into functions. Code would jump into the middle of other functions. Another problem with it was its use of macros and conditional assembly was so hard to decipher that I would assemble the code, run it through my disassembler, and use the latter as my starting point. (Macros are Spawn of the Devil.)
As someone who is generally very excited about the potential of LLMs to improve developer productivity I find this article a bit frustrating.
This isn't a productive way to use an LLM. The example was so trivial you could easily rewrite it from scratch in less time.
That would be irrelevant if the process scaled to more complex applications, but the blog shows it fails to understand or implement the simple example repeatedly. This left me with little hope that the technique scales.
I worry that creating so much hype will lead to some kind of backlash.
If backslash means less demand that drives down prices, I am all up for it, so I can be productive for cheaper.
At some point the food-tax dropped from 10% to 5% which I guess is good for people who need to eat!
I still do a fair bit of z80 programming for myself, and I'm very familiar with the Spectrum so this was a nice article to see, but it's a bit underwhelming how well this seemed to go.
Yeah, this was pointed out to me after publishing. Specifically, when the ASM was converted over /specs, it incorrectly wrote the amount of sales tax to the specs. Thus, the z80 implementation took that incorrect /spec and made it happen. At this stage, I'm sure if I had put some more care/effort into transpiling something from the source (i.e., connecting two or more LLMs as a check on that process) to specs, this problem could have been avoided.
It could have been avoided, but you didn't notice the problem and therefore couldn't avoid it.
To me it looks like your comparison to Tesla FSD is much more apt than you realize. If you want the "magic" to work you have to trust it, and accept that it will run you over the cliff the first chance it gets.
If you're a true fan you'll then head to hackernews to explain that "if only i had stopped it from running me over the cliff, the technology would have been amazing".
…and it changed the default tax rate from 25% to 10%.
It’s still impressive, but it’s basically advanced autocomplete. You still have to read and check every line and make sure it is doing what you expect it to do.
covered here https://news.ycombinator.com/item?id=43387590
but yes, indeed you still need to watch it like a hawk when using it outside of a tight eval loop (ie. make all | make property-tests) and/or implement some other form eval loop on the original /spec generation.
The author says you have to trust the LLMs though for it to work... You can downvote this but read the articles conclusions first...
Are there any demos showing how to manage an existing codebase? Everyone loves to demo how AI can create new programs with a snap, but the elephant in the room is how well AI works with existing codebases and manages things like naming conventions, APIs from other apps, etc.
That's an interesting question, and I spend plenty of time exploring this at my current client. The best tip I can give you right now is really - it depends.
For example .NET/Java have this horrendous convention of splitting files into separate locations by the hundreds onto the filesystem.
ie. com/yada/repository | com/yada/models | com/yada/controllers | com/yada/services | com/yada/dtos et al
See https://www.youtube.com/watch?t=1507&v=J1-W9O3n7j8&feature=y... for an excellent discussion from folks sharing learnings that this is an anti-pattern for the current generation of AI assistants. By splitting the tests + everything related to the code being modified into separate files the LLM does worse.
Depending on the 'uniqueness' of the codebase and how it has been 'structured for humans' (vs being structured for LLMs - see above) then one will need to do some funky stuff with building custom MCP tools that teach the AI assistants how to work with the codebase...
> manages things like naming conventions, APIs from other apps, etc.
These particular concerns can be handled via https://ghuntley.com/stdlib and it works very very well.
Ah yes, the famous “multiple source files” antipattern…
Could it be the LLM simply not up to the task of maintaining an enterprise codebase?
No, it is the hard-won lessons from the history of software engineering that are wrong.
It's not the LLM but the current generation of coding tools that need to be adjusted/tweaked to deal with em, yes.
Thanks for publishing this.
Can you also share your opinion if you compare the "code to spec" vs "spec to code" phases?
I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense.
PS: I learned BASIC on a friend's ZX Spectrum, and your post made me remember some forgotten childhood memories :) Extra thanks.
> I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense.
It would have been possible to go direct from intel asm to z80 asm without /specs.
> and the other as a human language
There's some research here from a couple years ago over at https://githubnext.com/projects/speclang/ which is all about /specs as the source of truth for creating an app.
I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation becomes dire.
> We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. [0]
I copied all the code that I could from TFA and pasted it into OpenAI's tokenizer. It counted ~15k tokens. Many other tokens were generated in TFA's chat, some of which are not visible to the user. I think it's fair to assume that the entire chat was at least 25k tokens, right? Therefore, I believe that by the end of that chat, 4o's performance was significantly degraded.
I think a major skill to develop for LLM supported coding is to compress a chat after just few thousand tokens into something like a step1.md file. Then start a new chat with "read step1.md" as the first prompt, and so on..
Is my logic sound here?
[0] https://arxiv.org/abs/2502.05167
It is sound! The real context window is indeed smaller than the advertised marketing number. This gets interesting in verbose languages such as Java, where I've seen +16k tokens wasted on the classpath/import stanzas within a class. Yick.
The current series of prompts I'm using is somewhat 'manually manage/allocate memory' of the LLM context window.
It starts with a PROMPT.md with this content.
```
create or update implementation_status.md with the implementation status
study /specs and implement what has not been implemented yet in /src
create property-based tests inline in the source file that is being implemented
run "make" to verify the implementation after each change
```
If it goes off the rails (and it does) - restart the chat and use this.
```
@prompt
continue
```
None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.
> None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.
The LLM coding tool I have the most experience with is Windsurf IDE + Sonnet 3.5. In Windsurf you can define both global and project rules in .md files. I have found that managing those files very closely is key to success. They just tried to automate that with auto-generated "Memories" but they are generally slop, and I delete them. Managing the project rule file will save you so much pain. That is where I define the frameworks and APIs to use.
<3 you get it. It's a new skillset to be learned and when it clicks, you get incredible outcomes!
Getting Supabase MCP running recently was another "oh shit!" moment for me.
Prompting to check the schema is great, but prompting against live data in the DB is amazing! For example:
> When logged in as user me@domain.com, I can't see the Edit button on src/pages/projects/show.tsx. Please use MCP to check if I have the correct permissions in organization_roles. If I do, look at the UI code to see if we are checking it correctly."
Wowzers!
Describing an annotated ASM program is something I would have thought LLMs should be perfect for, but it failed immediately. The author really published a whole blog post without even mentioning that core business logic wasn't working.
I admittedly don't really understand the purpose behind any of the intermediate LLM steps here. I'd have assumed an ASM to ASM translation would make the most sense and be the least error prone.
Did he input all the prompts in the same session? It's kinda nonsense bs. Any disassembler is more useful than this.
No it's a bunch of sessions with manual interventions reprompts and attempts all through out according to the author.
Downvote it if you want it's factually correct. Read the article.
There is no comment or string containing "Food", "Hygiene" or etc. in the asm, because rodata is not dumped. If he inputted the listed asm in a separated session, how could LLM predicted them?
It was more about applying /specs, to generate specs. Then applying /stdlib + /specs to generate an application for another platform without using the original reference application/disassembler. Proving a point that the technique works to re-create any software from /specs via LLM...
TLDR: No, it can not.