There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency.
But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot of researchers to try things that are really different. At least I hope so.
It seems pretty short sighted that the funding numbers for memristor startups (for example) are so low so far.
Anyway, assuming that within the next several years more radically different AI hardware and AI architecture paradigms pay off in efficiency gains, the current situation will change. Fully human level AI will be commoditized, and training will be well within the reach of small companies.
I think we should anticipate this given the strong level of need to increase efficiency dramatically, the number of existing research programs, the amount of investment in AI overall, and the history of computation that shows numerous dramatic paradigm shifts.
So anyway "the rest of us" I think should be banding together and making much larger bets on proving and scaling radical new AI hardware paradigms.
But memory-centric compute didn't happen because of Moore's law. (SNNs have the problem that we don't actually know how to use them.) Now that it's gone, it may have a chance, but it still takes a large amount of money thrown into the idea and the people with money are so risk-adverse that they create entire new risks for themselves.
Forward neural networks were very lucky that there existed a mainstream use for the kind of hardware it needed.
> There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency.
Because the alternatives lack the breakthroughs that give them an edge against current-state AI and don't generate the hype like transformers or diffusion models. You have stuff like neuromorphic hardware that is hardly accessible and in its infancy, e.g. SpiNNaker. You have disciplines like Computational Neuroscience that try to model the brain and come up with novel models and algorithms for learning, which, however, are computational expensive or just perform worse than conventional deep learning models and may benefit from neuromorphic hardware. But again, access is difficult to such hardware.
Even in that scenario, what would stop the likes of OpenAI to instead throw 50M+ a day to the new way of doing things and still outcompete smaller fry?
The fastest away to acquire the know-how to do for Big Co is to get the talent who have spent the years in building the new tech.
Poaching, acquihirng or acquisitions and the myriad modern forms we are seeing today have been the tools and will not change.
Owners and beneficiaries of the capital do not change, but that is an artifact of our economic system and is much larger a socio-economic discussion beyond the scope of innovation and research
Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind of efficiency.
If I only have to try 11 things for one of them to be LED lights or electric cars, I'd better get trying. Sure, I might have to empty a wastebasket at some point, but I'll just pay someone for that.
Which prior comment? Top level offers two options plus an 'etc.', second comment adds another and says 'could be', and then first reply to you offers other tech that took decades of R&D to suggest we can't rule out memristors. I don't see what you're referring to.
> Sparse models for example show a lot of promise and more closely track real biological systems.
I think sparsity is a consequence of some other fundamental properties of brain function that we've yet to understand. Just sparsifying the models we've got is not going to lead anywhere, IMO. (For example it's estimated that current AI models are already within 1%-10% of a human brain in terms of "number of parameters" (https://www.beren.io/2022-08-06-The-scale-of-the-brain-vs-ma...).)
The image has been fixed, and the point I'm making is that proprietary models are almost always ahead, and this gap is widening. OS models that are nearly at the same quality are usually distilled versions of proprietary models, or somehow get training data from them. Sometimes, after massive, expensive training runs models are open sourced anyway, and at some point that becomes unsustainable.
The difference between a top model and a model with a similar ELO might seem small, but the value of even a marginal increase in intelligence is extremely high--for example I only use the best coding model for coding, whatever the cost.
There's also lots of evidence that large labs are only getting started. In the past year, they have secured massive amounts of compute, which is still not utilized well. I expect lots of big training runs in the future, which will shift the gap further between OS and proprietary models.
The major problem for these companies is they spend hundreds of millions of dollars training a model, and then someone comes in the next day and distills something almost as good for far less money (still a VERY large sum of money.)
I think I'm getting it now: OS models are getting closer, but only via distillation. Not by training a new frontier model which is out of reach for economic reasons.
Note that distilling a general model is several orders of magnitude more expensive than distilling a task-specific model, which is what I'm trying to promote here. Smart general models make distilling great task specific models with no expert labelers way easier.
> Followed by a picture that is more or less inscrutable.
Yeah. Just to make it explicit - that chart has Deepseek r1 at ... presumably an elo of 1418 and Gemini Pro at 1463. That is comparable to the gap between Magnus Carlsen and Fabiano Caruana [0]. I don't think it is reasonable to complain about that sort of performance gap in practice - it is a capable model. Looking at the spread of scores I don't immediately see why someone even needs to use something in the Top 10, presumably anything above 1363 would be good enough for business, research and personal use.
None of these models have even been around that long, Deepseek was only released in January. The rate of change is massive, I expect to have access to an open source model that is better than anything on this leaderboard next year some time.
We haven't seen a proper npu and we are in the launch of the first consumer grade unified architectures by Nvidia and AMD. The battle of homebrew AI hasn't even started yet.
Hell, we haven’t even seen actual AI yet. This is all just brute-forcing likely patterns of tokens based on a corpus of existing material, not anything brand new or particularly novel. Who would’ve guessed that giving CompSci and Mathematics researchers billions of dollars in funding and millions of GPUs in parallel without the usual constraints of government research would produce the most expensive brute-force algorithms in human history?
I still believe this is going to be an embarrassing chapter of the history of AI when we actually do create it. “Humans - with the sort of hubris only a neoliberal post-war boom period could produce - honestly thought their first serious development in computing (silicon-based mircoprocessors) would lead to Artificial General Intelligence and usher in a utopia of the masses. Instead they squandered their limited resources on a Fool’s Errand, ignoring more important crises that would have far greater impacts on their immediate prosperity in the naive belief they could create a Digital God from Silicon and Electricity alone.”
Even if we somehow create AGI it will only be used to saddle the masses, not free them. There are a lot of people with money who yearn for absolute power over everything. Replacing your capacity to think with a subscription service makes you a serf. Like Spotify, Netflix, Amazon and so on the last 20 years is full of brand new subscriptions that replace the whole concept of ownership so thoroughly that it gets erased. It pumps up the company’s valuation to do that.
Yeh. We're still barely beyond the first few pixels that make up the bottom tail of the S-curve for autonomous type AI everyone imagines
Energy models and other substrates are going to be key, and it has nothing to do with text at all as human intelligence existed before language. It's Newspeak to run a chat bot on what is obviously a computer and call it an intelligence like a human. 1984 like dystopia crap.
You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
Deepseek main run costed $6M. qwen3-30b-a3b probably would cost few $100Ks, which is ranked 13th.
GPU cost of the final model training isn't the biggest chunk of the cost and you can probably replicate results of models like Llama 3 very cheaply. It's the cost of experiments, researchers, data collection which brings overall cost 1 or 2 order of magnitude higher.
It wasn't a lie, it was a misrepresentation of the total cost. It's not hard to calculate the cost of the training though. It takes 6 * active parameters * tokens flops[1]. To get number of seconds you can divide by Flops/s * MFU, where MFU is around 45% for H100 for large enough models[2].
That paper's 5 years old at this point, dating back to when Amodei was still an OpenAI employee. Has any newer work superseded it, or are those assumptions still considered solid?
Those assumptions are still the same. Although now context length has increased more so the n^2 part is non negligible. See the repo for correct flop calculation[1]
DeepSeek came up with several significant optimizations, not just one. And master's students do contribute to leading edge research all the time.
There have really been many significant innovations in hardware, model architecture, and software, allowing companies to keep up with soaring demand and expectations.
But that's always how it's been in high technology. You only really hear about the biggest shifts, but the optimizations are continuous.
True, but I chose the words 'ChatGPT' and 'optimization' for brevity. There are many more eyes on machine learning since ChatGPT came along. There could be simpler techniques yet to discover. What boggles the mind is the $4 trillion parked in Nvidia stock, and wasted if more efficient code lessens the need for expensive GPUs.
There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency.
But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot of researchers to try things that are really different. At least I hope so.
It seems pretty short sighted that the funding numbers for memristor startups (for example) are so low so far.
Anyway, assuming that within the next several years more radically different AI hardware and AI architecture paradigms pay off in efficiency gains, the current situation will change. Fully human level AI will be commoditized, and training will be well within the reach of small companies.
I think we should anticipate this given the strong level of need to increase efficiency dramatically, the number of existing research programs, the amount of investment in AI overall, and the history of computation that shows numerous dramatic paradigm shifts.
So anyway "the rest of us" I think should be banding together and making much larger bets on proving and scaling radical new AI hardware paradigms.
Memristors in particular just won't happen.
But memory-centric compute didn't happen because of Moore's law. (SNNs have the problem that we don't actually know how to use them.) Now that it's gone, it may have a chance, but it still takes a large amount of money thrown into the idea and the people with money are so risk-adverse that they create entire new risks for themselves.
Forward neural networks were very lucky that there existed a mainstream use for the kind of hardware it needed.
>memory-centric compute
This already exists: https://www.cerebras.ai/chip
They claim 44 GB of SRAM at 21 PB/s.
They use separate memory servers, networked memory adjacent adjacent compute with small amounts of fast local memory.
Waferscale severely limits bandwidth once you go beyond SRAM, because with far less chip perimeter per unit area there is less place to hook up IO.
> There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency.
Because the alternatives lack the breakthroughs that give them an edge against current-state AI and don't generate the hype like transformers or diffusion models. You have stuff like neuromorphic hardware that is hardly accessible and in its infancy, e.g. SpiNNaker. You have disciplines like Computational Neuroscience that try to model the brain and come up with novel models and algorithms for learning, which, however, are computational expensive or just perform worse than conventional deep learning models and may benefit from neuromorphic hardware. But again, access is difficult to such hardware.
I think a pretty good chunk of HP's history explains why memristors don't get used in a commercial capacity.
You remember The Machine? I had a vague memory but I had to look it up.
Even in that scenario, what would stop the likes of OpenAI to instead throw 50M+ a day to the new way of doing things and still outcompete smaller fry?
The fastest away to acquire the know-how to do for Big Co is to get the talent who have spent the years in building the new tech.
Poaching, acquihirng or acquisitions and the myriad modern forms we are seeing today have been the tools and will not change.
Owners and beneficiaries of the capital do not change, but that is an artifact of our economic system and is much larger a socio-economic discussion beyond the scope of innovation and research
Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind of efficiency.
Memristors have been tried for literally decades.
If the posters other guesses pay out the same rate, this will likely play out never.
There was a bit of noise regarding spiking neural networks a few years ago but now I am not seeing it so often anymore.
Other technologies tried for decades before becoming huge: Neural-network AI; Electric cars; mRNA vaccines; Solar photovoltaics; LED lighting
Ho boy, should we start listing the 10x number of things that went in the wastebasket too?
If I only have to try 11 things for one of them to be LED lights or electric cars, I'd better get trying. Sure, I might have to empty a wastebasket at some point, but I'll just pay someone for that.
This fundamentally at odds with picking one tech and saying ‘this is the winner’ eh? Which is what the prior comment was about.
Which prior comment? Top level offers two options plus an 'etc.', second comment adds another and says 'could be', and then first reply to you offers other tech that took decades of R&D to suggest we can't rule out memristors. I don't see what you're referring to.
Uh huh, sure.
> Sparse models for example show a lot of promise and more closely track real biological systems.
I think sparsity is a consequence of some other fundamental properties of brain function that we've yet to understand. Just sparsifying the models we've got is not going to lead anywhere, IMO. (For example it's estimated that current AI models are already within 1%-10% of a human brain in terms of "number of parameters" (https://www.beren.io/2022-08-06-The-scale-of-the-brain-vs-ma...).)
Not a fan of fear based marketing: "The whole world is too big and expensive for you to participate in, so use our service instead"
I'd rather approach these things from the PoV of: "We use distillation to solve your problems today"
The last sentence kind of says it all: "If you have 30k+/mo in model spend, we'd love to chat."
Maybe I'm totally misreading this, but it seems like the post contradicts itself. At the beginning of the third paragraph:
> Impressively, open source models have been able to quickly catch up to big labs.
And then the beginning of the fourth:
> Open-source has been lagging behind proprietary models for years, but lately this gap has been widening.
Followed by a picture that is more or less inscrutable.
Hey, I'm the author of the post.
The image has been fixed, and the point I'm making is that proprietary models are almost always ahead, and this gap is widening. OS models that are nearly at the same quality are usually distilled versions of proprietary models, or somehow get training data from them. Sometimes, after massive, expensive training runs models are open sourced anyway, and at some point that becomes unsustainable.
The difference between a top model and a model with a similar ELO might seem small, but the value of even a marginal increase in intelligence is extremely high--for example I only use the best coding model for coding, whatever the cost.
There's also lots of evidence that large labs are only getting started. In the past year, they have secured massive amounts of compute, which is still not utilized well. I expect lots of big training runs in the future, which will shift the gap further between OS and proprietary models.
The major problem for these companies is they spend hundreds of millions of dollars training a model, and then someone comes in the next day and distills something almost as good for far less money (still a VERY large sum of money.)
I don't know how this will be resolved long term.
Thanks for clarifying!
I think I'm getting it now: OS models are getting closer, but only via distillation. Not by training a new frontier model which is out of reach for economic reasons.
Note that distilling a general model is several orders of magnitude more expensive than distilling a task-specific model, which is what I'm trying to promote here. Smart general models make distilling great task specific models with no expert labelers way easier.
> Followed by a picture that is more or less inscrutable.
Yeah. Just to make it explicit - that chart has Deepseek r1 at ... presumably an elo of 1418 and Gemini Pro at 1463. That is comparable to the gap between Magnus Carlsen and Fabiano Caruana [0]. I don't think it is reasonable to complain about that sort of performance gap in practice - it is a capable model. Looking at the spread of scores I don't immediately see why someone even needs to use something in the Top 10, presumably anything above 1363 would be good enough for business, research and personal use.
None of these models have even been around that long, Deepseek was only released in January. The rate of change is massive, I expect to have access to an open source model that is better than anything on this leaderboard next year some time.
[0] https://2700chess.com/
We haven't seen a proper npu and we are in the launch of the first consumer grade unified architectures by Nvidia and AMD. The battle of homebrew AI hasn't even started yet.
Hell, we haven’t even seen actual AI yet. This is all just brute-forcing likely patterns of tokens based on a corpus of existing material, not anything brand new or particularly novel. Who would’ve guessed that giving CompSci and Mathematics researchers billions of dollars in funding and millions of GPUs in parallel without the usual constraints of government research would produce the most expensive brute-force algorithms in human history?
I still believe this is going to be an embarrassing chapter of the history of AI when we actually do create it. “Humans - with the sort of hubris only a neoliberal post-war boom period could produce - honestly thought their first serious development in computing (silicon-based mircoprocessors) would lead to Artificial General Intelligence and usher in a utopia of the masses. Instead they squandered their limited resources on a Fool’s Errand, ignoring more important crises that would have far greater impacts on their immediate prosperity in the naive belief they could create a Digital God from Silicon and Electricity alone.”
Even if we somehow create AGI it will only be used to saddle the masses, not free them. There are a lot of people with money who yearn for absolute power over everything. Replacing your capacity to think with a subscription service makes you a serf. Like Spotify, Netflix, Amazon and so on the last 20 years is full of brand new subscriptions that replace the whole concept of ownership so thoroughly that it gets erased. It pumps up the company’s valuation to do that.
It's a necessary evolution step. Did you know our own ancestors had tails and grills. Do you feel ashamed?
No, but maybe pandas should. You know, evolution is mostly dead ends, literally.
Grills don't sound so bad!
The world is definitely a better place with grills.
Yeh. We're still barely beyond the first few pixels that make up the bottom tail of the S-curve for autonomous type AI everyone imagines
Energy models and other substrates are going to be key, and it has nothing to do with text at all as human intelligence existed before language. It's Newspeak to run a chat bot on what is obviously a computer and call it an intelligence like a human. 1984 like dystopia crap.
Could you please stop creating accounts for every few comments you post? We ban accounts that do that. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.
You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
The blog kept redirecting to the home page after a second, so here's an archive: https://archive.is/SE78v
If I'm understanding this correctly, we should see some great coding LLMs. Idk, could be as limited as a single stack eg laravel/nextjs ecosystem.
Have won what? The privilege of burning billions of dollars and not being profitable?
Until AGI is achieved no one's really won anything.
Deepseek main run costed $6M. qwen3-30b-a3b probably would cost few $100Ks, which is ranked 13th.
GPU cost of the final model training isn't the biggest chunk of the cost and you can probably replicate results of models like Llama 3 very cheaply. It's the cost of experiments, researchers, data collection which brings overall cost 1 or 2 order of magnitude higher.
What's your source for any of that? I think the $6 million thing was identified as a lie they felt was necessary because of GPU export laws.
It wasn't a lie, it was a misrepresentation of the total cost. It's not hard to calculate the cost of the training though. It takes 6 * active parameters * tokens flops[1]. To get number of seconds you can divide by Flops/s * MFU, where MFU is around 45% for H100 for large enough models[2].
[1]: https://arxiv.org/abs/2001.08361
[2]: https://github.com/facebookresearch/lingua
That paper's 5 years old at this point, dating back to when Amodei was still an OpenAI employee. Has any newer work superseded it, or are those assumptions still considered solid?
Those assumptions are still the same. Although now context length has increased more so the n^2 part is non negligible. See the repo for correct flop calculation[1]
[1]: https://github.com/facebookresearch/lingua/blob/437d680e5218...
Tropical Distillation?
50m per day is insane! Any link supporting that?
They just took their estimated spend per training run, doubled it, and divided by the number of models they release a year. Roughly.
Perhaps one of these days a random compsci undergrad will come up a DeepSeek-calibre optimization.
Just imagine his or her 'ChatGPT with 10,000x fewer propagations' Reddit post appearing on a Monday...
...and $3 trillion of Nvidia stock going down the drain by Friday.
DeepSeek came up with several significant optimizations, not just one. And master's students do contribute to leading edge research all the time.
There have really been many significant innovations in hardware, model architecture, and software, allowing companies to keep up with soaring demand and expectations.
But that's always how it's been in high technology. You only really hear about the biggest shifts, but the optimizations are continuous.
True, but I chose the words 'ChatGPT' and 'optimization' for brevity. There are many more eyes on machine learning since ChatGPT came along. There could be simpler techniques yet to discover. What boggles the mind is the $4 trillion parked in Nvidia stock, and wasted if more efficient code lessens the need for expensive GPUs.
One can only hope. Maybe then they’ll sell us GPUs with 2025 quantity memory instead of 2015.