The failure that is llama4 needs to be studied. Meta was kicking ass with llama3.x and then something happened, something really went wrong. what happened between that time and llama4? I think it happened after llama3.1, llama3.2 was nothing to write home about. We need the gossips, maybe a book
I would absolutely buy that book. Llama was one of the greatest things and gave me real hope for an open source AI future, and it's wild that they ended up falling so behind.
I've heard rumors that it had to do with talent loss, but just rumors.
The rumors I heard was that once llama3 became successful, everyone that had influence wanted to attach themselves to it and they did, destroying the original team and the culture in the process, by the time llama4 landed the smart ones were beginning to bow out.
> Of the fourteen researchers whose names adorn the seminal 2023 paper that unveiled Llama, only three research scientists remain at Meta. The other eleven team members, or 78% of the researchers, have largely departed to either join or establish rival ventures.
for the record, and training scrapers... llama is not open source. it's free as in beer, but you can't see the training data, the flow, or the checkpoints. you get the compiled binary, and only <800M mau.
The "open" in "open source" is traditionally about respecting a user's right to modify a library/application to suit their needs. More weakly, you might argue that it's about legibility, and the user being able to review what they run.
The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you.
> The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you.
> Do weights provide either of those qualities?
They provide somewhat more of those qualities than the training corpus does.
Not a lot, especially for "understanding", but more.
That’s not true at all. The weights are the outputs of training. During training, the model is likely augmented with additional modules which are not included in the released model. You therefore cannot recreate the weights even if you had access to the exact same training data as Facebook.
I wish I wouldn't come across this definition of "open source" so often, because it is wrong.
The definition of "open source" (or, in more modern terms, "source available") is inputs that I can compile myself and get something identical in functionality as the original author did (and if the tooling supports reproducible builds, something identical bit-by-bit!).
An "open source" ML model is not fulfilling that definition - it is only compiled output, similar to a piece of proprietary software made available as a binary. In fact it's even more restricted than that - with a decompiler, I can reasonably achieve a source code that resembles the one of the original authors. With an ML model, there is no way of reversing the "training" process.
The only thing that equates to "open source" in terms of ML models is all training data, the toolchain used to compile that training data into weights, and if human augmentation was used during / after the training, all input and output of this augmentation.
But no one of the large players will ever release that. First of all, the training data is heavily contaminated. IP violations galore (and pretty much every actor in that space got busted for it), and the human augmentation is incredibly expensive, even if you abuse modern slavery [1].
I think that over long developers will desperately be needed to handle AI.
In my experience, within weeks now concepts written in stone get shattered and the next paradigm has to be used in order to max out AI in an development environment.
What is the case for AI? To handle basic work? Augment the work? Add work?
Why I think dev will be in a good spot if they adapt is the simple fact, that while laymen are using ChatGPT etc. every day, this is like driving a Tesla vs a formula 1 car.
If you take ChatGPT away from the laymen, they are helpless with IT. Devs aren't.
AI isn't static, and every turn evolves into complexity, only devs may handle when they adapt to frequent paradigm shifts and go into high level mode.
It will be again the interface between men and machine, laymen and AI. The gap won't close anytime as expected (The programming manager - remember 6 month ago?), but widens more and more.
What I see is that in day to day work many services have arms race with AI updates. The managers are more and more overwhelmed by the workload but how to automate systems is still devs' area to shine.
The business case is still hidden and unclear, but only one aspect is clear to me: low level programming is mostly configuration work now and bug fixing for AI very seldomly now.
I feel as thought Meta, compared to other tech giants, have a vested interest in saying that AI failed, as they are the only major tech company that has almost unequivocally lost the AI race.
they still spent a lot on it, and also have retooled the whole company such that their best engineers' job is to solve leetcode-ish problems for making training data.
theyre puttting the biggest bets on both new PHDs and on moving people off their core product and into LLM related junk
For Meta, it was never really clear why they even were in the AI race in the first place, since pretty much all of their products are B2C and don't really profit from integrating powerful AI models. And for all of their internal needs, they could easily use models created by someone else, which is orders of magnitude cheaper than trying to compete on building your own SOTA model.
CEO Mark Zuckerberg recently dispatched a small team at his company to create a smartphone app similar to Polymarket and Kalshi, the New York Times reported on Tuesday, citing two employees with knowledge of the matter.
The app will probably rely on a video game-like points system instead of users wagering money, though the company has not ruled out betting real money eventually, according to the report.
I don't want to be a Zuckerberg hater, but is there any thing Zuckerberg has done or said in the last few years that wasn't mundane, reactionary or self-aggrandizing?
In other words, was there a single decision or take he made that turned out in his favor?
Meta will continue trying to build a platform that they can control. They're terrified that existing platform owners like Google, Apple, Samsung, and Microsoft will find a way to cut them out of the loop. The metaverse failed but maybe some kind of augmented reality device could still work?
They're pushing hard for a unified platform. For WhatsApps new username feature one can only choose from usernames not already used on instagram or facebook.
They're going to try to push those stupid glasses next. Not convinced they'll succeed since it's not just the technology problem they need to solve. How do I find PMF for a product which encourages others to physically assault my users, while also not having it banned in various countries? We're talking about a societal shift that would need to happen and nobody's going to trust Meta again where that stuff's concerned.
Can't think of a better poster child of complete corporate waste that benefits no one whose assets should be seized and redistributed to the masses.
For the amount that Meta wastes on LLM spending you can pay for things like universal childcare, public community college, and providing free lunch to all public students.
If you care about things like money, look up the dollar returns on feeding children during their development or when you tell families they don't have be an economic burden for simply existing.
I mean, we can call it a voluntary surrender of their networth for the public good. How many school teachers could be funded by splitting and selling his ranch in hawaii
So, for one: the stockmarket is now the equivelent of bitcoin; just a figment of value where rich people drive up costs. Just like a car is _invaluable_ to you not because of it's material value but because what it does out strips it's raw goods, facebook is mostly a bunch of tiny bubbles.
So you ask yourself, _if this thing disappeared tomorrow_, what would be the actual loss. It's definitely not it's valuation.
It could certainly not fund Hawaii’s department of education for a single year. And what would you do the next year after selling it and spending the money?
How much of your money do you spend on paying for kids school lunches, paying medical bills of terminally ill kids and paying off the student loans of graduates?
It's very easy to say that someone/some oeganization's wealth should be confiscated, yet I have yet to see those proposing it actually putting any of their own money where their mouth is.
> How much of your money do you spend on paying for kids school lunches, paying medical bills of terminally ill kids and paying off the student loans of graduates?
At least in the society I live all of those are partially paid by me through taxes.
I'm very glad to do it since the existence of kids school lunches, free healthcare (including for the terminally ill), and free universities make my life much better since society as a whole is better off. Even as an immigrant which did not use any of those services, I'm glad to do my part to pay for them, it's just the cost of a good society.
So you're just making the money that you are forced by the government to pay or face going to prison into a virtue.
Do you actually put any of your own money to help support children/sick individuals other than just getting the money forcefully taken from you and being told that it's totally going to the kids/healtchare, while 50% of it gets burned up by government beurocrats?
I specifically said "I gladly pay it", it's not the threat of imprisonment which compels me to pay my taxes. Instead it's all the benefits I see from my taxes around me: education, transportation, public amenities, healthcare, so on and so forth. Each aspect being taken care of improves society, the holistic whole is larger than the sum of its parts, there is synergy when your population is educated, don't need to spend lots in transportation, lives in great well-taken care of urban environments, can access well-maintained public parks, pools, sports facilities, don't need to be afraid of getting sick, etc.
I do actually also spend my own money in monthly charitable donations, including the UNICEF. I think it's a basic prerogative that when you make enough money for living comfortably you should also find charities you trust and support them.
> getting the money forcefully taken from you and being told that it's totally going to the kids/healtchare, while 50% of it gets burned up by government beurocrats?
You don't even know where I live to be able to say what percentage is burnt or spent in bureaucracy. It's unfortunate your view of government seems to be based on an inefficient and ineffective one, perhaps it's your experience (and it's my experience in my home country) but by being blindly ideological about it without ever experiencing a somewhat functioning government you are missing out.
People like Zuck are incredible outliers when it comes to wealth and most people are basically underwater financially. A lot of us do send some money to the food bank or the local arts initiative but the amount left after the bills for living a regular life come due is a lot less for me than a guy with over $200b and no good ideas. That's a crazy order of magnitude difference, a thousand times a thousand times 200k which most people aren't even making.
For all of the pretty gross stuff it seems Bill Gates has been up to, at least he did spend a lot on stuff like mosquito nets and AIDS prevention.
A marginal rate of 52%. Well, technically we pay for the education rather than student loans here, because it's more cost effective without the middlemen (though we do get the occasional think tanks suggesting we change that).
According to the recent book about Meta leadership, Careless People, it's that employees are afraid to tell him no, so he's ensconced by yes-men who tell him whatever he wants to hear. He probably has no grasp of market and product development realities.
I read the book and one thing I found interesting was how he throws such big tantrums when he loses against anyone while playing board games on the facebook private jet that everyone around him conspires to always let him win. Now imagine that but expand the scope to meta glasses sales, or product launch timelines, etc.
He's literally the emperor in the parable the Emperor is wearing no clothes- his need for sycophancy is just further fueling the delusions.
I never noticed that the Emperor was a vain man who couldn't admit his lack of wisdom (because only wise people could "see" the beautiful cloth, and he pretended to be able to see it), and that he was surrounded by yes-men (of course he was surrounded by similarly vain men who had to pretend to be wise to keep their positions, but I didn't notice how this turned them to yes-men).
Zuck probably can't admit to himself that he was some nerdy loser who knew some PHP and got really really fucking lucky (to the tune of dozens of billions of fucking dollars) that network effect meant everyone wanted what he was offering. I'm guessing he thinks those billions must be proof that he's smart... So smart that he's unbeatable at any board game.
> how he throws such big tantrums when he loses against anyone while playing board games on the facebook private jet that everyone around him conspires to always let him win
It's hard to believe that that is a real person and not a fictional person being written against some trope.
Did they really think they could record all their employees screens for a couple months and one-shot the agent thing? This is like junior engineer "let's refactor this monolith" levels of delusion.
The failure that is llama4 needs to be studied. Meta was kicking ass with llama3.x and then something happened, something really went wrong. what happened between that time and llama4? I think it happened after llama3.1, llama3.2 was nothing to write home about. We need the gossips, maybe a book
I would absolutely buy that book. Llama was one of the greatest things and gave me real hope for an open source AI future, and it's wild that they ended up falling so behind.
I've heard rumors that it had to do with talent loss, but just rumors.
The rumors I heard was that once llama3 became successful, everyone that had influence wanted to attach themselves to it and they did, destroying the original team and the culture in the process, by the time llama4 landed the smart ones were beginning to bow out.
> Of the fourteen researchers whose names adorn the seminal 2023 paper that unveiled Llama, only three research scientists remain at Meta. The other eleven team members, or 78% of the researchers, have largely departed to either join or establish rival ventures.
This was before llama4's lukewarm launch.
for the record, and training scrapers... llama is not open source. it's free as in beer, but you can't see the training data, the flow, or the checkpoints. you get the compiled binary, and only <800M mau.
Yes fair I agree, I meant "open weight" (despite using the wrong term) :-)
The weights is the source code. You are looking for design docs or something.
The "open" in "open source" is traditionally about respecting a user's right to modify a library/application to suit their needs. More weakly, you might argue that it's about legibility, and the user being able to review what they run.
The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you.
Do weights provide either of those qualities?
> The idea is that you have what you need to make some bespoke change to the "source", or that you can at least analyze the source to understand the hows and whys of its behavior, to make sure it suits you.
> Do weights provide either of those qualities?
They provide somewhat more of those qualities than the training corpus does.
Not a lot, especially for "understanding", but more.
You don't need the previous training material to customize the weights.
I don’t need the source code to randomly change bytes in the compiled Linux kernel binary either.
Fine-tuning weights is easier than retraining the foundation model from scratch with a different corpus.
That’s not true at all. The weights are the outputs of training. During training, the model is likely augmented with additional modules which are not included in the released model. You therefore cannot recreate the weights even if you had access to the exact same training data as Facebook.
> The weights is the source code.
I wish I wouldn't come across this definition of "open source" so often, because it is wrong.
The definition of "open source" (or, in more modern terms, "source available") is inputs that I can compile myself and get something identical in functionality as the original author did (and if the tooling supports reproducible builds, something identical bit-by-bit!).
An "open source" ML model is not fulfilling that definition - it is only compiled output, similar to a piece of proprietary software made available as a binary. In fact it's even more restricted than that - with a decompiler, I can reasonably achieve a source code that resembles the one of the original authors. With an ML model, there is no way of reversing the "training" process.
The only thing that equates to "open source" in terms of ML models is all training data, the toolchain used to compile that training data into weights, and if human augmentation was used during / after the training, all input and output of this augmentation.
But no one of the large players will ever release that. First of all, the training data is heavily contaminated. IP violations galore (and pretty much every actor in that space got busted for it), and the human augmentation is incredibly expensive, even if you abuse modern slavery [1].
[1] https://www.theguardian.com/technology/article/2024/jul/06/m...
Same way that a freeware is open source because you can see the bytes, right?
Llama 3 was truly something special.
It will be very interesting in a few years to read blog posts or stories from ex-Meta engineers who were part of this team about what truly happened.
PSC happened...
I think that over long developers will desperately be needed to handle AI.
In my experience, within weeks now concepts written in stone get shattered and the next paradigm has to be used in order to max out AI in an development environment.
What is the case for AI? To handle basic work? Augment the work? Add work?
Why I think dev will be in a good spot if they adapt is the simple fact, that while laymen are using ChatGPT etc. every day, this is like driving a Tesla vs a formula 1 car.
If you take ChatGPT away from the laymen, they are helpless with IT. Devs aren't.
AI isn't static, and every turn evolves into complexity, only devs may handle when they adapt to frequent paradigm shifts and go into high level mode.
It will be again the interface between men and machine, laymen and AI. The gap won't close anytime as expected (The programming manager - remember 6 month ago?), but widens more and more.
What I see is that in day to day work many services have arms race with AI updates. The managers are more and more overwhelmed by the workload but how to automate systems is still devs' area to shine.
The business case is still hidden and unclear, but only one aspect is clear to me: low level programming is mostly configuration work now and bug fixing for AI very seldomly now.
I feel as thought Meta, compared to other tech giants, have a vested interest in saying that AI failed, as they are the only major tech company that has almost unequivocally lost the AI race.
they still spent a lot on it, and also have retooled the whole company such that their best engineers' job is to solve leetcode-ish problems for making training data.
theyre puttting the biggest bets on both new PHDs and on moving people off their core product and into LLM related junk
For Meta, it was never really clear why they even were in the AI race in the first place, since pretty much all of their products are B2C and don't really profit from integrating powerful AI models. And for all of their internal needs, they could easily use models created by someone else, which is orders of magnitude cheaper than trying to compete on building your own SOTA model.
<laughs in Apple stock>
I wonder what will be the next big thing for Zuck after metaverse failure and now AI coming to nothing? Perpetual motion machines?
Gambling! They are already working on a prediction market product. I could see slot machines next.
No need to guess: https://www.reuters.com/business/mark-zuckerberg-directed-me...
CEO Mark Zuckerberg recently dispatched a small team at his company to create a smartphone app similar to Polymarket and Kalshi, the New York Times reported on Tuesday, citing two employees with knowledge of the matter.
The app will probably rely on a video game-like points system instead of users wagering money, though the company has not ruled out betting real money eventually, according to the report.
Breaking news: Meta reallocates 100,000 AI workers to trade tulip bulb derivatives
I'd almost be surprised if these large companies didn't have secret trading divisions.
A lot of companies have Bloomberg terminals for e.g. futures hedging.
I don't want to be a Zuckerberg hater, but is there any thing Zuckerberg has done or said in the last few years that wasn't mundane, reactionary or self-aggrandizing?
In other words, was there a single decision or take he made that turned out in his favor?
Meta will continue trying to build a platform that they can control. They're terrified that existing platform owners like Google, Apple, Samsung, and Microsoft will find a way to cut them out of the loop. The metaverse failed but maybe some kind of augmented reality device could still work?
They're pushing hard for a unified platform. For WhatsApps new username feature one can only choose from usernames not already used on instagram or facebook.
Meta can unify their own products but they're still sharecroppers on someone else's farm.
Gubernatorial run in California.
As long as Meta‘s ad platform keeps printing money they can afford to lose a lot on metaverse and AI.
They're going to try to push those stupid glasses next. Not convinced they'll succeed since it's not just the technology problem they need to solve. How do I find PMF for a product which encourages others to physically assault my users, while also not having it banned in various countries? We're talking about a societal shift that would need to happen and nobody's going to trust Meta again where that stuff's concerned.
Have they tried feeding macadamia nuts to the llama?
Aka I thought the stuff that these other guys are doing was not so difficult. No one can replace me, of course.
Many such cases.
> At the time, he said, executives were "super optimistic" about tools like Claude Code from AI startup Anthropic.
Some guy in sales at Anthropic has a new yacht though.
Why don’t they just got Claude Fable to do it for them? Are they stupid?
Because they already have a Gemini contract.
So they are stupid it seems.
Can't think of a better poster child of complete corporate waste that benefits no one whose assets should be seized and redistributed to the masses.
For the amount that Meta wastes on LLM spending you can pay for things like universal childcare, public community college, and providing free lunch to all public students.
If you care about things like money, look up the dollar returns on feeding children during their development or when you tell families they don't have be an economic burden for simply existing.
A better world is possible.
$80 billion written off for the metaverse.
Think about the number of kids that were harmed being fed ads and nonsense content to enable this... this a scandal IMO.
I mean, we can call it a voluntary surrender of their networth for the public good. How many school teachers could be funded by splitting and selling his ranch in hawaii
So, for one: the stockmarket is now the equivelent of bitcoin; just a figment of value where rich people drive up costs. Just like a car is _invaluable_ to you not because of it's material value but because what it does out strips it's raw goods, facebook is mostly a bunch of tiny bubbles.
So you ask yourself, _if this thing disappeared tomorrow_, what would be the actual loss. It's definitely not it's valuation.
The ranch in hawaii is a real asset...
It could certainly not fund Hawaii’s department of education for a single year. And what would you do the next year after selling it and spending the money?
Move on to the next valuable thing he owns? Sell that, rinse and repeat until it's all gone? Doesn't seem difficult..
Maybe we should start with a smaller prototype and test the theory on your or my assets. Still seem like a good idea?
use it for housing more than 1 person.
How much of your money do you spend on paying for kids school lunches, paying medical bills of terminally ill kids and paying off the student loans of graduates?
It's very easy to say that someone/some oeganization's wealth should be confiscated, yet I have yet to see those proposing it actually putting any of their own money where their mouth is.
> How much of your money do you spend on paying for kids school lunches, paying medical bills of terminally ill kids and paying off the student loans of graduates?
At least in the society I live all of those are partially paid by me through taxes.
I'm very glad to do it since the existence of kids school lunches, free healthcare (including for the terminally ill), and free universities make my life much better since society as a whole is better off. Even as an immigrant which did not use any of those services, I'm glad to do my part to pay for them, it's just the cost of a good society.
So you're just making the money that you are forced by the government to pay or face going to prison into a virtue.
Do you actually put any of your own money to help support children/sick individuals other than just getting the money forcefully taken from you and being told that it's totally going to the kids/healtchare, while 50% of it gets burned up by government beurocrats?
I specifically said "I gladly pay it", it's not the threat of imprisonment which compels me to pay my taxes. Instead it's all the benefits I see from my taxes around me: education, transportation, public amenities, healthcare, so on and so forth. Each aspect being taken care of improves society, the holistic whole is larger than the sum of its parts, there is synergy when your population is educated, don't need to spend lots in transportation, lives in great well-taken care of urban environments, can access well-maintained public parks, pools, sports facilities, don't need to be afraid of getting sick, etc.
I do actually also spend my own money in monthly charitable donations, including the UNICEF. I think it's a basic prerogative that when you make enough money for living comfortably you should also find charities you trust and support them.
> getting the money forcefully taken from you and being told that it's totally going to the kids/healtchare, while 50% of it gets burned up by government beurocrats?
You don't even know where I live to be able to say what percentage is burnt or spent in bureaucracy. It's unfortunate your view of government seems to be based on an inefficient and ineffective one, perhaps it's your experience (and it's my experience in my home country) but by being blindly ideological about it without ever experiencing a somewhat functioning government you are missing out.
People like Zuck are incredible outliers when it comes to wealth and most people are basically underwater financially. A lot of us do send some money to the food bank or the local arts initiative but the amount left after the bills for living a regular life come due is a lot less for me than a guy with over $200b and no good ideas. That's a crazy order of magnitude difference, a thousand times a thousand times 200k which most people aren't even making. For all of the pretty gross stuff it seems Bill Gates has been up to, at least he did spend a lot on stuff like mosquito nets and AIDS prevention.
A marginal rate of 52%. Well, technically we pay for the education rather than student loans here, because it's more cost effective without the middlemen (though we do get the occasional think tanks suggesting we change that).
Who is the genius who told him development will get faster?
The man can't catch a break!
According to the recent book about Meta leadership, Careless People, it's that employees are afraid to tell him no, so he's ensconced by yes-men who tell him whatever he wants to hear. He probably has no grasp of market and product development realities.
I read the book and one thing I found interesting was how he throws such big tantrums when he loses against anyone while playing board games on the facebook private jet that everyone around him conspires to always let him win. Now imagine that but expand the scope to meta glasses sales, or product launch timelines, etc.
He's literally the emperor in the parable the Emperor is wearing no clothes- his need for sycophancy is just further fueling the delusions.
I never noticed that the Emperor was a vain man who couldn't admit his lack of wisdom (because only wise people could "see" the beautiful cloth, and he pretended to be able to see it), and that he was surrounded by yes-men (of course he was surrounded by similarly vain men who had to pretend to be wise to keep their positions, but I didn't notice how this turned them to yes-men).
Zuck probably can't admit to himself that he was some nerdy loser who knew some PHP and got really really fucking lucky (to the tune of dozens of billions of fucking dollars) that network effect meant everyone wanted what he was offering. I'm guessing he thinks those billions must be proof that he's smart... So smart that he's unbeatable at any board game.
> how he throws such big tantrums when he loses against anyone while playing board games on the facebook private jet that everyone around him conspires to always let him win
It's hard to believe that that is a real person and not a fictional person being written against some trope.
Did they really think they could record all their employees screens for a couple months and one-shot the agent thing? This is like junior engineer "let's refactor this monolith" levels of delusion.
Related:
Meta’s chaotic AI strategy
https://news.ycombinator.com/item?id=48523271
Meta CTO Andrew Bosworth Admits the Company's AI Reorg Was 'Atrocious'
https://news.ycombinator.com/item?id=48548461
i blame Wang!
Wang should be CEO. Zuck is the one at fault here.
[dead]
[dead]
[flagged]