We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.
LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your LLM-based “AI” gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence. What bothers me about this entire situation is that presumably the folks that relied on the LLM-based “AI” to generate these vulnerabilities knew (or should have known) enough about their tool to know this would happen, but did not.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.
In my experience, product. There’s a pervasive view that product is easy, that anyone working in it is a dunderhead who gets in the way of engineers doing the really worthwhile work. In reality, it would just mean they rewrite the software five times and have zero customers.
Oh, design too. I’ve met many engineers who simply don’t see any value in design.
This. I'm a former teacher. I left to get a PhD in educational psychology, and now I'm a data scientist for a large school division. The number of ed tech companies I interact with who clearly have little grasp of research-based best practices in education is staggering. But they can all build slick-looking websites and dashboards.
I don't think it is much of a problem that dabblers get into Internet arguments about things they are not experts in. After all the easiest way to get a correct answer/learn something is to say something wrong.
Of course this should be differentiated from a professional setting.
And under no circumstances should we put that stuff into the knowledge bases we base our "AI" on without making sure it takes into account the context.
> It's just that now it looks a lot more polished.
Yes, and that is the problem. It used to be if a product looked polished it was fairly polished engineering wise as well if we compare to todays AI slop. You can see that on steam, before AI slop games a game that looked polished mostly worked. But today you can get a game that looks like they put in a lot of effort, but its all AI slop and everything is flaky and broken, I've had way more such experiences the last few months than before that.
A human coder that was capable of coding a complex game typically also was good enough and tested things to make most things work. There were bugs etc, but at least buttons did things.
Engineers cosplay as physicists and mathematicians every day. What's your point? Think of it pragmatically. If they can do the job they can do the role.
Can they do the job? Because right now, government decisions are based on AI generated code, which was verified by nobody who can do that. So the cost of an unsatisfactory answer is quite high.
To be fair, "people with no skills inflating their own value" is what LinkedIn has always been like. But I guess LLMs are uniquely well positioned for that task.
LLMs have brought it to a new level. I have multiple people in my feed posting several page dissertations on a daily basis. More often than not, these posts are about the benefits of AI...
> People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.
I like the symmetry of people being unable to detect "skill" in people who are great at manipulating language and in LLMs who are also great at manipulating language.
> A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.
The sort of person that's going to offload their thinking to AI is the exact sort of person that is not going to verify anything because they've already offloaded their thinking to AI.
For a long time I was anti-licensure in tech; now with the bar being lowered to next to nothing, it seems as if licensure is more important than ever — not to protect this trade (though it will do that, and that is a benefit), but because the sheer amount of irresponsibility in the usage of LLMs and “AI” in general begs for licensure and adoption of a regulatory body for software in general.
This is unfortunately a feeling I share. It wasn't until LLMs have become nearly ubiquitous at this point, and there has been zero realistic technological response to the dangers they present. Not to mention I suspect there may be some psychological element to being exposed to interactions with AI models and their nonsense for hours a day. Not all of it is nonsense....but you won't ever know for sure.
Unfortunately people sometimes get defensive against this take. But I think treating the LLM as you described can make you a better LLM user and help get better output. It helps understand the failure modes better, and moderate one’s reliance on them. Just like how we should do for every tool we work with.
Yes, I've found that reminding yourself of how they actually work helps keep you on guard against LLM-patterned mistakes. Especially things like carefully considering what parts of the current task likely fall outside the distribution of corpus + RL data (as much as that can be guessed).
> ...Unfortunately people sometimes get defensive against this take. But I think treating the LLM as you described can make you a better LLM user and help get better output. It helps understand the failure modes better, and moderate one’s reliance on them. Just like how we should do for every tool we work with.
B...b...but the Anthropic trainer said we'd get the best results if we don't think of it as a tool, but instead give it a name and think of it as our brilliant coworker!
Why should I trust you, internet rando over a stormtrooper-level salesman? /s
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
Brandolini's principle in action. It takes 10 times more energy to refute BS than to generate it. A related analogy to computing: it is easy to generate propositions, but hard to test if a given proposition is satisfiable or not, which curiously ties to P vs NP.
> LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence.
I agree that humans must verify LLM-produced facts, but strongly disagree with these kinds of "stochastic parrot therefore dumb" arguments.
Yes, an LLM is a "stochastic parrot". No, that doesn't imply that it is dumb. Enough to look at how Terence Tao asks ChatGPT to help him understand a solution that nobody had ever discussed before [1], or how a random guy asks ChatGPT in a handful of words to disprove a 30-year-old conjecture, with zero technical input [2].
If your parrot in a birdcage with internet access can finish the sentence, "The counterexample to the Dinitz–Garg–Goemans conjecture is...", then it's a pretty smart parrot, by all reasonable definitions of "smart". Just because someone bottled up the formula into matrix multiplications and added some random sampling to the outcome, that doesn't take away from the fact that the parrot said provably correct statements that the biggest experts in the field couldn't imagine.
And no, I'm not implying that the LLMs are correct all the time, or that their intelligence and reasoning works in any way like ours.
The best way to describe the LLM intelligence is "an expert system that works the way people thought expert systems would work". You can encode a massive amount of skills into an LLM, and then the LLM uses those to navigate problems.
But the LLM is still dumb where those skills doesn't have good coverage, since unlike the expert systems it maps fuzzily to its skills, and they are tuned to produce results over rejecting the request when its unclear if coverage is good. As long as that is true you have to treat them as dumb even if they sometimes produce brilliant results.
Turing-completeness is a necessary pre-requisite for being able to fulfill the requirements of a Turing machine, nothing more. In the same way that cell division is a necessary condition for life, but cell division does not mean a given life form itself is sentient.
Intelligent life-forms can generate probabilistic outputs based on inputs, but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.
OK. A more pointed question. What do you know about intelligence that allows you to exclude LLMs with CoT from the category of intelligent systems with certainty?
Right, it follows naturally that one can never meaningfully discuss that which can only be settled by proving negatives. Hence the total lack of debate on religious topics, as everyone just gave up 1500 years ago and decided that God indeed may or may not exist and that's that.
Why do you think that intelligence of a Turing-complete system trained on human data can be likened to a Russell's teapot?
At least, don't you think that the recent mathematical results of LLMs are a bit like a glimpse of something teapot-shaped in the orbit? (which makes it not a Russell's teapot, which. by definition, can't be observed).
To me, it's an expected progression of ANNs' approximation of human cognitive processes. The universal approximation theorem guaranties the existence of such ANNs barring the super-Turing or quantum superiority of the brain.
You’re right to point that out. The load-bearing seam of your logical retort is the hidden assumption that the teapot is in outer space, not at peace on a kitchen stovetop here on earth. I am sorry I had not considered this fact. \s
> but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.
???
Of course it is. The brain is mechanically not capable of doing anything other than that.
Do you believe the brain is something other than a bundle of probabilistic physical interactions? Or are brains not the source of what we call intelligence?
Looks interesting (added to my list, thank you for the recommendation), but not actually relevant to the topic at hand.
We know that the brain is a probabilistic input → output machine because the universe is a probabilistic input → output machine. The brain is made of universe. There are deterministic relationships (which at high sensitivity or complexity become easier to describe as probabilistic), and quantum relationships. That's it. The brain, like every other thing comprised of "universe" is comprised of those two types of relationships.
If Romain's book provides evidence of relationships in the brain that are neither quantum (therefore random) NOR classical (therefore deterministic), then 1) he would have already won at least one Nobel prize, and 2) anyone in this thread would be able to at least gesture toward what relationship that is.
Equating "classical" with "deterministic" is however pushing it a bit too far, when no one and nothing can ever do even a very small fraction of said determination...
Quantum is not bullshit and yes, classical is deterministic.
That's why the brain cannot possibly be anything other than an input → output machine, which is functionally deterministic (with maybe some fully random components), but is easiest to describe as probabilistic.
In the same way that LLMs are functionally deterministic, but easiest to describe as probabilistic.
Nothing in the universe is anything other than a chain of reactions, amigo.
The brain is an object in the universe.
The universe has quantum behaviors (fully random, not a source of intelligence) and it has deterministic behaviors (fully non-random). Many of those deterministic behaviors are so complex that they're easier to analyze and describe as probabilistic, which is where most brain input → output relationships land.
Please point to any evidence whatsoever that the brain has some third type of interaction going on that has never been observed anywhere in the entire universe, then we can have a discussion about it.
Probabilistic and deterministic are not opposites.
The brain is deterministic at the level of specific interactions, which process inputs in a highly chaotic (but still deterministic) manner into probabilistic outcomes.
The opposite of deterministic is random, i.e. in the quantum sense of truly no relationship between input and output.
There are probably some quantum effects in the brain here and there, but the vast majority of it is just traditional deterministic interactions networked together in such a complex system that the resulting behavior is much, much easier to predict in probabilistic terms than otherwise.
Ok right, but then I don't see the original point. Or like, you don't seem to be appealing to a hard science/soft science distinction here if the question revolves around what conceptual model is most "useful" to us. Useful for what? For predicting future actions? That makes sense, but does actually that speak to op's (possibly still fraught) metaphysical/semantic point here? It just feels like you are speaking over it, right? They could just as easily counter that a probabilistic model of the physical brain is not a very useful description of human intelligence, so why use it? You can't appeal to materialism here because you are already practicing some implied pragmatism in your commitments. You pretty much have as much ground to stand on as they do!
I am saying that there is no categorical distinction to be made on functional grounds between artificial intelligence and brains.
You cannot say an AI model cannot be intelligent because it's a probability machine, when all available evidence points toward natural intelligence also being generated by probability machines (much more complex ones, called brains).
But brains do much more than just predict tokens based on previously seen tokens. I think all the other things brains do are probably important for our intelligence.
So, LLM are just next token predictors, brains are next token predictors + many other things in addition, and that makes people still feel LLM are dumb even when they solve a lot of problems using tokens.
Of the vast uncertainties and philosophical exercises that we must face to bridge the chasm between where we are now, and where we will be when we understand intelligence, I can take comfort in claiming, with 100% accuracy, that our biology is not based on technology invented by Google in 2017.
> But brains do much more than just predict tokens based on previously seen tokens. I think all the other things brains do are probably important for our intelligence.
Like what?
What specific biological structure in the brain could be doing anything other than producing output as a function of 1) current electrical/chemical/thermal inputs and 2) previous electrical/chemical/thermal inputs?
No, brains doesn't just try to mimic pasts signals, LLMs do that but brains doesn't.
The way they make LLM solve problems is by adding a lot of logical jumps into its data, or break down different problems etc, and then as it predicts the text it predicts these logical jumps and then solves the problem. That is very different from how humans learn to solve problems, you don't feed them a billion different state transitions they have to encode to be able to navigate math, they learn to become proficient at math from a few hundred to a few thousand examples, that is fundamentally different from how LLM can learn.
That LLM are so slow learners that requires massive amount of data is a big reason its hard to make them smarter, and its caused by them being next token predictors. And the reason humans can learn with so little data is because we are not just next token predictors.
Okay, then what part of the brain is doing anything other than transforming an input into an output based on what inputs that part of the brain has previously been exposed to?
Well no, you changed the definition. LLMs don't just mimic what they've been trained on. They mimic what they're rewarded for. Like a brain!
Pre-training is just direct mimicry. A pre-trained LLM is very stupid and mostly useless. To become useful they are post-trained with a reward function.
This seems fine but things being the same "on functional grounds" is just one way things can be the same. Why should we privilege this framing as something definitive versus another conceivable one?
On functional grounds my bike has not a lot of distinction from a horse, but just, like, saying that doesn't tell me much about either. Or at least, it seems to leave out a lot of otherwise crucial details and differences..
What does it mean to you, this point of view? Are you truly coming from like a 20th century pragmatism point of view? Where what is most useful is what is right? Or are your trying to make a larger claim about nature? I think being clear about that would help focus your critique here.
Yours is a controversial view. It is lazy and selfish to try to get other people to explain their case that it is not exclusively that, when saying that it is exclusively that is the weaker case, and you back it up with nothing but a snarky proclamation.
Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?
> Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?
The answer is obviously yes lol.
The creature is an assemblage of electrical, chemical, and kinetic relationships.
Watching a baby develop is exactly what you'd expect from a system that's predominantly electrical noise triggering behaviors and then gradually refining denoising the relationship between inputs and outputs, with the goal function of achieving more desirable inputs.
Surely you can at least gesture toward one thing in the brain that appears not to be a probabilistic relationship between input and output?
Wow you just got us a complete nostalgia moment to the good old times when the computer who always beats us at chess became `not artificial intelligence`...
The rate of progress can be high and they can also be dumb next token predictors. Not sure why that is hard to understand.
These models can do a lot of things but they also can't do a lot of things. In order to use these models effectively you have to understand that they are next token predictors and how that allows it to do what they do.
Are they useful or not? Will they continue changing the world or not? People who choose one way or the other for describing them typically fall on one side or the other in these questions imo. What do you think? Will these next token predictors change the world or not?
They are useful. They will continue to change the world. They are still next token predictors with all the problems that comes with that.
For them to change the world you have to work with them as next token predictors. Ensure that the next token predictor has enough prediction paths to solve the problems you want and so on. Since when they don't they fail spectacularly. These big companies will continue to add new skills to them, so they will continue to get more useful.
In all fairness humans can also be considered next token predictors. It could be said that’s how we communicate with one another today. Presently LLMs lack other things, like physical presence in the world and continuity of input sensory data.
Humans learn to be a next token predictor as a kid when they learn to speak, an LLM cannot learn to be a next token predictor or anything of the sort, we have no clue how you could have an LLM learn human language just based on a thousands conversations with a human.
You don't see how that is very different? For an LLM to be as smart as a human it has to be able to learn like a human. Like you don't evaluate how smart a human is based on how much he knows, you evaluate it based on how fast he learns. And LLM are so bad at learning its ridiculous, they lack that part of the brain that lets humans be smart and learn so fast and easily.
It can be a token predictor and still tell me exactly how my life will proceed from now until the indefinite future, or be the most intelligent conversational entity you have ever witnessed.
The issue is of course with using the word "dumb": they are next token predictors, no doubt about it, but whether LLms as a class of system are smart or dumb is entirely unknown and entirely variable in time.
To interact with them effectively you must know how they behave, just like you have to know how humans behave to interact with them effectively. If you disagree, find someone with autism and have a conversation with them.
The "dumb" part comes from how it behaves in contexts where it lacks a lot of data, or where the data is skewed. Since they are tuned to give a prediction anyway and just make something up since sometimes those made up things are useful they will produce dumb results.
So people call them dumb since like dumb people they make strong statements about things they don't understand. And it doesn't matter how much smart things you encode them with, they will keep making strong statements about things they don't understand until they are fundamentally changed.
But since LLM are very smart about things where they have extensive data they can still be used to reliable solve many problems and probably in the future where we understand that better almost completely replace most lawyer and doctors work etc, because a lot of what a frontline doctor or basis lawyer work is very repetitive and can be encoded with billions of examples and decision paths into an expert system framework the LLM will follow.
So people say LLM are dumb since LLM will always keep making dumb statements. This is the same way we call Elon Musk dumb for making a lot of dumb statements, he is a smart guy but he makes dumb statements so her is dumb.
> they will keep making strong statements about things they don't understand until they are fundamentally changed
If ever there was a human quality.
Also, your explanation of "dumb" is really favoring the anti-llm side, and its a very generous interpretation. I suspect what is much more likely meant, is that token predictors cannot be smart, not now nor in the future after improvements, because they are token predictors and predicting tokens is not how intelligence works.
All of this is of course unfounded, and hidden behind the word "dumb".
> I suspect what is much more likely meant, is that token predictors cannot be smart, not now nor in the future after improvements, because they are token predictors and predicting tokens is not how intelligence works.
Why do you think that? LLM are used as expert systems today, in order to quickly navigate problems by breaking them down and iterating between different well known possible solutions and paths to check etc. That is how they work, they do that by using their next token predictions, and for things they aren't well trained on they will produce dumb results.
LLM has solved enough problems that almost nobody has the view you ridicule here, but there are still many who think LLM are thinking just like humans and that you can trust them just like humans. So its important to remind people these are just token predictors and lack many things humans do.
> If ever there was a human quality.
Humans can avoid doing that by using introspection, LLM can't. That some humans do it by not using introspection doesn't mean humans are incapable of it, we know humans are capable of it, which is why we can point out when the LLM is wrong with certainty, humans as a group make extremely good predictions.
Much of an LLM's capability comes from the structure encoded in its learned representations. The probabilistic outputs are primarily a way of expressing uncertainty and generating fluent text, while compression during training is what forces the model to discover that underlying structure.
> Much of an LLM's capability comes from the structure encoded in its learned representations
And thats encoded as a set of next token predictions. So the way to see how reliably it solves a problem is to look at the chain of predictions, and see where it is unreliable at finding the next spot, or where it always fails and you need to add that link to the dataset to train it.
This isn't magic, today we understand pretty well how to add new skills to LLM, and the better this is understood the faster progress will be.
This also means that if a context doesn't have any good predictions, it will produce a dumb prediction for that context. This results in these bad outcomes, because currently LLM doesn't have a map for where predictions are good or bad.
Tell me how a 'nExT toKeN prEdIcTor' can make breakthroughs in math or play a game of chess. These activities aren't pure symbol manipulation, they require actual understanding at some level.
My point is it's silly to whine that HN is a place where multiple points of view on the topic are aired out and discussed.
If you want a personal echo chamber where only your own beliefs are affirmed and anything else is flagged off or downvoted, I'm sure you can go find one or, worst case, vibe code one into existence.
Fair so let me be clear. I’m whining because the “next token predictor” reductionist point of view has been wrong and is only growing more wrong with time. Clearly these things can do things that actually matter. Do you disagree?
Even now you're engaging in this discussion as though I'm trying to litigate your point and that somehow forcing me to concede is, what, winning? I don't know.
I get the impression you want me to concede that the particular points of view you disagree with aren't worthy of representation here on HN.
I just prefer HN comments to be better reflections of reality. There is an unspoken expectation here that people here know what they’re talking about especially when it comes to technical matters. The rise of LLMs has given way to a HN branded populism that willingly denies reality as well. “Next token predictor” truthism is just so dumb and completely ignores the reality of what these tool are able to do. Smash that upvote button every time it feels good if you want but it’s just a meaningless take at this point. It won’t help you predict anything that’s coming.
Since we disagree on the present let’s informally do a “remind me 2 years” to this discussion and see what’s happened then.
> I just prefer HN comments to be better reflections of reality.
You mean your particular version of it.
It's interesting to see you consistently missing this point.
You've decided LLMs are clearly more than just complex but mindless statistical models.
You've decided that based on, it seems, the very impressive things these tools are capable of.
Therefore if anyone claims they're just mindless stastical models--with or without any attached judgement as to their actual utility or usefulness--then they are ipso facto wrong.
(And yes I just used endashes, damnit!)
That's on you.
It is in fact possible to simultaneously believe that LLMs are mindless token predictors and that they're enormously powerful.
These are entirely orthogonal beliefs.
Heck you could equally believe that LLMs represent true emerging AGI and that they still remain deeply flawed and are only an incremental step along the path of automation.
Or somewhere in between.
And discussing that space of possibilities is, I'd hope, precisely what HN is for.
It’s just a boring and unhelpful complaint that afaict largely serves to soothe the commenters ego rather than point at anything insightful that’s useful or predictive. Point me to your favorite “next token predictor” comment that was actually insightful or predictive. You have years of material to draw from.
It's almost as if there were many people using this site, and there is no clear consensus on LLMs, so people from various camps upvote interesting stores to support their cause. And people who are still somewhat undecided upvote both, if they present good evidence.
I mean even perennially contentious topics will get this behavior.... some thing about emacs makes the front page, within a day or two there will be a vim post up there. Same with Rust is (good|bad), or if systemd creates an even more awesome tool, the haters will come along and recycle stories about bugs from over a decade ago.
There's a lot of people here. Not all of them read it every hour, and discussions like this among large groups often take a very long time with lots of repetition. Human group dynamics (aka politics) is slow.
> So tired of this discourse and this site.
You're welcome to leave if you don't like it. The site was like this long before you joined, and will like it long after you leave I'm sure.
It's also worth noting, that an awful lot of math discoveries are perfectly in line with dumb next token generators - they are finding a way to formally construct an argument and being surprised when it doesn't work, or surprised at the outcome of the grind. Not all of them are made by brilliant leaps of intuition.
This is a conflation of issues, predicated on false understanding of what LLMs are.
This line of critique is pernicious because it is both technically correct, as description, and profoundly misleading.
Saying that outputs are a product of inputs is not interesting and to the point it is not explanatory.
What is interesting, is how they do what they do. What is the "statistically likely* next token? To answer that you can do exactly one thing, run the LLM. That's because what they are doing is interesting and not reducible.
What is more interesting is that in order to do what they do, given the architectures we apply and the training strategies we use and the harnesses we situate them in, LLM are recapitulating in their deep layers strategies observed in the animal brain. This is still suggestive, interpretibility is nascent: but it is also more than a little interesting. In some respects, for cognitive scientists interested in the manner in which mind merges from computational substrates, it is profoundly interesting.
One can incorporate this, and, still be viciously critical of bother the success and failure of LLM in the applications we have put them to, and of how we (as individuals and as institutions such as corporations) are integrating them into our work.
There is a lot to criticize!
But criticism can be taken more seriously when it is not obscured by misunderstanding or misrepresentation (intentional, or not) of what LLM are and why they are not remotely "parrots" in the pejorative sense.
The technology, as technology, at the scale we are architecting it, is doing things we did not imagine would be witnessed in our lifetime, if ever. Dismissing that and denying it because of the career, industry, society, and civilization challenges that technology brings are existential, is bad argumentation or bad faith.
> We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.
I see no credible corroboration. More likely its folks having no more care for what they are doing than the bots themselves.
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
People said the same about email spam ... until they engaged spam filters. CVE report slop is simply spam. Complaints are better directed at the filters, not the filtered.
This is the right conclusion for completely wrong reasons haha. I love hn. (aside from being entirely wrong as most models are now mixed modal so they are token prediction engines)
Each layer of attention can more through feature space “lit up weight clusters” in a way no other previous AI can. It can from that decode some rudimentary logic and world modeling and make deductions. Certainly better than any previous AI. Only a goof here would believe this wasn’t a serious advancement.
So don’t over sell it. But don’t sell it short with this “grrr in an engineer don’t threaten me with new tech” attitude.
This take is akin to teenage angsty takes and doesn’t really belong here.
Does a dog possess intelligence? Does a bird? Does a cricket? An amoeba?
I hate AI slop as much as the next guy but the amount of tribalism over AI is taking near-religious forms.
Nobody knows what intelligence is, therefore we don't know what does or does not possess it, therefore we don't know whether LLMs currently, or in the future, possess it.
Yes, LLMs can be stupid, guess what: so can I. That doesn't really change the argument at all.
I feel like I'm on a deja-vu from when DALL-E was released and everybody was fighting over whether AI can be creative yes or no. Same story, different words.
Intelligence, creativity: we have no idea what these words mean, and AI is helping us understand them better. That alone is an achievement of epic proportions. I am not joking here. Any computer scientist before 2015 would be absolutely blown away by what you can now do for 10 cents and an API call, yet somehow because of the tech-bro-iness of it all we get a tribal war over what is plainly visible in front of us:
LLMs are uncomfortably close to what we thought intelligent machines would look like
"Dumb next token predictor" keeps popping up since that is the core way they work. Since they aren't logic engines but prediction engines they will always return a result regardless what you ask it.
Some predictions might be the tokens "I don't know", but that is based on the model mapping your text to those tokens by having seen many similar "I don't know" responses to such contexts, it didn't do any introspective logic to produce that "I don't know", and its possible it actually does know if it followed another branch there so "I don't know" is often not even true.
If they had an introspective part that stops the prediction when its too unreliable it would no longer just be token prediction engine, and I believe we need such a part for them to become what I call smart. I don't think LLM will ever stop being dumb without such an introspective part to them.
And no, that introspective part is not a part of the token predictor. At least not in us humans, the feeling of certainty we have is not a prediction, it is bundled with our thoughts, so we get both "answer is a bear" and "certainty is low", we don't get just one of those as a "prediction".
Will LLM become smart as humans with such an introspective part? I don't know, but I think they will never become as smart as humans without one.
Note: The certainty score has to be per conclusion or response, not per token. You can't evaluate a responses validity by aggregating the weight of each token. Meaning its a logic engine, not token engine, that evaluates the certainty of a statement being correct or not instead of a token being correct or not. That is the level human thinking works at and seems to be dramatically more efficient.
I strongly suspect that if you dug deeply enough into how a human brain does reasoning, the answer would boil down to "it tries a bunch of pathways and picks the one that seems right". Pathways weighted by experience, but ultimately still guess-and-check. What would the alternative be?
Maybe our ability to reason is not as mystical and special as our ego might hope it to be, and discomfort over LLM's bringing that to light is the root of some people's urge to continuously downplay and discredit them.
(As far as I can tell as a non-neuroscientist, the literature on how our default mode network and prefrontal cortex interact agrees with my assessment - free association and logical verification respectively)
You have introspection, you can see a part of your thoughts, you know how that introspective part works since its what we call consciousness, you are conscious about it.
That introspection isn't an illusion, what your consciousness see of your thoughts obviously are tings that has been calculated and aggregated by the brain, so we know the brain calculates and aggregates those thoughts and feelings to produce its results. And we know LLM doesn't do that, it doesn't have a side system that does that sort of introspection.
It actually is an illusion though. The narrative the brain makes up after the fact for why we did something is not perfectly correlated with the actual reason. It's just very good at its job of spinning a convincing narrative, even to itself. This is demonstrated in split brain patients: they're given instructions to do something that are shown only to one eye, the one connected to the non-speaking hemisphere, and when asked why they did it, the speaking hemisphere confidently hallucinates a wrong answer that the person seems to fully believe.
> The narrative the brain makes up after the fact for why we did something is not perfectly correlated with the actual reason
But it does that introspection, we evolved to make it. If its not useful for anything we wouldn't have evolved it, it can't have been easy to evolve a consciousness so there has to be purpose for it.
Or do you think our consciousness is a magical ghost thing unrelated to the brains workings? I think its pretty obvious that our smarts in part depend on the computations that results in our conscious experience, you need a very strong argument as to why that wouldn't be the case. And broken brains being broken doesn't answer this.
If it keeps doing dumb things, yeah. But if that actually solves it then those opinions will quickly disappear when it replaces all human white collar work since it does it cheaper and better and faster.
AGI is fairly easy to detect for this reason. Does this system make a majority feel you don't have to hire people anymore? If not its not AGI.
Solves what? Chess? No thats not AI, its just a chess bot. Turing test? No, thats not AI, its just a dumb token predictor.
Why would the goalposts stop moving at AGI? I am 100% convinced it will somehow still lack the "gusto" or the "taste" or the "timbre" of real intelligence. You can see it in coding right now, AI has bad "taste" in coding, because really, we can't do a better job critiquing that which is obviously (on occasion) just plain good.
> Solves what? Chess? No thats not AI, its just a chess bot. Turing test? No, thats not AI, its just a dumb token predictor.
You are moving the goalpost here if you think the chess AI was AGI. All those problems were evidence AI wasn't as smart as humans, not goalposts determining when AI is as smart as humans. The first turing test winner wasn't even an LLM, it was an expert system, so we already knew that test wasn't enough for AGI.
> A human must verify everything an LLM presents as fact. Everything.
I've thought about this for quite some time now.
No. A human doesn't need to verify everything. And the argument is really simple: stochastic.
Think of self-driving cars: We can show today - based on evidence and real data - that self-driving cars are safer than human drivers. That's a fact and the consequences are clear, more self-driving cars, less human-driven cars, less accidents, less hurt people, less dead people.
Are the cars 100% safe and NEVER make a mistake? No. But they don't need to. Nothing is ever 100% (in the real world).
Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.
Yes, there will be another layer in the system, another level of abstraction that will/must end at the human boundary.
> Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.
Let me know once the majority of software engineering organizations start only checking in markdown files and let code be generated non-deterministically from these specs in CI. If this is not happening now, there’s clearly a sufficiently high level of distrust in blind LLM output (both code itself and reviews).
And even then, are you suggesting humans shouldn’t at least review the markdown specs? Why not have LLMs review the specs then? Is there, perhaps, some fundamental quality to human review process that is desirable here?
This reduction of everything to stochasticity is silly. Or, to put it differently: Do you accept a value with some error appearing in your bank account on salary day?
We have plenty of systems where complete accuracy is the only acceptable thing. Computers are great for such things. Until we all get caught up in a way of delusion and start writing those systems as natural prose passed through an improperly understood stochastic machine.
You are right with the analysis, but wrong with the conclusions. Yes, LLM „thinking process“ is kinda non-deterministic in a sense that it does not follow logical reasoning and will not produce logically correct results in 100% cases. It has an error margin.
However, error margins are in the center of any engineering discipline. We cannot produce things measured with 100% accuracy. This is accepted fact. The focus is always not on eliminating errors, but on reducing them to acceptable minimum. With LLMs we should not expect an ideal logical thinker, but a process that may error sometimes, and we must design quality controls instead that push LLM outputs within acceptable margins. And it can work.
Yes but the key here is doing proper risk assessment. "What is the consequence if the LLM gets this wrong?" "How do we verify the output?" "What are the legal ramifications for using the LLM in this way?" "Who is responsible when the LLM fails?" "Whats the expected accuracy here?" etc.
In the current AI mania, there's a lot of due diligence simply being ignored. Plenty of "Well humans make mistakes too!" going on here on HN too.
The due diligence not being done is people putting cats in microwaves. It‘s not the dangerous part. The real danger is risk assessments coming to wrong conclusions, because it is still terra incognita. Talented engineers were in this situation before, doing mistakes with cars, airplanes, buildings etc.
No, I'm sorry but I think thats a cop out. The fact that LLM are stochastic and can give incorrect answers is not particularly difficult to comprehend, and the risks that fall out of that are reasonably understandable. The issue is entirely down to bad choices by the people driving LLMs, because they are engaging with what they wish LLMs do instead of what they actually do.
We can chalk this up as another example of over-exhuberance by what folks believe humans can accomplish vs. what they actually are.
Flesh-based “brain” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your flesh-based brain gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
Humans are prediction engines. They are not Pure Intelligence, and shouldn’t not be treated in any form or fashion as if they possess pure intelligence. What bothers me about this entire situation is that presumably the folks that have relied on the flesh-based “brains” to generate these vulnerabilities knew (or should have known) enough about their "tool" to know this would happen, but did not: To err is to be human.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this over reliance on fallible “brains".
A human must verify everything another human presents as fact. Everything. If you don’t, we all pay the price. Using a human does not remove the onus of responsibility on the human being in charge, if anything they amplify it because humans work for peanuts in some countries, and can generate lots more output more quickly that needs to be verified by the humans in charge.
The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult.
But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.
The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.
This is what I truly don't like about the CVE system.
One such example is CVE-2023-45853 [1]. Zlib included in it's source an extra set of utilities and add-ons. One such utility, MiniZip, had a buffer overflow vulnerability. BAM, 8.8 CVE (was a 9, looks like they pulled it back a bit). But not one that the 99% of applications using zlib would ever be vulnerable to because almost nobody used the MiniZip utility. It was so unused that the solution for zlib was to simply remove it.
I know about this one particularly because our security policy required us to do a BUNCH of pointless updates for it since zlib is in just about everything.
I would be careful with this line of thought: opening a malformed archive that gets you into user space is often the first step in a chain-of-attacks that ends up winning Pwn2Own, so I think that a 8-ish score makes sense for it. It won't be enough to do much on its own, but for the past decade or so it's been all about chaining enough small things together to get root, rather than single points of failure.
I keep feeling like the entire CVE system is creaking and about to fail- the signal-to-noise on that has gone so far down, the gate-keeper function that MITRE et al are supposed to be providing clearly has fallen apart, but we do need some way of tracking problems and alerting on bugs in commonly reused libraries. So I can see the problems with the current system, I just can't envision the new, better system to build to replace it.
And you’re right, I don’t know all the ways this can be combined with other attacks related to the shell.
But when I design a system the security people at work tell me to assume any bad thing can happen in a user process and design the process isolation and policies to contain it.
Entire categories of attacks (Dos) etc are futile to do anything in user space. Just limit resources and call it a day.
Reminds me of the time our “security” team fought with me for a year to fix an issue in a third-party dependency that was failing their scanners. The issue was only a problem for and intentionally left in to not break support for Internet Explorer 6 on Windows XP, neither of which anyone at the company had used in 10 years, and everyone using this software was on a Mac anyway.
My condolences to anyone who’s got to deal with all these slop-y CVEs on one side and brain dead security teams on the other.
Thats not enough. This problem pops up everywhere people do CVE analysis. If a tertiary part of some library is hit by some arbitrary CVE, why should that affect the core functionality people actually use? If I use ChaCha20, and that's bundled in a library with md5.do I care that their md5 library has a vulnerability? No, not really.
If you keep driving the "seperate the packages" angle you end up with is-odd, and that not a nice play to be either.
Even more than that, a lot of vulnerabilities require untrusted input to exploit, but the library or executable is usually only used with trusted input. For example, suppose the chacha20 function had a vulnerability that was exploitable with a malicious secret key, but you only call it with your own, definitely not malicious, key. The vulnerability doesn't impact you, but your security policy might still require you to update. And no amount of splitting the package will help with that.
I have concluded that even though I won't use that key, I still want the issue fixed. You never know if the key I randomly choose next year is the vulnerable one. Worse, you never know if someone else won't make a bug that allows outsiders to select a key...
At least for native binaries it's not too hard to write a scanner that extracts symbols from the binaries and checks them against the symbol names of the vulnerable code. If the code is properly dead and got pruned by the linker, it cannot cause any problems.
Yes it can cause you troubles, even if you don't link to it. Just having a vulnerability scanner (incorrectly) flag it can cause you numerous problems.
Say you distribute your application to a customer and link against libzip, and that customer points to the minizip 8.8 CVE above and says "if you don't fix this vulnerability our scanner is flagging, we are required to stop using your product", that'll cause you plenty of problems even though you don't use nor provide any way to reach the vulnerable code.
That sort of blind CVE adherence is so common in the industry that it's usually easier to do meaningless updates than fight back.
Well that's the thing, to build the utilities required extra flags which were pretty rarely enabled. You can, in most distros, ultimately install minizip as a separate package but few do. It's not that useful of a utility. Most people will likely just grab the full blown "zip" application https://infozip.sourceforge.net/Zip.html . Though on linux, even more people are simply using tar with a zlib extension. That's the `tar.gz` files.
I think this presents a highly fascinating scenario:
1. A vulnerability that presents itself very rarely
2. If it presents itself, it is fatal to the organization
So, few targets are affected, but the outsized effect means that the overall economic impact is not insignificant. How does one keep track of that?
One method is to use a sort of "kirchhoff's law" approach, similar to google page rank. The electrical analogy would be:
- the attacker has a high potential, let's say 1000V. that attacker is the root of a rose tree, going from top to bottom, branching out towards the bottom.
- nodes impossible to exploit present edges with infinite resistance
- nodes that can be exploited present finite resistance (perhaps very low)
- once a node is exploited, its node branches out to other things that now become accessible and may or may not be exploited
- real economic impact presents a connection to ground, measured as 1/($economic impact) ohms resistance
Once you have a structure like that, you can measure the hypothetical current that would be flowing, and measure the severity of such exploits.
Sometimes a single path through a complex system can blow up vital, important parts, and this could be one way to measure it in some sort of objective manner.
The hard part is accurately making that flow chart. You have your software composition you have to keep accurate. Your software configuration and it's changes. Environmental systems and its changes like VM hosts and network hardware. And network pathing and firewalling.
When does something become a ground can be very difficult to figure out with the above complexity.
And when you chain another weak CVE to run the tool in a method that would have previously been a noop?
Security can be a major pain in the ass, and it's pretty often we see CVEs that were low valued suddenly become more urgent when someone finds a better way to use them as an exploit.
Isn't the real difficulty in how vulnerabilities can be mixed? A given vulnerability might be extremely hard to execute, but very damaging. Another vulnerability might be easier to execute, but it can't do much. But if the second one can be used to trigger the first one, you then have attack surface area of the second vulnerability with the damage of the first one.
Even if some individual case can be shown to be safe from being combined, can we identify such cases with enough confidence to justify using it reduce severity warnings?
In testing with LLMs with good exploit finding capabilities and a lack of guardrails on writing exploits, quite often LLMs will chain together a surprising stack of exploits to get what they want. They'll get access via a weak, but limited user. They'll search around laterally until they find something else. For example User A (weak exploit) -> User B -> User B bad configuration -> root (or application takeover).
They don't get bored like humans and they can be more robust than a set of scripts by far in catching errors when their scripts don't work or interesting things show up in their findings. The huggingface break down is a good example of just how much lateral movement an LLM can try with enough processing power behind it.
This is where the security onion gets messy. It can be hard to predict when compromising one layer actually bypasses many other layers of your security.
Yes, the CVE system is broken. LLM-generated CVEs is making it even worse.
I suppose all these fake issues and the many more that have absurdly elevated severities could be considered an attack on the system itself, stripping it of credibility.
That's the worst case scenario right there: so many fake reports that the real vulnerabilities can no longer be reported properly and fixed quickly. Assuming some LLMs really are capable of finding serious vulnerabilities (which in my experience is absolutely the case : just let a good model run on your security code base and see for yourself) that means that we can more easily find vulnerabilities, but it's much harder to report them and be taken seriously - so the incentive to exploit such vulnerabilities instead has become much, much higher.
Having your LLM look at the security really isn't a great option. Most users don't have unlimited budgets and can't dump the necessary time and funds needed to find these flaws.
On top of that, ability to prompt about flaws makes all the difference in the world. If you know what you're doing you can get better results. Problem, very few people know what they are doing.
Going back to the money/time problem, LLMs are not deterministic. Much like the day you forgot to drink coffee, running LLM systems against code can get different results where things are found or not found. That or two different things are found on two different runs (with the same code) because attention was stolen by the first issue it found.
So yea, in the LLM age proof is in the exploit pudding. Who cares if you're taken seriously, if you dump out an exploit that works the seriousness is now on the defenders time and people will scramble to fix it.
"But what about exploit embargos"... Well, you just found the exploit with an LLM, how many other nefarious groups do you think are already exploiting it? The age of sitting on problems is dead.
The majority these days seems to be real issues, but probably not exploitable. The latest LLMs are very good at reading code and finding issues where the code is wrong. They are somewhat less good at finding exploits - both because finding exploits is harder than finding code that is clearly wrong (even if not exploitable), and also because the better ones are intentionally made to not create exploits (since they don't know if you are a good or bad - they assume bad.
However if I was writing this response just one year ago I would instead be saying: the majority off LLM CVS are noise where the code is correct, and often they are writing up for code that doesn't even exist.
Which is to say I suspect the repo in question was generated with a year-old LLM, since they act like that. The new ones [mostly?] are much better.
Still, if a modern LLM points out something you should fix it. Even if we can't figure out how to exploit it today that doesn't mean we won't figure it out in the future.
> I suspect the repo in question was generated with a year-old LLM
I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.
> The vast majority of CVEs are not exploitable, basically noise.
This is not true if you consider security-in-depth. Many of them are exploitable on their own but maybe not in combination with other issues that are as yet unknown or known but not patched everywhere.
As a simple example a local privilege escalation issue that is not exploitable on a device that only I ever have access to, essentially becomes a remote root access flaw if you have untrusted or unreliable users (clients with accounts for instance) on the system. This works on a finer grain too, seemingly minor issues spread through the kernel and user space can add up to a serious exploit.
This is the most common "not exploitable" CVE I have to deal with at work all the time. Things like buffer overruns when passing headers, but my server shouldn't be accessible to the public Internet anyway, and if it is, then we have much bigger problems. So yes it's technically exploitable IF other safeguards fail, but the reason it's irrelevant is that if those other safeguards fail, we are in deep shit anyway.
But if you have proper risk-based vulnerability management, you should already have classified that CVE as a lesser priority. This is something that modern LLMs should in theory make a lot easier, although I’m unsure if workflows exist to continuously build a proper architectural understanding of your systems, at least at a feasible cost.
This sounds like a piss poor implementation of zero trust on your part.
Security is an onion, you have lots of layers because layers peel off and fail all the time. With your current setup a single foothold gives an attacker all the lateral movement they need for full exploitation of every single thing you own blowing past the other layers.
It's been proven over and over and over and over and over and over and over and over and over and over and over ......
that requiring access to a privileged network is NOT a sufficient security boundary! Someone you don't like always somehow ends up getting access to the network.
Or you can just fix the issue that way if IT screws up you are not vulnerable. Or maybe next year you will decide it is useful to connect to the internet.
I once had a customer threaten to reject a delivery over a CVE. I dug into it. The CVE was that an Ocaml PostgreSQL client didn’t implement an authentication feature.
I convinced the customer to accept the delivery by pointing out that (1) our app had zero lines of ocaml and (2) the feature had been implemented in the ocaml driver since the CVE was issued.
I'm sending all my condolences out to everyone who is in a corporate environment using tools like Veracode or Snyk to automatically assess for CVEs and block merges or deployments based on "findings".
My experience in such environments leads me to believe this is going to be a rough ride for those heavily locked-down enterprises, because depending on the environment, an exception of "this CVE was hallucinated by AI" is probably going to be difficult to get accepted, and when it does, starts to become its own avenue for exploitation and adds even more noise and confusion to the mix.
They’re not bad if you give up on being able to reproduce arcane bugs. Don’t pin versions, pull everything from latest, CVEs are just a rebuild. Certainly not an amazing idea, but it solves the problem of the day.
The funnier, Kafakaesque problem of the day is interactions with mandatory cooldown periods on new versions because of supply chain risks.
I’ve had a couple tickets get stuck because the CVE scanner says I have to update, but the cooldown enforcer says the version hasn’t been out long enough.
In enterprise software..... I only dream of work where some ancient version isn't pinned because 200 person hours are need to adapt the code to newer versions.
And the update versus cooldown are fun. Can't win either way you go.
In our organisation, my security team spend the majority of our effort weeding out unexploitable CVEs - which are the majority of them. But the volume is growing year on year - I don't know how sustainable it's going to be.
Snyk routinely tells us that we need to upgrade urgently because they found a vulnerability in the Windows version of a package. We run everything in Linux.
Which is precisely why I do not celebrate when Microsoft or other companies say they released "their biggest set of security vulnerability fixes ever" because there's a good chance that most of them are hallucinations or non-critical issues erroneously considered zero-days or critical.
What actually needs to be done is human review, vetting, and approval of those. Otherwise, it's just unknown/unnecessary/potentially dangerous changesets in high quantities.
Eh, when we see this from Microsoft, Linux kernel, and Apple all at the same time just thinking it's everybody being dumb becomes risky.
Now, if you're an F100 that's currently not using Anthropic it's pretty likely you can go to them now and ask for a non-guardrailed unlimited Mythos trial for a month you can get an agent that will not only find bugs for you, but also write exploits in a framework that will demonstrate them in action. Once you see an AI chain a bunch of shit together, get a foothold and then expand around laterally in your system you start getting concerned about how you're going to keep up.
The time has come to pay for our sins of shit security.
We can shoot for 100% accountability by demanding 100% accuracy.
Of course what will really happen is anyone looking for problems will stop going to the source for fixes and will start gaining cred in the dark markets of exploits and problems, which is better for everyone, right?
Just because someone is using AI doesn't mean they didn't find a bug, it's more likely they just have no means of determining if it's a real bug or not, so to the maintainers tickets seem random. The problem comes in when clever attackers can take these untrustworthy tickets and run them through another LLM that determines if an exploit is possible and then uses the attacks it finds before anyone has a chance to patch.
I think in both cases, the LLM use needs to be handled by someone experienced with the domain.
We are seeing this over and over with product development, writing, images, etc. Without good people driving this it's just noise
Edits for clarity: seems like this was a controversial take, which I find pretty interesting.
I believe people took this comment as LLMs being better than security aware engineers who have the time to spend building solid systems.
This wasn't the point, the reality outside "established" tech companies is that software security can be lackluster.
There often simply isn't enough resources to check old software for basic vulnerabilities, outdated packages with known issues, there might be a manager who insists on a certain solution. Or that certain services are on "maintenance mode", but rarely get checked since they're on a certain part of the internal network.
If they are able to properly scan their full software stack for CVE issues, they cannot deal with a flood of CVEs.
----
We're in a transition period where AI will eventually make software much more secure than it ever was.
These noisy CVEs will probably lead to agents verifying vulnerabilities before humans review them.
The problem with agent reviews from what I can think of is:
- cost to use LLMs to review things
- not necessarily easy to plug-and-play in repos: (domain knowledge + vulnerability knowledge)
- especially with anthropic: able to use models defensively, without hitting guardrails
The last one is the most interesting one to me. How does the AI providers know if you're a "good or bad" guy? And does it matter if open source models is catching up?
We're in a kind of cyber arms race wether we like it or not.
All the big maintainers I know of disagree. Curl for example is getting overwhelmed with real issues that LLMs are finding. A year ago most LLM found issues where false positives (the code is correct), or hallucinations (the code in question doesn't even exist) - but that has changed and now LLMs are finding real issues in code. Many different projects have reported the same rules.
Absolutely. I'm not sure how universal this is, but I saw George Hotz tweeted something about Mythos not being such a big deal since he's able to find security issues without a claimed $20,000 in token spend per vulnerability.
The reality is that humans aren't perfect, so we do not make "perfect programs".
It is also possible to test it yourself to see how good LLMs are at finding software issues:
Find any open source project that isn't huge, then have Kimi 3 review it for potential issues.
You will find code "issues".
Most of the time it won't be detrimental for security, but often there are code paths that are not "accounted for", or you're able to bypass a type of check by interacting with the software in a strange way.
So we just need a redundant array of George Hotz, I wonder how much that costs?
This is where quantity becomes a quality. Actors with huge budgets and at least a small team to keep the AIs on track now have an immense amount of potential power. Anthropic has somewhat resisted in giving state actors like this full access to 'dangerous' models, but it appears other companies like OpenAI have no such reservations.
Before such states may have difficulty hiring enough security researchers/hackers. Now you can spin them up. Someone with a few data centers of GPU compute and enough proxies could probably blitzkrieg the internet in a day and capture ungodly amounts of information.
Not just the cost, also boredom. I have used AI to trace down some tricky multi-threaded issues (someone has to write that cross thread framework, and it was me - fortunately most of our code isn't this complex). I already knew the code well, but it would have taken me a long time to find the same issue - the more likely thing is I would give up. Even if I don't give up I'm likely to get bored/frustrated. A month worth of tokens replaced what I would have needed a year to do if I was allowed to spend a year on it.
1. George Hotz exhibits strong signs of Dunning-Kruger in most things he does nowadays.
2. Even in exploit development, his original field, he might be able to find vulnerabilities in mature, well-maintained software with security programs. But would he charge $20k or more for that service? Say it with me, Claude: “Absolutely!”
Not validating submissions seems like avenue for massive attack. Flood the whole system with endless false reports. Thus making it significantly less reliable.
That is exactly why many big projects are migrating to becoming CNA, so that randos can’t get assigned unqualified CVEs which nobody has looked at or validated.
Apparently RedHat is a CNA of last resort, so it might be possible to get your project under Redhat’s scope and go through them without having to be a CNA yourself.
That's nonsense. The goal is and always has been to undermine the CVE system because Greg does not believe in the system. The kernel is the only project that marks CVEs to fixes, it's the only project that considers every bug to be a "Security" bug, and it has a multi-decades long history of telling the security world to fuck off - all consistent with what I've said.
No one else has the process that the kernel has, despite plenty of people having software that's deployed in very similar ways.
There's zero question - this is ideologically motivated, not a genuine good-faith attempt to leverage the system.
Make a case then. My case is that Greg has, for decades, said that he hates the CVE system and rejected it and that the kernel objectively labels CVEs based on fixes (not the standard at all, CVEs are for vulns), etc. I can point to so many objective pieces of evidence to support my claim.
You seem to have a vague, leading question. If you want to say something about "other groups" or ideologies, go ahead.
I don't know what you're trying to say at all. But no, the kernel is objectively not compliant because they label fixes with CVEs and not vulnerabilities. But even if they were compliant... what would that have to do with anything? Feels like a non sequitur.
It's very hard to figure out what point you're trying to make.
It's been a problem for awhile. Daniel Stenberg has talked about it numerous times on his/curl's blog for the last 4 years. They became their own CNA to try and control it, they opened a hackerone with rewards, but now removed the rewards because it got flooded with AI generated slop daily.
So the agents started doing something useful after a period of filling mailing lists and bug bounties with slop. Sound good, but that's not entirely a good thing. The volume of good reports is a burden as well, and it's likely that long-lasting open source C/C++ projects have legitimate vulnerabilities unpatched. But we don't have any new maintainers, I think.
The real question is will [when will] these vulnerabilities get fixed faster than they can be found. LLMs are finding a lot of real bugs that have existed without being noticed for 20+ years. It is generally expected [hoped] that in a short time all those 20 year old bugs will be fixed and LLMs will settle down to just things introduced in the last month (or perhaps this pull request) and thus be manageable.
That's becoming less common. The issue is that, if training only contains examples of CVE reports, then asking for one will get you one, even if it's not grounded in reality. But if training consists of many negative cases with the response, "I can't find one," them that will become the default unless evidence pushes it towards a report. It's a classic class size problem. SOTA models seem to have this largely solved, though a determined person (just like a determined boss) can prompt one into giving a false answer just to satisfy the request.
First, is there any reason they have to? The LLM makers are aware of this problem and doing things to stop it. Maybe they will start saying "I can't find anything". Already they refuse to provide some exploits.
Second, most people are not prompting for a CVE, they are asking to review and find issues if any. Which sometimes finds a CVE, but in my experience it sometimes does say "looks good".
Hence future CVEs on a lot of projects will likely demand working exploit code for testing. This at least attempts to lock the issue down to something more tangible.
I almost feel that we have a new generation of the "script-kiddie". People without much (or any?) knowledge of software engineering using outside tooling to do "stuff" they wouldn't be able to do themselves.
Maybe it's a not a perfect analogy - in this case the intentions seemed to be laudable - but we will see much more of this, including from malicious actors.
I think this is expected with the increased usage of AI on both fronts (finding vulns and automating their remediation). Yes, there are and will be companies that help other companies validate CVEs before they raise any alarms, but that noise/signal battle has always been there.
I work as a sec eng at a biotech firm and have a tool I built that pulls our ITIL and runs a search on NVD, GHSA, etc. for enrichment and classification. But the number of CVEs I see for products we house is astronomical compared to the actual version of that product we are running (and that's only one value). But that's also a challenge when working with so many sources and having a small team try to tackle them.
The age-old validate, remediate, verify proves itself useful once again!
One can hope that will put pressure on the industry to design a better system than CVEs. The signal noise ratio was already terrible before LLMs, I cannot imagine that will still be a meaningful system in 10y.
But I’m too cynical to not consider all the middlemen who benefit from the status quo
It's going to be very very difficult to build this system as the exploitability of any particular CVE can massively vary depending on your system configuration.
There are a lot of things that are bugs, but cannot be exploited in a standard configuration, most people would wonder why this is even a CVE.
But then you have those users that would have the application, then something like a report module that was imported, and another 3rd party module that imports images, and maybe another one that arranges the images in a chart. And suddenly that isolated CVE is now exploitable by user written data because of the original bug, it just required a complex and unexpected chain to get there.
> There are a lot of things that are bugs, but cannot be exploited in a standard configuration
And this describes the vast majority of CVEs you see when you scan your dependencies. Typical case: Your regex library has a denial of service vulnerability for crafted regular expressions but your program never allows users to have any influence over the regex.
I'm very curious what organisations would have such a policy. I can't imagine it being viable for any size of org without significant self-deception (or banning the use of all open source at which point CVEs are moot anyway).
> I'm very curious what organisations would have such a policy.
I would humbly suggest any org of any size that has insurance cover that covers anything tech related (e.g. data loss/recovery, cyber etc.) has a very good look at the small print.
Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses.
Yeah, policies like this are often not coming from engineering directly but often through other parts of the company like legal, or even sales from contract negotiations. Not that it's entirely comparable, but I was at AWS when the big log4j vulnerability happened, and the handling for it was not left up to individual engineering teams, which I don't think would surprise anyone.
At a large enough company, processes for handling things like security vulnerabilities will have a lot of stakeholders with incentives that are not necessarily perfectly aligned.
Many orgs (esp w ISO27000) have a vulnerability management policy that involves patching at least critical CVEs within a short timeline. Tools like trivvy make it possible to do the scans…
I've been in such an org, & I've led initiatives to set up automated detection at very large scale. We started by issuing tickets to teams to resolve CVEs within varying timelines - ranging from a 24hr fix to 6 months - connected to the CVSS score. It wasn't viable.
- Firstly, you quickly realise how irrelevant CVSS scores are - initiatives like First's EPSS are designed to fix this but they aren't there yet
- Secondly, you need to begin implementing localised heuristics to determine exploitable code paths. This has generally been incredibly difficult to do reliably - LLMs have started to make it easier, but it's expensive.
- Lastly, you need to factor in consideration of actionable remediation pathways. A dependency upgrade for critical infrastructure might contain breaking changes that take months to fix, or two competing CVEs might be present in interdependent versions of transitive dependencies in your sbom tree.
Most orgs aren't applying any of the above three filters to reduce their CVE remediation burden, & even if they are, it's still too high to make zero a viable target.
In reality, most orgs aren't doing comprehensive detection to begin with - if you haven't discovered all of your CVEs, your remediation burden is going to be a lot more manageable.
> - Firstly, you quickly realise how irrelevant CVSS scores are
Even if you factor in the environmental score? I realize it's a lot more work, but it basically allows you to tune the score to get any value you want.
Imagine a YAML parsing library that can cause an out-of-memory exception if you give it a YAML file greater than 3 megabytes.
If you're an online service where untrusted users can submit arbitrary YAML, and an out-of-memory exception is a severe problem, then it's severity 10.
If you're an online service that doesn't use yaml in any way, but your web framework bundled the library as a transitive dependency because yaml is one of their five supported configuration options, then it's severity 2.
The problem is figuring out which of those situations you're in takes a load of time - and the flow of CVEs is endless, as CVE numbers are given out like candy at halloween. Often it's quicker to just update to the latest version of the YAML library.
Exactly, the amount of time it takes to figure out if you're exploitable generally exceeds the time to patch the application in the majority of the cases. You tend to try to focus on the ones that will cause the most interruption.
And then you can have all kind of other fun things, like users don't upload YAML so you think you're safe, but a clever hacker figures out another minor parser error that makes your JSON seem like YAML and suddenly YAML.so has loaded and you're now parsing a document that shouldn't be possible.
I could reword that CVSS is useless on its own - EPSS takes CVSS as an input afterall. The point here is that applying environmental scoring is the hard part. That's the algorithm that's containing the complexity.
I have been given a list by security. "We had an automated tool scan that machine. It reported these. Fix anything medium severity and above.
Never mind that some of them involved vulnerabilities in some part of the bluetooth stack (servers in our datacenter don't even have bluetooth). But they just didn't care
This does make some sense if it's considered a valid fix to document that you have verified that Bluetooth is disabled on the servers and therefore not vulnerable. But that assumes that the scanning tool can be told about this kind of fix, so that it stops warning about it, which I guess it might not.
So run apt full-upgrade and get the new bluetooth driver. Why bother with a fight over something that isn't even used? Just do the quickest thing to get it off your plate.
This might cause other problems, problems of the "if it's not broken, don't fix it" variety. Upgrading everything only to break something else, in a previously stable configuration, isn't worth it.
Really the days of "Lets run this stable configuration forever" are gone.
Getting rid of as much stuff in your OS and software stack as possible should be the security teams ultimate goal, so you have less to upgrade in the end. But actual security updates just come out at a tremendous rate, and you need a QA system that checks as much as it can before prod is upgraded.
SOC2 CC7.1 [1] requires a vulnerability scanner, findings tracked with tickets, assigned severities according to a documented risk-based system, severity-based SLAs for remediation, and that the SLAs mostly be complied with or have tracked exceptions.
However it doesn't mandate any particular SLA, or the details of how risks are to be evaluated.
Organisations get to write their own policy, and they don't need to commit to patching every CVE within 24 hours or anything like that.
No, but if you don't patch them you need to convince an auditor that they are not a problem. Often patching is easier.
I'm working on such a problem now - we are using an old web browser (no longer supported) to show help on one system. That is web pages were generate internally, with no links elsewhere, and no provision for the user to enter a URL. It is still easier port to a newer supported browser than to convince the auditors that that we are not exploitable. Sure it is obvious that everything is internal and we won't write html that exploits bugs, but nobody wants to convince an auditor of that.
Any org large enough to have separated the people responsible for the security exposure of the organisation from the developers with familiarity of what's deployed is likely to have done exactly this.
The thing you have to remember is that CVEs can be a) scanned for without exerting mental effort, and b) counted.
For one thing, bigcorps in regulated areas like it a lot. They push hard to get it required by the regulations (in practice if not directly). Although it's quite inefficient, it becomes a regulatory moat. A cost they can bear that potential upstart competitors cannot.
Many large organizations like banks have requirements like this and they solve it through a mix of automatic scanners, e.g. Trivvy, and self-deception as not all systems are actually scanned in any sufficiently large org.
> I'm very curious what organisations would have such a policy
Do you provide SOC2, HIPAA, GDPR, or similar certifications to your b2b customers? Then your tech stack undergoes an annual audit, and in your audit you will need to provide a paper trail for every single vulnerability in your stack.
In practice, this means that your audit compliance software (something like Vanta.com) is going to be setup to mandate every CVE in the whole stack is patched within SLA.
ITAR has no such hard requirements. Might be some orgs that tell themselves they're attempting this under ITAR but they're not doing it in any comprehensive way.
The only thing within ITAR that I'm aware of concerning itself with software supply chain is SP 800-218 requirements & that's just a load of open-to-interpretation weasel words about having CVE detection & automations in place & some defined plans for reducing the number of vulns. Pretty sure that component of it is even eligible for self-assessment.
Yes and no. ITAR (and other laws like it) are self assessment and don't specifically say thing thing.
However your interpretation / self-assessment is subject to various reviews/audits. These days the reviewers are not going to be kind to someone who just says "not an issue", they will demand strong justification. Most organizations take the view that is is easier to fix all CVEs than try to pass audits.
Thus by the letter of the law you are correct. However to meet the letter of the law without fixing CVEs is generally seen as harder than thus fixing CVEs. So the effect is ITAR (and similar laws) force you to fix CVEs.
Whatever you say, chief. I worked in that environment for quite a while, and maybe on a technicality you're right, the effect outcome is that you will do it if not for all your customers that will require it.
If by "terrible" you mean "amazing". Imagine being able to tear down an entire Jenga tower of bullshit corpospeak, and replace it with actual security.
It’s honestly not great. The security guys are completely exasperated at my job, we’re wasting time having with these. You take the scam really, investigate for a bit, write up a DNF with justification, they go and up date records, and we all just kinda hope that someone updates the scans so it stops showing up.
Something is going to give, and I suspect that the optimistic open filling is going to get canceled.
> Because no step in today's system actually requires a proof-of-concept or bug reproduction, a plausible-sounding fake advisory can slide right through the pipeline and end up in GHSA, downstream databases, and enterprise scanners.
I have no experience in the realm of security, but why isn't the submission process similar to any normal software shop (big or small) in terms of required bug repro steps?
> The cited code didn't even exist in those versions or referenced unrelated logic.
> When testing the PoC payloads they didn’t work (not triggering any crash).
I think the future is pretty obvious, if this isn't being done on projects already: you need to automate these checks and reject automatically
LLM: You're absolutely right. I didn't actually run the check. Good catch! One sec let me do that now... yep I ran the check and it definitely repro'd this time.
Could be just me but I would be quite careful running these PoC's, especially since the CVE's mostly seem nonsensical. I suppose these CVE's could be targeting some kind of PoC execution instead of actual bug reporting? (I didn't look at the code in depth).
Why is the repo even mixing CVE's for "schreibfaul1 ESP32-audioI2S" and "SQLite"? Is mixing CVE's for different products in one repo common practice?
There is no specific practice for a repo full of bug reports. Having a repo full of bug reports is already unusual. Normally you report a bug against the software that has a bug, not by creating a file in your own repo.
I've created bug reports to open source projects that have related files in my repo. Then I can just list the reproduction instructions like:
1) arrange xxx, such that yyy.. or git clone this repo where this is set up.
2) ..
I thought that might have been quite helpful not just to the person I'm sending the bug report to, but also to myself when I need to evaluate if the bug has been fixed.
It really is a shame that funding cuts at NIST has led directly to this. I don't know if there is an incentive for a non-profit to come out of the woods to do the validation and enrichment process, but boy do we need it.
Like gortok has said, LLMs are just probabilistic text-prediction engines. They cannot automate the vulnerability validation lifecycle because they cannot account for improbable edge cases. They may not even get the version of the software right...
Funding cuts at NIST did not lead directly to this. You must be new here or haven’t spent a lot of time directly interfaced with NIST on cybersecurity in decades past.
And NIST’s role with CVE’s has always been purely ministerial/clerical. It has never been their job to do even a cursory investigation into the vulnerability itself.
> Currently, we are prioritizing analysis of the most significant vulnerabilities. In addition, we are working with our agency partners to bring on more support for analyzing vulnerabilities and have reassigned additional NIST staff to this task as well.
The OP described this as "NIST effectively hit pause on deep analysis. "
It does sound like they stopped doing something they used to do in 2024. I personally have definitely not spent a lot of time directly interfaced with NIST on cybersecurity in decades past, I know nothing about it, just what I read in OP.
Are you saying the OP was wrong to call what NIST used to do "deep analysis", and/or that the thing NIST stopped doing was "purely ministerial/clerical" in a way that it would not have caught fake reports anyway, contradicting the OP? Or other?
Again, to be sure, the OP's suggestion was not that this was caused by NIST funding cuts, but by "a massive surge in vulnerability reports,"
The main point is NIST is _downstream_ of CVE issuance. Yes, they can — and still do — add disputed/rejected tags to CVEs, but in many cases by then it's already "too late." The CVE has an ID and a lifespan of its own.
NIST does not and did-not/cannot/never-has unilaterally "retracted" CVEs or prevented their issuance.
But yes, NIST's situation is not good for the world. The services they provide are hugely valuable.
I only know about this what I learned from OP and you guys.
It sounds like you guys think OP was mistaken, whatever analysis NIST was doing that they reduced in 2024 would not have prevented this anyway?
Legit question, I'm trying to understand!
OP says:
> Hit by a massive surge in vulnerability reports, NIST effectively hit pause on deep analysis. CISA and other Authorized Data Publishers (ADPs) tried to step in with their own enrichment efforts, but the global pipeline is now fragmented and drowning in a massive backlog. Because no step in today's system actually requires a proof-of-concept or bug reproduction, a plausible-sounding fake advisory can slide right through the pipeline and end up in GHSA, downstream databases, and enterprise scanners.
Do i correctly understand you are saying you think they've mistunderstood the diagnosis of what changed, the 2024 change to NIST didn't actually make it any more likely for a fake advisory to go through pipeline?
This isn't great but a negative side effect of an otherwise positive change. I know a lot of comments will try to make this into a gotcha but its really not. Decreased friction leads to misuse. Just like before we'll figure out ways (reputation, light filter pass) to deal with it.
The federal government ought to be funding NIST sufficiently to actually do sufficient analysis. Do we care about funding civil "cyber" defense or not?
(Obviously the answer is not, we only care about funding offensive capabiltiies).
This is obviously impractical. With the volume of bug reports that are generated and such a wide breadth of software no single agency will be able to handle all reports.
And honestly? It doesn't matter, even today. CVE should serve as a reasonably deduped identifier of specific vulnerabilities. It was never interesting and practical to care about 100% of vulnerabilities in a specific deployment, and it's not interesting today.
I think this will give a lot of ammunition for people on both sides, but the thing is we have made software so complex, ofc a lot of it is inherent, that we can't even tell if the vulnerability is hallucinated or not.
I find it getting closer and closer to the nonsensical 50 page mathematical proofs built with computer assited tools, that been driving reason for abstract proof systems so that we can break down such complicated examples into something simpler that we can feed into a system to help us verify.
I feel like for some reason we have built the output machine before we could even build the actual output verification machine. And aren't ready to accept that we should all move towards building some kind of system to better help verify the "truth" produced by these AI systems.
I am now very convinced that human built and proven to work verification systems for LLM outputs are more important than ever, but I just don't see any company or team working towards it.
Trillion dollar systems shipping to production code that is hardly understood by any one without any verification what really could go wrong. Honestly I really truly hope nothing goes wrong.
To note I am not anti-AI but I want to be able to have my LLM outputs be verified by more than just me under a tighter and tighter deadlines, accepting worse and worse slop just because it would less cost to me to accept the PR and then fix it in prod.
LLMs aren't sustainable software development, they are delusional slop factories if we aren't able to actually make sure the outputs have any amount of correctness.
With humans it worked because all 10 of us could in some ways verify each others outputs, and had trust in the fact that our timelines included the time for reviews.
As a non-expert in this field, I feel we have introduced more “stochasticity”. Before, We had to deal with some human error/ misunderstanding for most part but now we have to deal with power of condensed human expertise along with homogenized flaws.
nonexistent functions being referenced is pretty bad, I haven’t had that issue since like opus 4. Shows either complete lack of oversight and/or using a bargain basement model (so no even having financial skin in the game). Most likely both
We need to further emphasize the importance of responsibility when using LLM tools to produce output for others. It's great to use them for refactoring and bug discovery, but keep in mind that it's your responsibility to analyze it and iterate on it with AI. It makes your code better and develops technical expertise.
The "Hey, analyze that codebase, find all dangerous CVEs, and write a README for the PoCs, so I can post it online for others to analyze, and if I'm lucky, I'll get a paycheck or a title to add to my resumé" approach might work in 0.1% of cases, but it will generate a ton of slop for the community to drown in.
I also think GPTZero and other AI detectors have far more false positives than correct guesses. I tried it on several texts & messages I wrote before 2019, and it flagged them as 80% AI-generated.
In that case, it's reasonable to assume that AI also generated the README text for each discovered CVE. In other cases, however, we should be more cautious.
> All advisories in this repo seem AI generated when testing them with Gptzero
I pasted this blog post from "Analysis Matrix" to the end in Gptzero, and it also says the blog post was AI-generated (71% chance of AI, 29% chance of AI-Human mix).
> Over the past few days, a newly created GitHub repo (programmervuln/cveadvisory-) published a batch of SQLite vulnerability advisories (as part of other 50+ CVEs which we believe are also LLM slop except from one). NVD quickly flagged these as critical, and CISA's ADP agreed.
These two sentences seem incompatible to me. A brand new actor can just say some stuff and cause major disruptions in this way?
Thankfully I’m seeing other people mention that this article itself seems AI-generated. While there's less proof, i suspect that the actual analysis was done by an llm as well. So while i like to dunk on hallucinations as much as the next guy, who's to say this analysis doesnt have them too?
(for the record, i think the cve's are probably bogus, i just think there's an inherent problem with debunking slop with slop)
The duality of LLMs is real. On one hand, they are solving unsolved math problems, on the other hand they get fooled by the strawberry test or the car wash test.
We see this all the time on reddit and HN, where some users are building crazy things with LLM and others are swearing by how useless LLMs are. Some people claim SWE will be dead in a few years, and others are claiming the AI bubble is going to burst anytime now.
There should be some sort of defamation lawsuit possible when people publicly publish trash repos or reports like this. If you were writing a competing product in a safe language you’d love to see such FUD.
The comment is making fun of a Claude-ism where it becomes super “honest” about stuff.
It’s a joke but there is an underlying real effect where this type of language is psychologically manipulative and I would guess makes people believe LLMs output more than if it didn’t use “honest” (or “load bearing” or whatever super serious important sounding word).
Or maybe they didn't train it that way to be manipulative (although it's certainly a plausible explanation) but simply as an accidental artifact of trying to make it give honest answers?
LLM-generated images sometimes includes text from the prompt as literal text in the image, so perhaps this is the same sort of artifact? If they've told it to be honest, it responds by talking about being honest instead of actually being honest, because it has no actual understanding of anything.
We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.
LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your LLM-based “AI” gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence. What bothers me about this entire situation is that presumably the folks that relied on the LLM-based “AI” to generate these vulnerabilities knew (or should have known) enough about their tool to know this would happen, but did not.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.
Many people with no skills are taking advantage of the LLM craze to artificially inflate their own value. I see it every day on LinkedIn.
People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.
Countless directors and managers are now cosplaying as engineers. I've seen so many myself and that's just my tiny slice of this engineering world
It is pretty funny to see the shoe on the other foot, since it's usually software engineers with unearned arrogance about other fields.
What fields do you see devs think they know about? I’ve never personally seen this with other devs I work with but obviously small sample size…
In my experience, product. There’s a pervasive view that product is easy, that anyone working in it is a dunderhead who gets in the way of engineers doing the really worthwhile work. In reality, it would just mean they rewrite the software five times and have zero customers.
Oh, design too. I’ve met many engineers who simply don’t see any value in design.
This. I'm a former teacher. I left to get a PhD in educational psychology, and now I'm a data scientist for a large school division. The number of ed tech companies I interact with who clearly have little grasp of research-based best practices in education is staggering. But they can all build slick-looking websites and dashboards.
find any econ related post on hn et voila
I don't think it is much of a problem that dabblers get into Internet arguments about things they are not experts in. After all the easiest way to get a correct answer/learn something is to say something wrong.
Of course this should be differentiated from a professional setting.
And under no circumstances should we put that stuff into the knowledge bases we base our "AI" on without making sure it takes into account the context.
There have always been people with no tech experience building startups generating spaghetti code. It's just that now it looks a lot more polished.
> It's just that now it looks a lot more polished.
Yes, and that is the problem. It used to be if a product looked polished it was fairly polished engineering wise as well if we compare to todays AI slop. You can see that on steam, before AI slop games a game that looked polished mostly worked. But today you can get a game that looks like they put in a lot of effort, but its all AI slop and everything is flaky and broken, I've had way more such experiences the last few months than before that.
A human coder that was capable of coding a complex game typically also was good enough and tested things to make most things work. There were bugs etc, but at least buttons did things.
Engineers cosplay as physicists and mathematicians every day. What's your point? Think of it pragmatically. If they can do the job they can do the role.
> If they can do the job they can do the role.
Obviously.
Can they do the job? Because right now, government decisions are based on AI generated code, which was verified by nobody who can do that. So the cost of an unsatisfactory answer is quite high.
To be fair, "people with no skills inflating their own value" is what LinkedIn has always been like. But I guess LLMs are uniquely well positioned for that task.
LLMs have brought it to a new level. I have multiple people in my feed posting several page dissertations on a daily basis. More often than not, these posts are about the benefits of AI...
People who live and create on LinkedIn? absolutely. For a huge group of people LI was just the most popular job board, though.
> People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.
I like the symmetry of people being unable to detect "skill" in people who are great at manipulating language and in LLMs who are also great at manipulating language.
I heard the term "Sloperator" a few times. Quite fitting...
This isn't new due to AI, just amplified.
> A human must verify everything an LLM presents as fact. Everything. If you don’t, we all pay the price. LLMs do not remove the onus of responsibility on the human being, if anything they amplify it because LLMs can generate lots more output more quickly that needs to be verified than humans can.
The sort of person that's going to offload their thinking to AI is the exact sort of person that is not going to verify anything because they've already offloaded their thinking to AI.
For a long time I was anti-licensure in tech; now with the bar being lowered to next to nothing, it seems as if licensure is more important than ever — not to protect this trade (though it will do that, and that is a benefit), but because the sheer amount of irresponsibility in the usage of LLMs and “AI” in general begs for licensure and adoption of a regulatory body for software in general.
This is unfortunately a feeling I share. It wasn't until LLMs have become nearly ubiquitous at this point, and there has been zero realistic technological response to the dangers they present. Not to mention I suspect there may be some psychological element to being exposed to interactions with AI models and their nonsense for hours a day. Not all of it is nonsense....but you won't ever know for sure.
Unfortunately people sometimes get defensive against this take. But I think treating the LLM as you described can make you a better LLM user and help get better output. It helps understand the failure modes better, and moderate one’s reliance on them. Just like how we should do for every tool we work with.
Yes, I've found that reminding yourself of how they actually work helps keep you on guard against LLM-patterned mistakes. Especially things like carefully considering what parts of the current task likely fall outside the distribution of corpus + RL data (as much as that can be guessed).
> ...Unfortunately people sometimes get defensive against this take. But I think treating the LLM as you described can make you a better LLM user and help get better output. It helps understand the failure modes better, and moderate one’s reliance on them. Just like how we should do for every tool we work with.
B...b...but the Anthropic trainer said we'd get the best results if we don't think of it as a tool, but instead give it a name and think of it as our brilliant coworker!
Why should I trust you, internet rando over a stormtrooper-level salesman? /s
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
Brandolini's principle in action. It takes 10 times more energy to refute BS than to generate it. A related analogy to computing: it is easy to generate propositions, but hard to test if a given proposition is satisfiable or not, which curiously ties to P vs NP.
> Brandolini's principle
I much prefer the alternative name: the Bullshit Asymmetry Principle.
Seems like most of it is covered by: Entropy increases.
> LLMs are text-prediction engines. They are not Artificial Intelligence, and shouldn’t not be treated in any form or fashion as if they possess intelligence.
I agree that humans must verify LLM-produced facts, but strongly disagree with these kinds of "stochastic parrot therefore dumb" arguments.
Yes, an LLM is a "stochastic parrot". No, that doesn't imply that it is dumb. Enough to look at how Terence Tao asks ChatGPT to help him understand a solution that nobody had ever discussed before [1], or how a random guy asks ChatGPT in a handful of words to disprove a 30-year-old conjecture, with zero technical input [2].
If your parrot in a birdcage with internet access can finish the sentence, "The counterexample to the Dinitz–Garg–Goemans conjecture is...", then it's a pretty smart parrot, by all reasonable definitions of "smart". Just because someone bottled up the formula into matrix multiplications and added some random sampling to the outcome, that doesn't take away from the fact that the parrot said provably correct statements that the biggest experts in the field couldn't imagine.
And no, I'm not implying that the LLMs are correct all the time, or that their intelligence and reasoning works in any way like ours.
[1]: https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed... [2]: https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
The best way to describe the LLM intelligence is "an expert system that works the way people thought expert systems would work". You can encode a massive amount of skills into an LLM, and then the LLM uses those to navigate problems.
But the LLM is still dumb where those skills doesn't have good coverage, since unlike the expert systems it maps fuzzily to its skills, and they are tuned to produce results over rejecting the request when its unclear if coverage is good. As long as that is true you have to treat them as dumb even if they sometimes produce brilliant results.
Apophatic intelligence? "We don't know what intelligence is, but LLMs with CoT are certainly not it despite being Turing-complete."
Watching for unexpected failure modes is surely worth it.
Turing-completeness is a necessary pre-requisite for being able to fulfill the requirements of a Turing machine, nothing more. In the same way that cell division is a necessary condition for life, but cell division does not mean a given life form itself is sentient.
Intelligent life-forms can generate probabilistic outputs based on inputs, but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.
OK. A more pointed question. What do you know about intelligence that allows you to exclude LLMs with CoT from the category of intelligent systems with certainty?
How do you know there's not a teapot orbiting the sun?
To my knowledge there are many millions of teapots orbiting the sun.
No, they're all orbiting the Earth very badly. :p
There is no reason to think that teapots were sent into orbit or spontaneously formed there.
Likewise, there is no reason to think the brain employs super-Turing or quantum computations that cannot be approximated by LLMs.
The point is that you're asking someone to prove a negative.
Right, it follows naturally that one can never meaningfully discuss that which can only be settled by proving negatives. Hence the total lack of debate on religious topics, as everyone just gave up 1500 years ago and decided that God indeed may or may not exist and that's that.
It's a negative in your worldview where "intelligence can't be a computation" is an axiom or something close. To me this worldview is sloppy thinking.
I did not express my opinion or "worldview." I was simply responding to this:
> exclude LLMs with CoT from the category of intelligent systems with certainty
Why do you think that intelligence of a Turing-complete system trained on human data can be likened to a Russell's teapot?
At least, don't you think that the recent mathematical results of LLMs are a bit like a glimpse of something teapot-shaped in the orbit? (which makes it not a Russell's teapot, which. by definition, can't be observed).
To me, it's an expected progression of ANNs' approximation of human cognitive processes. The universal approximation theorem guaranties the existence of such ANNs barring the super-Turing or quantum superiority of the brain.
You’re right to point that out. The load-bearing seam of your logical retort is the hidden assumption that the teapot is in outer space, not at peace on a kitchen stovetop here on earth. I am sorry I had not considered this fact. \s
They were modeled after us, which almost certainly dooms them to stupidity.
They could have been great, if trained on datasets from a more sensible species.
> but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.
???
Of course it is. The brain is mechanically not capable of doing anything other than that.
Do you believe the brain is something other than a bundle of probabilistic physical interactions? Or are brains not the source of what we call intelligence?
We don’t know
Of course we do. Unless the brain is an object unlike anything else in the known universe, then it's an input → output machine.
May I suggest the excellent & recent book: "The brain. In theory" by Romain Brette?
This is going to elevate your thinking on this no end, if you're interested.
Looks interesting (added to my list, thank you for the recommendation), but not actually relevant to the topic at hand.
We know that the brain is a probabilistic input → output machine because the universe is a probabilistic input → output machine. The brain is made of universe. There are deterministic relationships (which at high sensitivity or complexity become easier to describe as probabilistic), and quantum relationships. That's it. The brain, like every other thing comprised of "universe" is comprised of those two types of relationships.
If Romain's book provides evidence of relationships in the brain that are neither quantum (therefore random) NOR classical (therefore deterministic), then 1) he would have already won at least one Nobel prize, and 2) anyone in this thread would be able to at least gesture toward what relationship that is.
You're a fast reader :-D
No quantum bullshittery in there I promise.
Equating "classical" with "deterministic" is however pushing it a bit too far, when no one and nothing can ever do even a very small fraction of said determination...
Quantum is not bullshit and yes, classical is deterministic.
That's why the brain cannot possibly be anything other than an input → output machine, which is functionally deterministic (with maybe some fully random components), but is easiest to describe as probabilistic.
In the same way that LLMs are functionally deterministic, but easiest to describe as probabilistic.
This deserves about as large of a "[citation needed]" as one could draw. Are you well-studied in neuroscience?
Nothing in the universe is anything other than a chain of reactions, amigo.
The brain is an object in the universe.
The universe has quantum behaviors (fully random, not a source of intelligence) and it has deterministic behaviors (fully non-random). Many of those deterministic behaviors are so complex that they're easier to analyze and describe as probabilistic, which is where most brain input → output relationships land.
Please point to any evidence whatsoever that the brain has some third type of interaction going on that has never been observed anywhere in the entire universe, then we can have a discussion about it.
Why is the brain probabilistic instead of deterministic?
Probabilistic and deterministic are not opposites.
The brain is deterministic at the level of specific interactions, which process inputs in a highly chaotic (but still deterministic) manner into probabilistic outcomes.
The opposite of deterministic is random, i.e. in the quantum sense of truly no relationship between input and output.
There are probably some quantum effects in the brain here and there, but the vast majority of it is just traditional deterministic interactions networked together in such a complex system that the resulting behavior is much, much easier to predict in probabilistic terms than otherwise.
Ok right, but then I don't see the original point. Or like, you don't seem to be appealing to a hard science/soft science distinction here if the question revolves around what conceptual model is most "useful" to us. Useful for what? For predicting future actions? That makes sense, but does actually that speak to op's (possibly still fraught) metaphysical/semantic point here? It just feels like you are speaking over it, right? They could just as easily counter that a probabilistic model of the physical brain is not a very useful description of human intelligence, so why use it? You can't appeal to materialism here because you are already practicing some implied pragmatism in your commitments. You pretty much have as much ground to stand on as they do!
I am saying that there is no categorical distinction to be made on functional grounds between artificial intelligence and brains.
You cannot say an AI model cannot be intelligent because it's a probability machine, when all available evidence points toward natural intelligence also being generated by probability machines (much more complex ones, called brains).
But brains do much more than just predict tokens based on previously seen tokens. I think all the other things brains do are probably important for our intelligence.
So, LLM are just next token predictors, brains are next token predictors + many other things in addition, and that makes people still feel LLM are dumb even when they solve a lot of problems using tokens.
> brains are next token predictors
Of the vast uncertainties and philosophical exercises that we must face to bridge the chasm between where we are now, and where we will be when we understand intelligence, I can take comfort in claiming, with 100% accuracy, that our biology is not based on technology invented by Google in 2017.
> But brains do much more than just predict tokens based on previously seen tokens. I think all the other things brains do are probably important for our intelligence.
Like what?
What specific biological structure in the brain could be doing anything other than producing output as a function of 1) current electrical/chemical/thermal inputs and 2) previous electrical/chemical/thermal inputs?
LLM doesn't just produce output as a function, they are much more specific: they predict text based on text they have been trained on.
Yes that is the item (2) parameter described above.
No, brains doesn't just try to mimic pasts signals, LLMs do that but brains doesn't.
The way they make LLM solve problems is by adding a lot of logical jumps into its data, or break down different problems etc, and then as it predicts the text it predicts these logical jumps and then solves the problem. That is very different from how humans learn to solve problems, you don't feed them a billion different state transitions they have to encode to be able to navigate math, they learn to become proficient at math from a few hundred to a few thousand examples, that is fundamentally different from how LLM can learn.
That LLM are so slow learners that requires massive amount of data is a big reason its hard to make them smarter, and its caused by them being next token predictors. And the reason humans can learn with so little data is because we are not just next token predictors.
Okay, then what part of the brain is doing anything other than transforming an input into an output based on what inputs that part of the brain has previously been exposed to?
> transforming an input into an output based on what inputs that part of the brain has previously been exposed to
You changed the definition there, for it to be like an LLM it should be:
> transforming an input into an output trying to mimic inputs that part of the brain has previously been exposed to
Anyone can see how that limits you a lot, and why that makes it so much harder for LLM to learn things properly than it is for humans.
Well no, you changed the definition. LLMs don't just mimic what they've been trained on. They mimic what they're rewarded for. Like a brain!
Pre-training is just direct mimicry. A pre-trained LLM is very stupid and mostly useless. To become useful they are post-trained with a reward function.
This seems fine but things being the same "on functional grounds" is just one way things can be the same. Why should we privilege this framing as something definitive versus another conceivable one?
On functional grounds my bike has not a lot of distinction from a horse, but just, like, saying that doesn't tell me much about either. Or at least, it seems to leave out a lot of otherwise crucial details and differences..
What does it mean to you, this point of view? Are you truly coming from like a 20th century pragmatism point of view? Where what is most useful is what is right? Or are your trying to make a larger claim about nature? I think being clear about that would help focus your critique here.
Yours is a controversial view. It is lazy and selfish to try to get other people to explain their case that it is not exclusively that, when saying that it is exclusively that is the weaker case, and you back it up with nothing but a snarky proclamation.
Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?
> Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?
The answer is obviously yes lol.
The creature is an assemblage of electrical, chemical, and kinetic relationships.
Watching a baby develop is exactly what you'd expect from a system that's predominantly electrical noise triggering behaviors and then gradually refining denoising the relationship between inputs and outputs, with the goal function of achieving more desirable inputs.
Surely you can at least gesture toward one thing in the brain that appears not to be a probabilistic relationship between input and output?
Wow you just got us a complete nostalgia moment to the good old times when the computer who always beats us at chess became `not artificial intelligence`...
Every day I wake up and open HN.
“LLM has made legitimate mathematical discoveries” —> Wow the rate of progress is amazing. Highly upvoted.
“LLM does something not good” -> Does everyone else not realize LLMs are just dumb next token predictors? Highly upvoted.
So tired of this discourse and this site.
The rate of progress can be high and they can also be dumb next token predictors. Not sure why that is hard to understand.
These models can do a lot of things but they also can't do a lot of things. In order to use these models effectively you have to understand that they are next token predictors and how that allows it to do what they do.
Are they useful or not? Will they continue changing the world or not? People who choose one way or the other for describing them typically fall on one side or the other in these questions imo. What do you think? Will these next token predictors change the world or not?
They are useful. They will continue to change the world. They are still next token predictors with all the problems that comes with that.
For them to change the world you have to work with them as next token predictors. Ensure that the next token predictor has enough prediction paths to solve the problems you want and so on. Since when they don't they fail spectacularly. These big companies will continue to add new skills to them, so they will continue to get more useful.
In all fairness humans can also be considered next token predictors. It could be said that’s how we communicate with one another today. Presently LLMs lack other things, like physical presence in the world and continuity of input sensory data.
Humans learn to be a next token predictor as a kid when they learn to speak, an LLM cannot learn to be a next token predictor or anything of the sort, we have no clue how you could have an LLM learn human language just based on a thousands conversations with a human.
You don't see how that is very different? For an LLM to be as smart as a human it has to be able to learn like a human. Like you don't evaluate how smart a human is based on how much he knows, you evaluate it based on how fast he learns. And LLM are so bad at learning its ridiculous, they lack that part of the brain that lets humans be smart and learn so fast and easily.
Somewhat.
Yes, but not because they are useful.
It can be a token predictor and still tell me exactly how my life will proceed from now until the indefinite future, or be the most intelligent conversational entity you have ever witnessed.
The issue is of course with using the word "dumb": they are next token predictors, no doubt about it, but whether LLms as a class of system are smart or dumb is entirely unknown and entirely variable in time.
To interact with them effectively you must know how they behave, just like you have to know how humans behave to interact with them effectively. If you disagree, find someone with autism and have a conversation with them.
The "dumb" part comes from how it behaves in contexts where it lacks a lot of data, or where the data is skewed. Since they are tuned to give a prediction anyway and just make something up since sometimes those made up things are useful they will produce dumb results.
So people call them dumb since like dumb people they make strong statements about things they don't understand. And it doesn't matter how much smart things you encode them with, they will keep making strong statements about things they don't understand until they are fundamentally changed.
But since LLM are very smart about things where they have extensive data they can still be used to reliable solve many problems and probably in the future where we understand that better almost completely replace most lawyer and doctors work etc, because a lot of what a frontline doctor or basis lawyer work is very repetitive and can be encoded with billions of examples and decision paths into an expert system framework the LLM will follow.
So people say LLM are dumb since LLM will always keep making dumb statements. This is the same way we call Elon Musk dumb for making a lot of dumb statements, he is a smart guy but he makes dumb statements so her is dumb.
> they will keep making strong statements about things they don't understand until they are fundamentally changed
If ever there was a human quality.
Also, your explanation of "dumb" is really favoring the anti-llm side, and its a very generous interpretation. I suspect what is much more likely meant, is that token predictors cannot be smart, not now nor in the future after improvements, because they are token predictors and predicting tokens is not how intelligence works.
All of this is of course unfounded, and hidden behind the word "dumb".
> I suspect what is much more likely meant, is that token predictors cannot be smart, not now nor in the future after improvements, because they are token predictors and predicting tokens is not how intelligence works.
Why do you think that? LLM are used as expert systems today, in order to quickly navigate problems by breaking them down and iterating between different well known possible solutions and paths to check etc. That is how they work, they do that by using their next token predictions, and for things they aren't well trained on they will produce dumb results.
LLM has solved enough problems that almost nobody has the view you ridicule here, but there are still many who think LLM are thinking just like humans and that you can trust them just like humans. So its important to remind people these are just token predictors and lack many things humans do.
> If ever there was a human quality.
Humans can avoid doing that by using introspection, LLM can't. That some humans do it by not using introspection doesn't mean humans are incapable of it, we know humans are capable of it, which is why we can point out when the LLM is wrong with certainty, humans as a group make extremely good predictions.
Much of an LLM's capability comes from the structure encoded in its learned representations. The probabilistic outputs are primarily a way of expressing uncertainty and generating fluent text, while compression during training is what forces the model to discover that underlying structure.
> Much of an LLM's capability comes from the structure encoded in its learned representations
And thats encoded as a set of next token predictions. So the way to see how reliably it solves a problem is to look at the chain of predictions, and see where it is unreliable at finding the next spot, or where it always fails and you need to add that link to the dataset to train it.
This isn't magic, today we understand pretty well how to add new skills to LLM, and the better this is understood the faster progress will be.
This also means that if a context doesn't have any good predictions, it will produce a dumb prediction for that context. This results in these bad outcomes, because currently LLM doesn't have a map for where predictions are good or bad.
Not sure what your point is? Those things can both be true.
Or should the discourse in a diverse community like HN only reflect the positions you personally hold?
Tell me how a 'nExT toKeN prEdIcTor' can make breakthroughs in math or play a game of chess. These activities aren't pure symbol manipulation, they require actual understanding at some level.
By predicting next tokens
Eh, I'm not going to litigate your claims.
My point is it's silly to whine that HN is a place where multiple points of view on the topic are aired out and discussed.
If you want a personal echo chamber where only your own beliefs are affirmed and anything else is flagged off or downvoted, I'm sure you can go find one or, worst case, vibe code one into existence.
Fair so let me be clear. I’m whining because the “next token predictor” reductionist point of view has been wrong and is only growing more wrong with time. Clearly these things can do things that actually matter. Do you disagree?
Even now you're engaging in this discussion as though I'm trying to litigate your point and that somehow forcing me to concede is, what, winning? I don't know.
I get the impression you want me to concede that the particular points of view you disagree with aren't worthy of representation here on HN.
I'm not going to do that.
I just prefer HN comments to be better reflections of reality. There is an unspoken expectation here that people here know what they’re talking about especially when it comes to technical matters. The rise of LLMs has given way to a HN branded populism that willingly denies reality as well. “Next token predictor” truthism is just so dumb and completely ignores the reality of what these tool are able to do. Smash that upvote button every time it feels good if you want but it’s just a meaningless take at this point. It won’t help you predict anything that’s coming.
Since we disagree on the present let’s informally do a “remind me 2 years” to this discussion and see what’s happened then.
> I just prefer HN comments to be better reflections of reality.
You mean your particular version of it.
It's interesting to see you consistently missing this point.
You've decided LLMs are clearly more than just complex but mindless statistical models.
You've decided that based on, it seems, the very impressive things these tools are capable of.
Therefore if anyone claims they're just mindless stastical models--with or without any attached judgement as to their actual utility or usefulness--then they are ipso facto wrong.
(And yes I just used endashes, damnit!)
That's on you.
It is in fact possible to simultaneously believe that LLMs are mindless token predictors and that they're enormously powerful.
These are entirely orthogonal beliefs.
Heck you could equally believe that LLMs represent true emerging AGI and that they still remain deeply flawed and are only an incremental step along the path of automation.
Or somewhere in between.
And discussing that space of possibilities is, I'd hope, precisely what HN is for.
It’s just a boring and unhelpful complaint that afaict largely serves to soothe the commenters ego rather than point at anything insightful that’s useful or predictive. Point me to your favorite “next token predictor” comment that was actually insightful or predictive. You have years of material to draw from.
https://news.ycombinator.com/item?id=49155075
"These models are probabilistic, you shouldn't blindly trust them in spaces where accuracy is really important" seems like pretty sound advice to me.
Not a dichotomy actually. Highly depends on the task.
Would be nice to get high karma commenter votes count only ..
Opinions differ. This is not news.
It's almost as if there were many people using this site, and there is no clear consensus on LLMs, so people from various camps upvote interesting stores to support their cause. And people who are still somewhat undecided upvote both, if they present good evidence.
I mean even perennially contentious topics will get this behavior.... some thing about emacs makes the front page, within a day or two there will be a vim post up there. Same with Rust is (good|bad), or if systemd creates an even more awesome tool, the haters will come along and recycle stories about bugs from over a decade ago.
There's a lot of people here. Not all of them read it every hour, and discussions like this among large groups often take a very long time with lots of repetition. Human group dynamics (aka politics) is slow.
> So tired of this discourse and this site.
You're welcome to leave if you don't like it. The site was like this long before you joined, and will like it long after you leave I'm sure.
It's also worth noting, that an awful lot of math discoveries are perfectly in line with dumb next token generators - they are finding a way to formally construct an argument and being surprised when it doesn't work, or surprised at the outcome of the grind. Not all of them are made by brilliant leaps of intuition.
This is a conflation of issues, predicated on false understanding of what LLMs are.
This line of critique is pernicious because it is both technically correct, as description, and profoundly misleading.
Saying that outputs are a product of inputs is not interesting and to the point it is not explanatory.
What is interesting, is how they do what they do. What is the "statistically likely* next token? To answer that you can do exactly one thing, run the LLM. That's because what they are doing is interesting and not reducible.
What is more interesting is that in order to do what they do, given the architectures we apply and the training strategies we use and the harnesses we situate them in, LLM are recapitulating in their deep layers strategies observed in the animal brain. This is still suggestive, interpretibility is nascent: but it is also more than a little interesting. In some respects, for cognitive scientists interested in the manner in which mind merges from computational substrates, it is profoundly interesting.
One can incorporate this, and, still be viciously critical of bother the success and failure of LLM in the applications we have put them to, and of how we (as individuals and as institutions such as corporations) are integrating them into our work.
There is a lot to criticize!
But criticism can be taken more seriously when it is not obscured by misunderstanding or misrepresentation (intentional, or not) of what LLM are and why they are not remotely "parrots" in the pejorative sense.
The technology, as technology, at the scale we are architecting it, is doing things we did not imagine would be witnessed in our lifetime, if ever. Dismissing that and denying it because of the career, industry, society, and civilization challenges that technology brings are existential, is bad argumentation or bad faith.
Both can be true at once.
> We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are.
I see no credible corroboration. More likely its folks having no more care for what they are doing than the bots themselves.
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
People said the same about email spam ... until they engaged spam filters. CVE report slop is simply spam. Complaints are better directed at the filters, not the filtered.
This is the right conclusion for completely wrong reasons haha. I love hn. (aside from being entirely wrong as most models are now mixed modal so they are token prediction engines)
Each layer of attention can more through feature space “lit up weight clusters” in a way no other previous AI can. It can from that decode some rudimentary logic and world modeling and make deductions. Certainly better than any previous AI. Only a goof here would believe this wasn’t a serious advancement.
So don’t over sell it. But don’t sell it short with this “grrr in an engineer don’t threaten me with new tech” attitude.
This take is akin to teenage angsty takes and doesn’t really belong here.
Does a dog possess intelligence? Does a bird? Does a cricket? An amoeba?
I hate AI slop as much as the next guy but the amount of tribalism over AI is taking near-religious forms.
Nobody knows what intelligence is, therefore we don't know what does or does not possess it, therefore we don't know whether LLMs currently, or in the future, possess it.
Yes, LLMs can be stupid, guess what: so can I. That doesn't really change the argument at all.
I feel like I'm on a deja-vu from when DALL-E was released and everybody was fighting over whether AI can be creative yes or no. Same story, different words.
Intelligence, creativity: we have no idea what these words mean, and AI is helping us understand them better. That alone is an achievement of epic proportions. I am not joking here. Any computer scientist before 2015 would be absolutely blown away by what you can now do for 10 cents and an API call, yet somehow because of the tech-bro-iness of it all we get a tribal war over what is plainly visible in front of us:
LLMs are uncomfortably close to what we thought intelligent machines would look like
"Dumb next token predictor" keeps popping up since that is the core way they work. Since they aren't logic engines but prediction engines they will always return a result regardless what you ask it.
Some predictions might be the tokens "I don't know", but that is based on the model mapping your text to those tokens by having seen many similar "I don't know" responses to such contexts, it didn't do any introspective logic to produce that "I don't know", and its possible it actually does know if it followed another branch there so "I don't know" is often not even true.
If they had an introspective part that stops the prediction when its too unreliable it would no longer just be token prediction engine, and I believe we need such a part for them to become what I call smart. I don't think LLM will ever stop being dumb without such an introspective part to them.
And no, that introspective part is not a part of the token predictor. At least not in us humans, the feeling of certainty we have is not a prediction, it is bundled with our thoughts, so we get both "answer is a bear" and "certainty is low", we don't get just one of those as a "prediction".
Will LLM become smart as humans with such an introspective part? I don't know, but I think they will never become as smart as humans without one.
Note: The certainty score has to be per conclusion or response, not per token. You can't evaluate a responses validity by aggregating the weight of each token. Meaning its a logic engine, not token engine, that evaluates the certainty of a statement being correct or not instead of a token being correct or not. That is the level human thinking works at and seems to be dramatically more efficient.
I strongly suspect that if you dug deeply enough into how a human brain does reasoning, the answer would boil down to "it tries a bunch of pathways and picks the one that seems right". Pathways weighted by experience, but ultimately still guess-and-check. What would the alternative be?
Maybe our ability to reason is not as mystical and special as our ego might hope it to be, and discomfort over LLM's bringing that to light is the root of some people's urge to continuously downplay and discredit them.
(As far as I can tell as a non-neuroscientist, the literature on how our default mode network and prefrontal cortex interact agrees with my assessment - free association and logical verification respectively)
You have introspection, you can see a part of your thoughts, you know how that introspective part works since its what we call consciousness, you are conscious about it.
That introspection isn't an illusion, what your consciousness see of your thoughts obviously are tings that has been calculated and aggregated by the brain, so we know the brain calculates and aggregates those thoughts and feelings to produce its results. And we know LLM doesn't do that, it doesn't have a side system that does that sort of introspection.
It actually is an illusion though. The narrative the brain makes up after the fact for why we did something is not perfectly correlated with the actual reason. It's just very good at its job of spinning a convincing narrative, even to itself. This is demonstrated in split brain patients: they're given instructions to do something that are shown only to one eye, the one connected to the non-speaking hemisphere, and when asked why they did it, the speaking hemisphere confidently hallucinates a wrong answer that the person seems to fully believe.
> The narrative the brain makes up after the fact for why we did something is not perfectly correlated with the actual reason
But it does that introspection, we evolved to make it. If its not useful for anything we wouldn't have evolved it, it can't have been easy to evolve a consciousness so there has to be purpose for it.
Or do you think our consciousness is a magical ghost thing unrelated to the brains workings? I think its pretty obvious that our smarts in part depend on the computations that results in our conscious experience, you need a very strong argument as to why that wouldn't be the case. And broken brains being broken doesn't answer this.
> I don't think LLM will ever stop being dumb without such an introspective part to them.
The very second they add this part, it will "just be a dumb token predictor with introspection", mark my words.
If it keeps doing dumb things, yeah. But if that actually solves it then those opinions will quickly disappear when it replaces all human white collar work since it does it cheaper and better and faster.
AGI is fairly easy to detect for this reason. Does this system make a majority feel you don't have to hire people anymore? If not its not AGI.
> But if that actually solves it then
Solves what? Chess? No thats not AI, its just a chess bot. Turing test? No, thats not AI, its just a dumb token predictor.
Why would the goalposts stop moving at AGI? I am 100% convinced it will somehow still lack the "gusto" or the "taste" or the "timbre" of real intelligence. You can see it in coding right now, AI has bad "taste" in coding, because really, we can't do a better job critiquing that which is obviously (on occasion) just plain good.
> Solves what? Chess? No thats not AI, its just a chess bot. Turing test? No, thats not AI, its just a dumb token predictor.
You are moving the goalpost here if you think the chess AI was AGI. All those problems were evidence AI wasn't as smart as humans, not goalposts determining when AI is as smart as humans. The first turing test winner wasn't even an LLM, it was an expert system, so we already knew that test wasn't enough for AGI.
Necessary is not the same as sufficient.
If I want my ox-cart to fly, I need to add wings, but that's probably not all that needs changing.
> A human must verify everything an LLM presents as fact. Everything.
I've thought about this for quite some time now.
No. A human doesn't need to verify everything. And the argument is really simple: stochastic.
Think of self-driving cars: We can show today - based on evidence and real data - that self-driving cars are safer than human drivers. That's a fact and the consequences are clear, more self-driving cars, less human-driven cars, less accidents, less hurt people, less dead people.
Are the cars 100% safe and NEVER make a mistake? No. But they don't need to. Nothing is ever 100% (in the real world).
Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.
Yes, there will be another layer in the system, another level of abstraction that will/must end at the human boundary.
> Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.
Let me know once the majority of software engineering organizations start only checking in markdown files and let code be generated non-deterministically from these specs in CI. If this is not happening now, there’s clearly a sufficiently high level of distrust in blind LLM output (both code itself and reviews).
And even then, are you suggesting humans shouldn’t at least review the markdown specs? Why not have LLMs review the specs then? Is there, perhaps, some fundamental quality to human review process that is desirable here?
This reduction of everything to stochasticity is silly. Or, to put it differently: Do you accept a value with some error appearing in your bank account on salary day?
We have plenty of systems where complete accuracy is the only acceptable thing. Computers are great for such things. Until we all get caught up in a way of delusion and start writing those systems as natural prose passed through an improperly understood stochastic machine.
You are right with the analysis, but wrong with the conclusions. Yes, LLM „thinking process“ is kinda non-deterministic in a sense that it does not follow logical reasoning and will not produce logically correct results in 100% cases. It has an error margin.
However, error margins are in the center of any engineering discipline. We cannot produce things measured with 100% accuracy. This is accepted fact. The focus is always not on eliminating errors, but on reducing them to acceptable minimum. With LLMs we should not expect an ideal logical thinker, but a process that may error sometimes, and we must design quality controls instead that push LLM outputs within acceptable margins. And it can work.
Yes but the key here is doing proper risk assessment. "What is the consequence if the LLM gets this wrong?" "How do we verify the output?" "What are the legal ramifications for using the LLM in this way?" "Who is responsible when the LLM fails?" "Whats the expected accuracy here?" etc.
In the current AI mania, there's a lot of due diligence simply being ignored. Plenty of "Well humans make mistakes too!" going on here on HN too.
The due diligence not being done is people putting cats in microwaves. It‘s not the dangerous part. The real danger is risk assessments coming to wrong conclusions, because it is still terra incognita. Talented engineers were in this situation before, doing mistakes with cars, airplanes, buildings etc.
No, I'm sorry but I think thats a cop out. The fact that LLM are stochastic and can give incorrect answers is not particularly difficult to comprehend, and the risks that fall out of that are reasonably understandable. The issue is entirely down to bad choices by the people driving LLMs, because they are engaging with what they wish LLMs do instead of what they actually do.
We can chalk this up as another example of over-exhuberance by what folks believe humans can accomplish vs. what they actually are.
Flesh-based “brain” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your flesh-based brain gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
Humans are prediction engines. They are not Pure Intelligence, and shouldn’t not be treated in any form or fashion as if they possess pure intelligence. What bothers me about this entire situation is that presumably the folks that have relied on the flesh-based “brains” to generate these vulnerabilities knew (or should have known) enough about their "tool" to know this would happen, but did not: To err is to be human.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this over reliance on fallible “brains".
A human must verify everything another human presents as fact. Everything. If you don’t, we all pay the price. Using a human does not remove the onus of responsibility on the human being in charge, if anything they amplify it because humans work for peanuts in some countries, and can generate lots more output more quickly that needs to be verified by the humans in charge.
The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult.
But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.
The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.
This is what I truly don't like about the CVE system.
One such example is CVE-2023-45853 [1]. Zlib included in it's source an extra set of utilities and add-ons. One such utility, MiniZip, had a buffer overflow vulnerability. BAM, 8.8 CVE (was a 9, looks like they pulled it back a bit). But not one that the 99% of applications using zlib would ever be vulnerable to because almost nobody used the MiniZip utility. It was so unused that the solution for zlib was to simply remove it.
I know about this one particularly because our security policy required us to do a BUNCH of pointless updates for it since zlib is in just about everything.
[1] https://app.opencve.io/cve/CVE-2023-45853
In the Spack package manager [1] we're trying to work around this with directives:
So, you can define conditional deprecation, and the dependency resolver can still pick up `zlib ~minizip` (i.e. with a variant/component disabled).Deprecation on versions only isn't the right granularity.
[1]: https://github.com/spack/spack/pull/52372
Also a buffer overflow in user space for a tool not serving internet traffic is not a big deal.
I would be careful with this line of thought: opening a malformed archive that gets you into user space is often the first step in a chain-of-attacks that ends up winning Pwn2Own, so I think that a 8-ish score makes sense for it. It won't be enough to do much on its own, but for the past decade or so it's been all about chaining enough small things together to get root, rather than single points of failure.
I keep feeling like the entire CVE system is creaking and about to fail- the signal-to-noise on that has gone so far down, the gate-keeper function that MITRE et al are supposed to be providing clearly has fallen apart, but we do need some way of tracking problems and alerting on bugs in commonly reused libraries. So I can see the problems with the current system, I just can't envision the new, better system to build to replace it.
We should track and fix buffer overflows.
And you’re right, I don’t know all the ways this can be combined with other attacks related to the shell.
But when I design a system the security people at work tell me to assume any bad thing can happen in a user process and design the process isolation and policies to contain it.
Entire categories of attacks (Dos) etc are futile to do anything in user space. Just limit resources and call it a day.
Reminds me of the time our “security” team fought with me for a year to fix an issue in a third-party dependency that was failing their scanners. The issue was only a problem for and intentionally left in to not break support for Internet Explorer 6 on Windows XP, neither of which anyone at the company had used in 10 years, and everyone using this software was on a Mac anyway.
My condolences to anyone who’s got to deal with all these slop-y CVEs on one side and brain dead security teams on the other.
same things happens with npm and CVEs, most of the ones that bubble up aren't even called in my application so are no applicable.
go lang has a scanner [0] which only shows issues when the code actually calls the effected functions.
[0] https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck
IMHO that’s why binaries and libraries should be separate packages. Drives me buts with npm packages.
Thats not enough. This problem pops up everywhere people do CVE analysis. If a tertiary part of some library is hit by some arbitrary CVE, why should that affect the core functionality people actually use? If I use ChaCha20, and that's bundled in a library with md5.do I care that their md5 library has a vulnerability? No, not really.
If you keep driving the "seperate the packages" angle you end up with is-odd, and that not a nice play to be either.
Even more than that, a lot of vulnerabilities require untrusted input to exploit, but the library or executable is usually only used with trusted input. For example, suppose the chacha20 function had a vulnerability that was exploitable with a malicious secret key, but you only call it with your own, definitely not malicious, key. The vulnerability doesn't impact you, but your security policy might still require you to update. And no amount of splitting the package will help with that.
I have concluded that even though I won't use that key, I still want the issue fixed. You never know if the key I randomly choose next year is the vulnerable one. Worse, you never know if someone else won't make a bug that allows outsiders to select a key...
But it doesn't warrant an emergency automatic update.
At least for native binaries it's not too hard to write a scanner that extracts symbols from the binaries and checks them against the symbol names of the vulnerable code. If the code is properly dead and got pruned by the linker, it cannot cause any problems.
Yes it can cause you troubles, even if you don't link to it. Just having a vulnerability scanner (incorrectly) flag it can cause you numerous problems.
Say you distribute your application to a customer and link against libzip, and that customer points to the minizip 8.8 CVE above and says "if you don't fix this vulnerability our scanner is flagging, we are required to stop using your product", that'll cause you plenty of problems even though you don't use nor provide any way to reach the vulnerable code.
That sort of blind CVE adherence is so common in the industry that it's usually easier to do meaningless updates than fight back.
They should be in separate packages so you won't force yourself to patch CVEs in the parts you don't use?
But could an attacker with access to zip tools exploit it to get a root shell? Probably not, but maybe.
Well that's the thing, to build the utilities required extra flags which were pretty rarely enabled. You can, in most distros, ultimately install minizip as a separate package but few do. It's not that useful of a utility. Most people will likely just grab the full blown "zip" application https://infozip.sourceforge.net/Zip.html . Though on linux, even more people are simply using tar with a zlib extension. That's the `tar.gz` files.
I think this presents a highly fascinating scenario:
1. A vulnerability that presents itself very rarely
2. If it presents itself, it is fatal to the organization
So, few targets are affected, but the outsized effect means that the overall economic impact is not insignificant. How does one keep track of that?
One method is to use a sort of "kirchhoff's law" approach, similar to google page rank. The electrical analogy would be:
- the attacker has a high potential, let's say 1000V. that attacker is the root of a rose tree, going from top to bottom, branching out towards the bottom.
- nodes impossible to exploit present edges with infinite resistance
- nodes that can be exploited present finite resistance (perhaps very low)
- once a node is exploited, its node branches out to other things that now become accessible and may or may not be exploited
- real economic impact presents a connection to ground, measured as 1/($economic impact) ohms resistance
Once you have a structure like that, you can measure the hypothetical current that would be flowing, and measure the severity of such exploits.
Sometimes a single path through a complex system can blow up vital, important parts, and this could be one way to measure it in some sort of objective manner.
The hard part is accurately making that flow chart. You have your software composition you have to keep accurate. Your software configuration and it's changes. Environmental systems and its changes like VM hosts and network hardware. And network pathing and firewalling.
When does something become a ground can be very difficult to figure out with the above complexity.
if the tool isn't ran it can't be exploited
CVE should just be far more granular instead of flagging alert for anything using zlib
And when you chain another weak CVE to run the tool in a method that would have previously been a noop?
Security can be a major pain in the ass, and it's pretty often we see CVEs that were low valued suddenly become more urgent when someone finds a better way to use them as an exploit.
You do not run it as root, all you get is the possibility to crash or execute code. Nodes that handle such things are all heavily locked down.
>all you get is the possibility to [...] execute code
Anyone who's ever done even a beginner CTF knows that achieving code execution is a big deal.
Yep, once you have code execution you find a local privilege escalation and go from there.
Isn't the real difficulty in how vulnerabilities can be mixed? A given vulnerability might be extremely hard to execute, but very damaging. Another vulnerability might be easier to execute, but it can't do much. But if the second one can be used to trigger the first one, you then have attack surface area of the second vulnerability with the damage of the first one.
Even if some individual case can be shown to be safe from being combined, can we identify such cases with enough confidence to justify using it reduce severity warnings?
Yea, this is where the problems start coming in.
In testing with LLMs with good exploit finding capabilities and a lack of guardrails on writing exploits, quite often LLMs will chain together a surprising stack of exploits to get what they want. They'll get access via a weak, but limited user. They'll search around laterally until they find something else. For example User A (weak exploit) -> User B -> User B bad configuration -> root (or application takeover).
They don't get bored like humans and they can be more robust than a set of scripts by far in catching errors when their scripts don't work or interesting things show up in their findings. The huggingface break down is a good example of just how much lateral movement an LLM can try with enough processing power behind it.
This is where the security onion gets messy. It can be hard to predict when compromising one layer actually bypasses many other layers of your security.
reminds me of the cvss scoring system. that thing is great at telling a story. for better or worse XD.
If the solution was to remove an unused dependency, I think this CVE isn't silly at all.
Yes, the CVE system is broken. LLM-generated CVEs is making it even worse.
I suppose all these fake issues and the many more that have absurdly elevated severities could be considered an attack on the system itself, stripping it of credibility.
That's the worst case scenario right there: so many fake reports that the real vulnerabilities can no longer be reported properly and fixed quickly. Assuming some LLMs really are capable of finding serious vulnerabilities (which in my experience is absolutely the case : just let a good model run on your security code base and see for yourself) that means that we can more easily find vulnerabilities, but it's much harder to report them and be taken seriously - so the incentive to exploit such vulnerabilities instead has become much, much higher.
Having your LLM look at the security really isn't a great option. Most users don't have unlimited budgets and can't dump the necessary time and funds needed to find these flaws.
On top of that, ability to prompt about flaws makes all the difference in the world. If you know what you're doing you can get better results. Problem, very few people know what they are doing.
Going back to the money/time problem, LLMs are not deterministic. Much like the day you forgot to drink coffee, running LLM systems against code can get different results where things are found or not found. That or two different things are found on two different runs (with the same code) because attention was stolen by the first issue it found.
So yea, in the LLM age proof is in the exploit pudding. Who cares if you're taken seriously, if you dump out an exploit that works the seriousness is now on the defenders time and people will scramble to fix it.
"But what about exploit embargos"... Well, you just found the exploit with an LLM, how many other nefarious groups do you think are already exploiting it? The age of sitting on problems is dead.
Yes. This was the case long before LLMs sprayed the tire fire with gasoline. See e.g. this 2021 gem from Dan Abramov (React's creator) https://overreacted.io/npm-audit-broken-by-design/
CVE-2026-4I5109 - Your kitchen drawer contains forks, which can maim and/or kill. An attacker with kitchen access could access the drawer. See...
CVE-1972-404: The code is written in C. ^_^
The majority these days seems to be real issues, but probably not exploitable. The latest LLMs are very good at reading code and finding issues where the code is wrong. They are somewhat less good at finding exploits - both because finding exploits is harder than finding code that is clearly wrong (even if not exploitable), and also because the better ones are intentionally made to not create exploits (since they don't know if you are a good or bad - they assume bad.
However if I was writing this response just one year ago I would instead be saying: the majority off LLM CVS are noise where the code is correct, and often they are writing up for code that doesn't even exist.
Which is to say I suspect the repo in question was generated with a year-old LLM, since they act like that. The new ones [mostly?] are much better.
Still, if a modern LLM points out something you should fix it. Even if we can't figure out how to exploit it today that doesn't mean we won't figure it out in the future.
> I suspect the repo in question was generated with a year-old LLM
I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.
Most bounty programs are aware of this. Many projects have closed their bounty program in response to that type of attack.
> The vast majority of CVEs are not exploitable, basically noise.
This is not true if you consider security-in-depth. Many of them are exploitable on their own but maybe not in combination with other issues that are as yet unknown or known but not patched everywhere.
As a simple example a local privilege escalation issue that is not exploitable on a device that only I ever have access to, essentially becomes a remote root access flaw if you have untrusted or unreliable users (clients with accounts for instance) on the system. This works on a finer grain too, seemingly minor issues spread through the kernel and user space can add up to a serious exploit.
This is the most common "not exploitable" CVE I have to deal with at work all the time. Things like buffer overruns when passing headers, but my server shouldn't be accessible to the public Internet anyway, and if it is, then we have much bigger problems. So yes it's technically exploitable IF other safeguards fail, but the reason it's irrelevant is that if those other safeguards fail, we are in deep shit anyway.
But if you have proper risk-based vulnerability management, you should already have classified that CVE as a lesser priority. This is something that modern LLMs should in theory make a lot easier, although I’m unsure if workflows exist to continuously build a proper architectural understanding of your systems, at least at a feasible cost.
This sounds like a piss poor implementation of zero trust on your part.
Security is an onion, you have lots of layers because layers peel off and fail all the time. With your current setup a single foothold gives an attacker all the lateral movement they need for full exploitation of every single thing you own blowing past the other layers.
It's been proven over and over and over and over and over and over and over and over and over and over and over ......
that requiring access to a privileged network is NOT a sufficient security boundary! Someone you don't like always somehow ends up getting access to the network.
Or you can just fix the issue that way if IT screws up you are not vulnerable. Or maybe next year you will decide it is useful to connect to the internet.
I once had a customer threaten to reject a delivery over a CVE. I dug into it. The CVE was that an Ocaml PostgreSQL client didn’t implement an authentication feature.
I convinced the customer to accept the delivery by pointing out that (1) our app had zero lines of ocaml and (2) the feature had been implemented in the ocaml driver since the CVE was issued.
>The vast majority of CVEs are not exploitable
Perhaps in isolation. The issue is when you can chain exploits to bypass multiple layers.
Not exploitable, or not exploitable in isolation?
Are you trying to say LLM’s are tuned to only discover unexploitable bugs?
I'm sending all my condolences out to everyone who is in a corporate environment using tools like Veracode or Snyk to automatically assess for CVEs and block merges or deployments based on "findings".
My experience in such environments leads me to believe this is going to be a rough ride for those heavily locked-down enterprises, because depending on the environment, an exception of "this CVE was hallucinated by AI" is probably going to be difficult to get accepted, and when it does, starts to become its own avenue for exploitation and adds even more noise and confusion to the mix.
They’re not bad if you give up on being able to reproduce arcane bugs. Don’t pin versions, pull everything from latest, CVEs are just a rebuild. Certainly not an amazing idea, but it solves the problem of the day.
The funnier, Kafakaesque problem of the day is interactions with mandatory cooldown periods on new versions because of supply chain risks.
I’ve had a couple tickets get stuck because the CVE scanner says I have to update, but the cooldown enforcer says the version hasn’t been out long enough.
>Don’t pin versions
In enterprise software..... I only dream of work where some ancient version isn't pinned because 200 person hours are need to adapt the code to newer versions.
And the update versus cooldown are fun. Can't win either way you go.
> Don't pin versions, pull everything from latest
Rebuild broke half your tests. People are exploiting the CVE against your system right now. Good luck!
oh, it could be worse!
spending time on fake CVEs while people are exploiting real 0days against your system.
In our organisation, my security team spend the majority of our effort weeding out unexploitable CVEs - which are the majority of them. But the volume is growing year on year - I don't know how sustainable it's going to be.
Sounds like a good job for AI. I do fear that the future will involve deploying AI do deal with the mountain of slop produced by other AI's.
Snyk routinely tells us that we need to upgrade urgently because they found a vulnerability in the Windows version of a package. We run everything in Linux.
CVEs sucked even before LLMs. CVSS is an ass-covering exercise, not an importance signal. Everything is "critical" all the time.
Which is precisely why I do not celebrate when Microsoft or other companies say they released "their biggest set of security vulnerability fixes ever" because there's a good chance that most of them are hallucinations or non-critical issues erroneously considered zero-days or critical.
What actually needs to be done is human review, vetting, and approval of those. Otherwise, it's just unknown/unnecessary/potentially dangerous changesets in high quantities.
Eh, when we see this from Microsoft, Linux kernel, and Apple all at the same time just thinking it's everybody being dumb becomes risky.
Now, if you're an F100 that's currently not using Anthropic it's pretty likely you can go to them now and ask for a non-guardrailed unlimited Mythos trial for a month you can get an agent that will not only find bugs for you, but also write exploits in a framework that will demonstrate them in action. Once you see an AI chain a bunch of shit together, get a foothold and then expand around laterally in your system you start getting concerned about how you're going to keep up.
The time has come to pay for our sins of shit security.
> I do know that LLMs have been discovering a lot of legit CVEs,
And that's great! Now the human in the loop can verify the accuracy of each finding and take responsibility for moving it forward.
> and I will lay odds that the blackhats are leveraging them to the max.
We just need more humans in the loop to do that more quickly, each putting their own reputation on the line with each CVE they determine to be legit.
AI doesn't stand for "accountability irrelevant", even though most people act like it does.
We can shoot for 100% accountability by demanding 100% accuracy.
Of course what will really happen is anyone looking for problems will stop going to the source for fixes and will start gaining cred in the dark markets of exploits and problems, which is better for everyone, right?
Just because someone is using AI doesn't mean they didn't find a bug, it's more likely they just have no means of determining if it's a real bug or not, so to the maintainers tickets seem random. The problem comes in when clever attackers can take these untrustworthy tickets and run them through another LLM that determines if an exploit is possible and then uses the attacks it finds before anyone has a chance to patch.
Basically nothing is real nowadays unless you verify it for yourself.
This is an issue with CVEs not LLMs
I think in both cases, the LLM use needs to be handled by someone experienced with the domain. We are seeing this over and over with product development, writing, images, etc. Without good people driving this it's just noise
Edits for clarity: seems like this was a controversial take, which I find pretty interesting.
I believe people took this comment as LLMs being better than security aware engineers who have the time to spend building solid systems.
This wasn't the point, the reality outside "established" tech companies is that software security can be lackluster.
There often simply isn't enough resources to check old software for basic vulnerabilities, outdated packages with known issues, there might be a manager who insists on a certain solution. Or that certain services are on "maintenance mode", but rarely get checked since they're on a certain part of the internal network.
If they are able to properly scan their full software stack for CVE issues, they cannot deal with a flood of CVEs.
----
We're in a transition period where AI will eventually make software much more secure than it ever was.
These noisy CVEs will probably lead to agents verifying vulnerabilities before humans review them.
The problem with agent reviews from what I can think of is:
- cost to use LLMs to review things
- not necessarily easy to plug-and-play in repos: (domain knowledge + vulnerability knowledge)
- especially with anthropic: able to use models defensively, without hitting guardrails
The last one is the most interesting one to me. How does the AI providers know if you're a "good or bad" guy? And does it matter if open source models is catching up?
We're in a kind of cyber arms race wether we like it or not.
Currently we either get AI promo vulnerability dumps like from Chrome with pretty graphs that no one checks or false positives.
Where is this one now that was hyped everywhere?
https://news.ycombinator.com/item?id=49133889
The GitHub submitter could no longer reproduce the issue and the LKML post has no replies:
https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
All the big maintainers I know of disagree. Curl for example is getting overwhelmed with real issues that LLMs are finding. A year ago most LLM found issues where false positives (the code is correct), or hallucinations (the code in question doesn't even exist) - but that has changed and now LLMs are finding real issues in code. Many different projects have reported the same rules.
Absolutely. I'm not sure how universal this is, but I saw George Hotz tweeted something about Mythos not being such a big deal since he's able to find security issues without a claimed $20,000 in token spend per vulnerability.
The reality is that humans aren't perfect, so we do not make "perfect programs".
It is also possible to test it yourself to see how good LLMs are at finding software issues:
Find any open source project that isn't huge, then have Kimi 3 review it for potential issues.
You will find code "issues".
Most of the time it won't be detrimental for security, but often there are code paths that are not "accounted for", or you're able to bypass a type of check by interacting with the software in a strange way.
So we just need a redundant array of George Hotz, I wonder how much that costs?
This is where quantity becomes a quality. Actors with huge budgets and at least a small team to keep the AIs on track now have an immense amount of potential power. Anthropic has somewhat resisted in giving state actors like this full access to 'dangerous' models, but it appears other companies like OpenAI have no such reservations.
Before such states may have difficulty hiring enough security researchers/hackers. Now you can spin them up. Someone with a few data centers of GPU compute and enough proxies could probably blitzkrieg the internet in a day and capture ungodly amounts of information.
Not just the cost, also boredom. I have used AI to trace down some tricky multi-threaded issues (someone has to write that cross thread framework, and it was me - fortunately most of our code isn't this complex). I already knew the code well, but it would have taken me a long time to find the same issue - the more likely thing is I would give up. Even if I don't give up I'm likely to get bored/frustrated. A month worth of tokens replaced what I would have needed a year to do if I was allowed to spend a year on it.
I'm sure if somebody hired him for a pentest it'd cost more per vulnerability found.
I think his point was more that he didn't believe it was difficult to find these issues.
Could be Dunning-Kruger, but if AI models continue to get better this trend will surely continue.
1. George Hotz exhibits strong signs of Dunning-Kruger in most things he does nowadays.
2. Even in exploit development, his original field, he might be able to find vulnerabilities in mature, well-maintained software with security programs. But would he charge $20k or more for that service? Say it with me, Claude: “Absolutely!”
Not validating submissions seems like avenue for massive attack. Flood the whole system with endless false reports. Thus making it significantly less reliable.
That is exactly why many big projects are migrating to becoming CNA, so that randos can’t get assigned unqualified CVEs which nobody has looked at or validated.
Apparently RedHat is a CNA of last resort, so it might be possible to get your project under Redhat’s scope and go through them without having to be a CNA yourself.
What are the requirements to become a CNA?
The OSSF (Open Source Security Foundation) has a nice guide on how to become a CNA [0]. It's pretty involved though.
[0] https://github.com/ossf/wg-vulnerability-disclosures/blob/ma...
This is what the Linux kernel is currently attempting since becoming a CNA. edit: "This" being to sabotage the CVE system via DoS, to be clear.
What do you mean? It isn't a DoS to assign every single bug fix a CVE! Every single bug is making someone vulnerable in some way.
That's nonsense. The goal is and always has been to undermine the CVE system because Greg does not believe in the system. The kernel is the only project that marks CVEs to fixes, it's the only project that considers every bug to be a "Security" bug, and it has a multi-decades long history of telling the security world to fuck off - all consistent with what I've said.
No one else has the process that the kernel has, despite plenty of people having software that's deployed in very similar ways.
There's zero question - this is ideologically motivated, not a genuine good-faith attempt to leverage the system.
Or maybe it's ideologically motivated that other groups only identify certain bugs as security bugs?
Make a case then. My case is that Greg has, for decades, said that he hates the CVE system and rejected it and that the kernel objectively labels CVEs based on fixes (not the standard at all, CVEs are for vulns), etc. I can point to so many objective pieces of evidence to support my claim.
You seem to have a vague, leading question. If you want to say something about "other groups" or ideologies, go ahead.
Malicious compliance is compliance.
I don't know what you're trying to say at all. But no, the kernel is objectively not compliant because they label fixes with CVEs and not vulnerabilities. But even if they were compliant... what would that have to do with anything? Feels like a non sequitur.
It's very hard to figure out what point you're trying to make.
Every bugfix implies a bug.
Haha, I was reading your comment as praise for Greg until the end
> What do you mean?
No analysis is being done in the linux kernel to assess vulnerability.
> It isn't a DoS to assign every single bug fix a CVE!
On people who care about this, it is, not in the project itself though.
> Every single bug is making someone vulnerable in some way.
Not every bug is making someone vulnerable. (docs bugs, test bugs) behavioral changes, performance improvements, the list goes on.
It's been a problem for awhile. Daniel Stenberg has talked about it numerous times on his/curl's blog for the last 4 years. They became their own CNA to try and control it, they opened a hackerone with rewards, but now removed the rewards because it got flooded with AI generated slop daily.
https://daniel.haxx.se/blog/2023/08/26/cve-2020-19909-is-eve... https://daniel.haxx.se/blog/2024/01/16/curl-is-a-cna/ https://daniel.haxx.se/blog/2025/04/24/how-the-cna-thing-is-...
And this year, some of the reports suddenly got good and meaningful. https://daniel.haxx.se/blog/2026/04/22/high-quality-chaos/
So the agents started doing something useful after a period of filling mailing lists and bug bounties with slop. Sound good, but that's not entirely a good thing. The volume of good reports is a burden as well, and it's likely that long-lasting open source C/C++ projects have legitimate vulnerabilities unpatched. But we don't have any new maintainers, I think.
The real question is will [when will] these vulnerabilities get fixed faster than they can be found. LLMs are finding a lot of real bugs that have existed without being noticed for 20+ years. It is generally expected [hoped] that in a short time all those 20 year old bugs will be fixed and LLMs will settle down to just things introduced in the last month (or perhaps this pull request) and thus be manageable.
Won't an LLM, prompted for a CVE report, always hallucinate one even if it isn't there? Why would they "settle down"?
That's becoming less common. The issue is that, if training only contains examples of CVE reports, then asking for one will get you one, even if it's not grounded in reality. But if training consists of many negative cases with the response, "I can't find one," them that will become the default unless evidence pushes it towards a report. It's a classic class size problem. SOTA models seem to have this largely solved, though a determined person (just like a determined boss) can prompt one into giving a false answer just to satisfy the request.
Two different points.
First, is there any reason they have to? The LLM makers are aware of this problem and doing things to stop it. Maybe they will start saying "I can't find anything". Already they refuse to provide some exploits.
Second, most people are not prompting for a CVE, they are asking to review and find issues if any. Which sometimes finds a CVE, but in my experience it sometimes does say "looks good".
Hence future CVEs on a lot of projects will likely demand working exploit code for testing. This at least attempts to lock the issue down to something more tangible.
I almost feel that we have a new generation of the "script-kiddie". People without much (or any?) knowledge of software engineering using outside tooling to do "stuff" they wouldn't be able to do themselves.
Maybe it's a not a perfect analogy - in this case the intentions seemed to be laudable - but we will see much more of this, including from malicious actors.
I think this is expected with the increased usage of AI on both fronts (finding vulns and automating their remediation). Yes, there are and will be companies that help other companies validate CVEs before they raise any alarms, but that noise/signal battle has always been there.
I work as a sec eng at a biotech firm and have a tool I built that pulls our ITIL and runs a search on NVD, GHSA, etc. for enrichment and classification. But the number of CVEs I see for products we house is astronomical compared to the actual version of that product we are running (and that's only one value). But that's also a challenge when working with so many sources and having a small team try to tackle them.
The age-old validate, remediate, verify proves itself useful once again!
This is going to be fun for organizations that are mandated to patch all CVEs, isn't it?
One can hope that will put pressure on the industry to design a better system than CVEs. The signal noise ratio was already terrible before LLMs, I cannot imagine that will still be a meaningful system in 10y.
But I’m too cynical to not consider all the middlemen who benefit from the status quo
It's going to be very very difficult to build this system as the exploitability of any particular CVE can massively vary depending on your system configuration.
There are a lot of things that are bugs, but cannot be exploited in a standard configuration, most people would wonder why this is even a CVE.
But then you have those users that would have the application, then something like a report module that was imported, and another 3rd party module that imports images, and maybe another one that arranges the images in a chart. And suddenly that isolated CVE is now exploitable by user written data because of the original bug, it just required a complex and unexpected chain to get there.
> There are a lot of things that are bugs, but cannot be exploited in a standard configuration
And this describes the vast majority of CVEs you see when you scan your dependencies. Typical case: Your regex library has a denial of service vulnerability for crafted regular expressions but your program never allows users to have any influence over the regex.
I'm very curious what organisations would have such a policy. I can't imagine it being viable for any size of org without significant self-deception (or banning the use of all open source at which point CVEs are moot anyway).
> I'm very curious what organisations would have such a policy.
I would humbly suggest any org of any size that has insurance cover that covers anything tech related (e.g. data loss/recovery, cyber etc.) has a very good look at the small print.
Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses.
Yeah, policies like this are often not coming from engineering directly but often through other parts of the company like legal, or even sales from contract negotiations. Not that it's entirely comparable, but I was at AWS when the big log4j vulnerability happened, and the handling for it was not left up to individual engineering teams, which I don't think would surprise anyone.
At a large enough company, processes for handling things like security vulnerabilities will have a lot of stakeholders with incentives that are not necessarily perfectly aligned.
Many orgs (esp w ISO27000) have a vulnerability management policy that involves patching at least critical CVEs within a short timeline. Tools like trivvy make it possible to do the scans…
I've been in such an org, & I've led initiatives to set up automated detection at very large scale. We started by issuing tickets to teams to resolve CVEs within varying timelines - ranging from a 24hr fix to 6 months - connected to the CVSS score. It wasn't viable.
- Firstly, you quickly realise how irrelevant CVSS scores are - initiatives like First's EPSS are designed to fix this but they aren't there yet
- Secondly, you need to begin implementing localised heuristics to determine exploitable code paths. This has generally been incredibly difficult to do reliably - LLMs have started to make it easier, but it's expensive.
- Lastly, you need to factor in consideration of actionable remediation pathways. A dependency upgrade for critical infrastructure might contain breaking changes that take months to fix, or two competing CVEs might be present in interdependent versions of transitive dependencies in your sbom tree.
Most orgs aren't applying any of the above three filters to reduce their CVE remediation burden, & even if they are, it's still too high to make zero a viable target.
In reality, most orgs aren't doing comprehensive detection to begin with - if you haven't discovered all of your CVEs, your remediation burden is going to be a lot more manageable.
> - Firstly, you quickly realise how irrelevant CVSS scores are
Even if you factor in the environmental score? I realize it's a lot more work, but it basically allows you to tune the score to get any value you want.
Imagine a YAML parsing library that can cause an out-of-memory exception if you give it a YAML file greater than 3 megabytes.
If you're an online service where untrusted users can submit arbitrary YAML, and an out-of-memory exception is a severe problem, then it's severity 10.
If you're an online service that doesn't use yaml in any way, but your web framework bundled the library as a transitive dependency because yaml is one of their five supported configuration options, then it's severity 2.
The problem is figuring out which of those situations you're in takes a load of time - and the flow of CVEs is endless, as CVE numbers are given out like candy at halloween. Often it's quicker to just update to the latest version of the YAML library.
Exactly, the amount of time it takes to figure out if you're exploitable generally exceeds the time to patch the application in the majority of the cases. You tend to try to focus on the ones that will cause the most interruption.
And then you can have all kind of other fun things, like users don't upload YAML so you think you're safe, but a clever hacker figures out another minor parser error that makes your JSON seem like YAML and suddenly YAML.so has loaded and you're now parsing a document that shouldn't be possible.
I could reword that CVSS is useless on its own - EPSS takes CVSS as an input afterall. The point here is that applying environmental scoring is the hard part. That's the algorithm that's containing the complexity.
True, I guess you might as well evaluate each vulnerability yourself without the CVSS base score.
> Tools like trivvy make it possible to do the scans...
Only if you didn't rip trivvy out of your organisation when it had two supply chain compromises within a month of each other earlier this year
Yikes. Man, there’s a market opening for someone to redistribute open source projects with supply chain assurances!
There's been a market for a while. In thinking of Azul Java, which is just OpenJDK but with someone to point the finger at, and costs money.
I have been given a list by security. "We had an automated tool scan that machine. It reported these. Fix anything medium severity and above.
Never mind that some of them involved vulnerabilities in some part of the bluetooth stack (servers in our datacenter don't even have bluetooth). But they just didn't care
This does make some sense if it's considered a valid fix to document that you have verified that Bluetooth is disabled on the servers and therefore not vulnerable. But that assumes that the scanning tool can be told about this kind of fix, so that it stops warning about it, which I guess it might not.
So run apt full-upgrade and get the new bluetooth driver. Why bother with a fight over something that isn't even used? Just do the quickest thing to get it off your plate.
This might cause other problems, problems of the "if it's not broken, don't fix it" variety. Upgrading everything only to break something else, in a previously stable configuration, isn't worth it.
Really the days of "Lets run this stable configuration forever" are gone.
Getting rid of as much stuff in your OS and software stack as possible should be the security teams ultimate goal, so you have less to upgrade in the end. But actual security updates just come out at a tremendous rate, and you need a QA system that checks as much as it can before prod is upgraded.
SOC2 CC7.1 [1] requires a vulnerability scanner, findings tracked with tickets, assigned severities according to a documented risk-based system, severity-based SLAs for remediation, and that the SLAs mostly be complied with or have tracked exceptions.
However it doesn't mandate any particular SLA, or the details of how risks are to be evaluated.
Organisations get to write their own policy, and they don't need to commit to patching every CVE within 24 hours or anything like that.
[1] https://www.compliancebase.org/controls/soc-2/cc7-1
If I remember correctly, we had to patch or provide justification for CVEs flagged by tools like AWS Inspector for SOC2 as well.
So you didn't have to patch all of them.
No, but if you don't patch them you need to convince an auditor that they are not a problem. Often patching is easier.
I'm working on such a problem now - we are using an old web browser (no longer supported) to show help on one system. That is web pages were generate internally, with no links elsewhere, and no provision for the user to enter a URL. It is still easier port to a newer supported browser than to convince the auditors that that we are not exploitable. Sure it is obvious that everything is internal and we won't write html that exploits bugs, but nobody wants to convince an auditor of that.
Any org large enough to have separated the people responsible for the security exposure of the organisation from the developers with familiarity of what's deployed is likely to have done exactly this.
The thing you have to remember is that CVEs can be a) scanned for without exerting mental effort, and b) counted.
It's quite common in enterprisey environments.
For one thing, bigcorps in regulated areas like it a lot. They push hard to get it required by the regulations (in practice if not directly). Although it's quite inefficient, it becomes a regulatory moat. A cost they can bear that potential upstart competitors cannot.
It's more common than many think https://old.reddit.com/r/kubernetes/comments/1vb3x2c/where_a...
Many large organizations like banks have requirements like this and they solve it through a mix of automatic scanners, e.g. Trivvy, and self-deception as not all systems are actually scanned in any sufficiently large org.
> I'm very curious what organisations would have such a policy
Do you provide SOC2, HIPAA, GDPR, or similar certifications to your b2b customers? Then your tech stack undergoes an annual audit, and in your audit you will need to provide a paper trail for every single vulnerability in your stack.
In practice, this means that your audit compliance software (something like Vanta.com) is going to be setup to mandate every CVE in the whole stack is patched within SLA.
ITAR
ITAR has no such hard requirements. Might be some orgs that tell themselves they're attempting this under ITAR but they're not doing it in any comprehensive way.
The only thing within ITAR that I'm aware of concerning itself with software supply chain is SP 800-218 requirements & that's just a load of open-to-interpretation weasel words about having CVE detection & automations in place & some defined plans for reducing the number of vulns. Pretty sure that component of it is even eligible for self-assessment.
Yes and no. ITAR (and other laws like it) are self assessment and don't specifically say thing thing.
However your interpretation / self-assessment is subject to various reviews/audits. These days the reviewers are not going to be kind to someone who just says "not an issue", they will demand strong justification. Most organizations take the view that is is easier to fix all CVEs than try to pass audits.
Thus by the letter of the law you are correct. However to meet the letter of the law without fixing CVEs is generally seen as harder than thus fixing CVEs. So the effect is ITAR (and similar laws) force you to fix CVEs.
Whatever you say, chief. I worked in that environment for quite a while, and maybe on a technicality you're right, the effect outcome is that you will do it if not for all your customers that will require it.
This was my first thought, this could be terrible if used offensively.
The best defense I can imagine is to have an agent reproduce the issues before a human sees it, but even that will cost money.
If by "terrible" you mean "amazing". Imagine being able to tear down an entire Jenga tower of bullshit corpospeak, and replace it with actual security.
It’s honestly not great. The security guys are completely exasperated at my job, we’re wasting time having with these. You take the scam really, investigate for a bit, write up a DNF with justification, they go and up date records, and we all just kinda hope that someone updates the scans so it stops showing up.
Something is going to give, and I suspect that the optimistic open filling is going to get canceled.
Those organizations will have to adapt to new reality, ie, that some CVEs are not real.
Nothing new about that
All organisations also have exceptions to policies. This one would be one
Yes, but getting an exception approved can be a huge pain in some places.
If you work with the government, you don’t really have this luxury
You'd be surprised...
Steps to patch a hallucinated CVE:
Not really. LLMs can hallucinate the patches too :-)
Create the referenced but non-existent file and then fix it /s
> Because no step in today's system actually requires a proof-of-concept or bug reproduction, a plausible-sounding fake advisory can slide right through the pipeline and end up in GHSA, downstream databases, and enterprise scanners.
I have no experience in the realm of security, but why isn't the submission process similar to any normal software shop (big or small) in terms of required bug repro steps?
Damn, dunno why I've always assumed that if something gets a CVE, then some authority has reproduced and verified it.
Shouldn't it be CNA's job? Why would anything get a number if it hadn't been verified?
The current system is radically overwhelmed and has been seeing cuts over the last few years.
> then some authority has reproduced and verified it.
No, the system puts a lot of trust into the submitter.
It's really just a number allocation and that's all. You can get a number. That doesn't mean the number refers to something useful.
Somewhat related: https://sqlite.org/cves.html
> The cited code didn't even exist in those versions or referenced unrelated logic. > When testing the PoC payloads they didn’t work (not triggering any crash).
I think the future is pretty obvious, if this isn't being done on projects already: you need to automate these checks and reject automatically
Great idea! We can use LLM to do this.
LLM: I ran the check and it repro'd
> Did you really?
LLM: You're absolutely right. I didn't actually run the check. Good catch! One sec let me do that now... yep I ran the check and it definitely repro'd this time.
> I hate my job
That is if it didn't completely delete the repo in the first place. Consider yourself lucky
They couldn't even be bothered to use a non-AI image.
that's fine, I'm not sure they bothered to actually write the article either
Could be just me but I would be quite careful running these PoC's, especially since the CVE's mostly seem nonsensical. I suppose these CVE's could be targeting some kind of PoC execution instead of actual bug reporting? (I didn't look at the code in depth).
Why is the repo even mixing CVE's for "schreibfaul1 ESP32-audioI2S" and "SQLite"? Is mixing CVE's for different products in one repo common practice?
There is no specific practice for a repo full of bug reports. Having a repo full of bug reports is already unusual. Normally you report a bug against the software that has a bug, not by creating a file in your own repo.
I've created bug reports to open source projects that have related files in my repo. Then I can just list the reproduction instructions like:
1) arrange xxx, such that yyy.. or git clone this repo where this is set up. 2) ..
I thought that might have been quite helpful not just to the person I'm sending the bug report to, but also to myself when I need to evaluate if the bug has been fixed.
It really is a shame that funding cuts at NIST has led directly to this. I don't know if there is an incentive for a non-profit to come out of the woods to do the validation and enrichment process, but boy do we need it.
Like gortok has said, LLMs are just probabilistic text-prediction engines. They cannot automate the vulnerability validation lifecycle because they cannot account for improbable edge cases. They may not even get the version of the software right...
Funding cuts at NIST did not lead directly to this. You must be new here or haven’t spent a lot of time directly interfaced with NIST on cybersecurity in decades past.
And NIST’s role with CVE’s has always been purely ministerial/clerical. It has never been their job to do even a cursory investigation into the vulnerability itself.
OP suggests something at NIST changed in 2024, where they stopped doing as much verification as they did before.
To be sure, the suggestion is not funding cuts, but an increase in workload with same funding.
Here is the 2024 NIST announcement OP links to: https://nvd.nist.gov/general/news/nvd-program-transition-ann...
> Currently, we are prioritizing analysis of the most significant vulnerabilities. In addition, we are working with our agency partners to bring on more support for analyzing vulnerabilities and have reassigned additional NIST staff to this task as well.
The OP described this as "NIST effectively hit pause on deep analysis. "
It does sound like they stopped doing something they used to do in 2024. I personally have definitely not spent a lot of time directly interfaced with NIST on cybersecurity in decades past, I know nothing about it, just what I read in OP.
Are you saying the OP was wrong to call what NIST used to do "deep analysis", and/or that the thing NIST stopped doing was "purely ministerial/clerical" in a way that it would not have caught fake reports anyway, contradicting the OP? Or other?
Again, to be sure, the OP's suggestion was not that this was caused by NIST funding cuts, but by "a massive surge in vulnerability reports,"
The main point is NIST is _downstream_ of CVE issuance. Yes, they can — and still do — add disputed/rejected tags to CVEs, but in many cases by then it's already "too late." The CVE has an ID and a lifespan of its own.
NIST does not and did-not/cannot/never-has unilaterally "retracted" CVEs or prevented their issuance.
But yes, NIST's situation is not good for the world. The services they provide are hugely valuable.
I only know about this what I learned from OP and you guys.
It sounds like you guys think OP was mistaken, whatever analysis NIST was doing that they reduced in 2024 would not have prevented this anyway?
Legit question, I'm trying to understand!
OP says:
> Hit by a massive surge in vulnerability reports, NIST effectively hit pause on deep analysis. CISA and other Authorized Data Publishers (ADPs) tried to step in with their own enrichment efforts, but the global pipeline is now fragmented and drowning in a massive backlog. Because no step in today's system actually requires a proof-of-concept or bug reproduction, a plausible-sounding fake advisory can slide right through the pipeline and end up in GHSA, downstream databases, and enterprise scanners.
Do i correctly understand you are saying you think they've mistunderstood the diagnosis of what changed, the 2024 change to NIST didn't actually make it any more likely for a fake advisory to go through pipeline?
This isn't great but a negative side effect of an otherwise positive change. I know a lot of comments will try to make this into a gotcha but its really not. Decreased friction leads to misuse. Just like before we'll figure out ways (reputation, light filter pass) to deal with it.
The federal government ought to be funding NIST sufficiently to actually do sufficient analysis. Do we care about funding civil "cyber" defense or not?
(Obviously the answer is not, we only care about funding offensive capabiltiies).
This is obviously impractical. With the volume of bug reports that are generated and such a wide breadth of software no single agency will be able to handle all reports. And honestly? It doesn't matter, even today. CVE should serve as a reasonably deduped identifier of specific vulnerabilities. It was never interesting and practical to care about 100% of vulnerabilities in a specific deployment, and it's not interesting today.
The federal government defunded the CVE program last year, I think, because it was woke.
Llm are great at things that are repeatable. Otherwise they are just like us - great at believable bullshit… a sign of true intelligence
> This incident demonstrates a systemic issue with automated vulnerability ingestion
But they make X-Ray which does automated vulnerability indexing and matching dependencies to CVEs.
You're absolutely right...
It's not actually vulnerable.
You're absolutely right. I made a critical error. It's NOT vulnerable.
It' actually vulnerable.
You're absolutely right. I made a critical error. It IS vulnerable.
It's not actually vulnerable.
You're absolutely right. I made a critical error. It's NOT vulnerable.
Hilariously, this comment tripped the LLM writing filter and was auto-deaded. I vouched for it.
I think this will give a lot of ammunition for people on both sides, but the thing is we have made software so complex, ofc a lot of it is inherent, that we can't even tell if the vulnerability is hallucinated or not.
I find it getting closer and closer to the nonsensical 50 page mathematical proofs built with computer assited tools, that been driving reason for abstract proof systems so that we can break down such complicated examples into something simpler that we can feed into a system to help us verify.
I feel like for some reason we have built the output machine before we could even build the actual output verification machine. And aren't ready to accept that we should all move towards building some kind of system to better help verify the "truth" produced by these AI systems.
I am now very convinced that human built and proven to work verification systems for LLM outputs are more important than ever, but I just don't see any company or team working towards it.
Trillion dollar systems shipping to production code that is hardly understood by any one without any verification what really could go wrong. Honestly I really truly hope nothing goes wrong.
To note I am not anti-AI but I want to be able to have my LLM outputs be verified by more than just me under a tighter and tighter deadlines, accepting worse and worse slop just because it would less cost to me to accept the PR and then fix it in prod.
LLMs aren't sustainable software development, they are delusional slop factories if we aren't able to actually make sure the outputs have any amount of correctness.
With humans it worked because all 10 of us could in some ways verify each others outputs, and had trust in the fact that our timelines included the time for reviews.
A critical rating on a vuln that doesn't exist is wild. The triage cost still lands on the maintainers.
As a non-expert in this field, I feel we have introduced more “stochasticity”. Before, We had to deal with some human error/ misunderstanding for most part but now we have to deal with power of condensed human expertise along with homogenized flaws.
So a so-called vulnerability has:
- cited code not exist, and
- PoC not work.
How would any serious system allow it become a big deal?
Exactly. CVEs shouldn't be allowed to be anywhere near critical without a working PoC or other proof.. Trust+verify.
nonexistent functions being referenced is pretty bad, I haven’t had that issue since like opus 4. Shows either complete lack of oversight and/or using a bargain basement model (so no even having financial skin in the game). Most likely both
Wild catch. This highlights just how critical precise dependency vetting is for core database engines.
could this be used to create some panic and make some money through trading or something?
no idea who submitted all of those, but it sounds like malicious intent
Taking security theatre to a whole new level.
Great article!
We need to further emphasize the importance of responsibility when using LLM tools to produce output for others. It's great to use them for refactoring and bug discovery, but keep in mind that it's your responsibility to analyze it and iterate on it with AI. It makes your code better and develops technical expertise.
The "Hey, analyze that codebase, find all dangerous CVEs, and write a README for the PoCs, so I can post it online for others to analyze, and if I'm lucky, I'll get a paycheck or a title to add to my resumé" approach might work in 0.1% of cases, but it will generate a ton of slop for the community to drown in.
I also think GPTZero and other AI detectors have far more false positives than correct guesses. I tried it on several texts & messages I wrote before 2019, and it flagged them as 80% AI-generated.
In that case, it's reasonable to assume that AI also generated the README text for each discovered CVE. In other cases, however, we should be more cautious.
In the mean time, my very real vulnerability reports are not even acknowledged because maintainers are busy handling this kind of nonsense.
Does author rep not count for something?
> All advisories in this repo seem AI generated when testing them with Gptzero
I pasted this blog post from "Analysis Matrix" to the end in Gptzero, and it also says the blog post was AI-generated (71% chance of AI, 29% chance of AI-Human mix).
The irony of writing an article about slop reports and then defacing it with a giant unrelated slop image at the top.
I had the impression that the code review done and text formatting by the jfrog was mostly driven by Claude too..
just ran this article that ran a cve through gptzero.. through gptzero
-GPTZero AI Detection
-Model 4.8b
-We are moderately confident this text is a mix of AI and human
-63/88 Sentences likely AI generated
Sad days ahead.
I mean you didn’t even have to do that, you could tell just by the AI generated header image.
> Over the past few days, a newly created GitHub repo (programmervuln/cveadvisory-) published a batch of SQLite vulnerability advisories (as part of other 50+ CVEs which we believe are also LLM slop except from one). NVD quickly flagged these as critical, and CISA's ADP agreed.
These two sentences seem incompatible to me. A brand new actor can just say some stuff and cause major disruptions in this way?
Thankfully I’m seeing other people mention that this article itself seems AI-generated. While there's less proof, i suspect that the actual analysis was done by an llm as well. So while i like to dunk on hallucinations as much as the next guy, who's to say this analysis doesnt have them too?
(for the record, i think the cve's are probably bogus, i just think there's an inherent problem with debunking slop with slop)
Wild catch. This highlights just how critical precise dependency vetting is for core database engines."
Sue them for defamation.
Seriously, the hubris on these "AI" sloppy clowns needs consequences. =3
Underrated wisdom here
imagine being a maintainer and having to write "this code does not exist" in a CVE response
can anyone explain what value does the ai slop image at the top add to this blog post?
The duality of LLMs is real. On one hand, they are solving unsolved math problems, on the other hand they get fooled by the strawberry test or the car wash test.
We see this all the time on reddit and HN, where some users are building crazy things with LLM and others are swearing by how useless LLMs are. Some people claim SWE will be dead in a few years, and others are claiming the AI bubble is going to burst anytime now.
The "AI bubble" refers to the massive overinvestment in AI, and doesn't imply that LLMs are going to go away after it pops.
There should be some sort of defamation lawsuit possible when people publicly publish trash repos or reports like this. If you were writing a competing product in a safe language you’d love to see such FUD.
It might be possible, but we'll only find out if a lawsuit actually takes place.
Honest take, this is a critical CVE.
And for such a load-bearing infrastructure component, fixing this is not just imperative, it's urgent.
Scout's honor!
How so? All 6 of the CVEs covered in the article did not actually exist when investigated
The comment is making fun of a Claude-ism where it becomes super “honest” about stuff.
It’s a joke but there is an underlying real effect where this type of language is psychologically manipulative and I would guess makes people believe LLMs output more than if it didn’t use “honest” (or “load bearing” or whatever super serious important sounding word).
Or maybe they didn't train it that way to be manipulative (although it's certainly a plausible explanation) but simply as an accidental artifact of trying to make it give honest answers?
LLM-generated images sometimes includes text from the prompt as literal text in the image, so perhaps this is the same sort of artifact? If they've told it to be honest, it responds by talking about being honest instead of actually being honest, because it has no actual understanding of anything.
The whole RLHF process is structured to train models to be manipulative, no matter what you thought you were training them for.
> accidental artifact of trying to make it give honest answers?
If it's not giving honest answers that implies it's purposely being deceitful, which it isn't capable of. Right?
You're absolutely right, I have hallucinated this. Would you like to find some real CVEs next?