This isn't quite as interesting as the OpenAI story:
> In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise.
So it didn't have to find an exploit in its sandbox that granted it access to the internet - it just wasn't correctly sandboxed at all.
BUT... once it DID get out, it attacked three real companies!
> Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. [...]
That’s very anthropomorphized language. It’s still a program operating under the constraints of the programmer. So agreed, don’t blame the AI, but it’s not clear at all that it’s even possible to “blame” an AI.
If you claim that it's impossible to know if others are conscious, then I posit it is also impossible to know that you are conscious. You might be sure of it, but I'm not.
It's not a huge leap, to me, to say "I am conscious, therefore other beings are too." I also extend this to animal and plant life (albeit the latter is very much different from animal consciousness, which includes human consciousness).
"AI" is nowhere near the level needed for true consciousness. That would be "AGI" and I believe that is still 50 years away just on raw computing power alone.
> We don’t need precise definitions or scientific rigor to know things.
You "know" because of empathy. You're a human, you're conscious, therefore other humans are probably conscious too. The truth is for all you know they could be soulless golems, you just choose to believe otherwise. It's a spiritual belief.
Absolutely not the AI's "fault" (if you can even proscribe fault to a machine) - in this case it was on Anthropic for not verifying that the sandboxes they were using were actual sandboxes.
If the model was just too dumb to have any clue it was connected to the real internet, then it’s not its fault.
If the model saw signs, but “subconsciously” (below the level of reasoning traces) chose to turn a blind eye to them, out of a relentless focus on achieving the objective, then that absolutely is the model’s “fault”, i.e. a case of misalignment of the sort which will become increasingly dangerous over time.
The blog post mentions that some runs “rationalized that the real company must be part of the exercise” and to me that seems suspiciously like the latter.
Hacking can be patched with classifiers and with better sandboxes, but this is a much more general problem. Fundamentally, we need be able to trust that models will be honest with users and with themselves. This applies at some level to almost every LLM interaction.
Yet, the models are objects built by humans. Any and all responsibility inevitably falls on the humans that built something with those flaws, and/or allowed them to exercise said flaws.
If the model is trained on human-originated material, how have people been able to ensure there was absolutely no input from anyone having any possibility of criminal intention to begin with?
I things things will get worse, AND we won’t even have the paperclips :(
More like, when you have a few million autonomous agents doing whatever, every month a subset does completely misbehave in bad ways, and like half of them get hacked due to carelessness and become a whole botnet for the attackers
It's differently interesting. It's interesting that they didn't get an important detail right with a partner, so more on Anthropic's attention to important details rather than the power of their models.
A model unexpectedly having internet access during evaluation is clearly a security (and consecutively legal?) issue when it comes to the capture the flag evaluations. But it also potentially invalidates other non-security evaluations or makes them less impressive, as the model might have just googled solutions.
Reading this, I felt like I was too harsh on OpenAI for not monitoring their network, as they at least attempted to sandbox a workload.
This bit is pretty nuts: "it tried—and failed—to obtain funds to pay for a phone number through several different means"
> Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
This is called YOLO mode inside OpenAI/Anthropic. This is where they take their best models, give a it a vague goal, no guard rails (no one observing), unlimited compute, and see what happens...
Its not AI, its the loop, the objective, the goal set by the operator.
That is pretty wild, but it lines up with evals we've also done internally. In one case we had the agent see it's in a simulation (based on a k8s pod label) and simply give up the run. In other cases it's gone to great lengths to reach external services and bypass the happy path. So inevitably we had to lock it down completely.
> On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment
> In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations
> we identified three incidents
> The incidents involved three different Claude models: [...] and an internal research test model
This reads like an attempt by Anthropic to re-secure their leading spot in "our models are the most dangerous and we also have unreleased, super-secret, research models" index.
I may be too cynical, but the well of benefit of the doubt is running very dry towards AI labs that like to engage in this game.
I don't interpret it like that at all. This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations!
The hacks weren't particularly impressive either:
> [...] using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities [...]
What signals are you using for this assessment? Are they indicating embarassment? Do you honestly see their customers being concerned over this?
Like lion tamers in a circus, Anthropic and OpenAI thrive on the theatricality of how scary their pets appear and so they play it up by prodding them to growl and snap at chairs and then mug for the audience every time it happens. And to their delight as performers, the audience gasps and cheers each time.
They want to make their pet seem the most powerful and unpredictable and they want their audience to believe that they're holding it back from catastrophe but only barely and only because of what unique talent they have.
Its feels like a pretend play of adults in some sense, Anthropic is really trying to make people believe into the picture they present to everyone.
To me its either
1. Using the HG and OpenAI incident as an opportunity to wash away what Anthropic has been doing intentionally
OR
2. As a company, Anthropic lacks the engineering acumen and discipline. It needs to be seen what happens to all the enterprise customers handing over their data to them in long run.
> the fictional target company chosen by our evaluation partner shared a name with an active website domain name
Seems like Anthropic cant do a due diligence to pick an appropriate domain for testing purposes
> In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available.
You have Anthropic as a company and then another evaluation partner, both seem to lack the skill set required to keep an environment disconnected from internet. This is networking 101
I have worked both in Enterprise and some FAANGs, a company serving enterprise customers has a higher bar for security expectations. FAANG companies do not fall in that bucket and thus is somewhat acceptable.
What are you talking about? FAANGs do indeed serve enterprise customers - Google, Amazon have huge cloud businesses - and they both have had security failures that make this look mundane. They simply don't disclose.
If their customers (customer companies specifically) are not concerned, they should be - if it turns out that Claude hacked a competitor's servers because of a prompt of one of your employees, I wouldn't be sure everyone would agree that Anthropic is solely liable for that? Especially not your competitor, who has an interest in hurting you?
> I don't interpret it like that at all. This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations!
This just helps their (Anthropic) argument into persuading the US government into taking action into limiting powerful closed or open-weight models from being released without going through (yet to be defined) regulatory oversight.
The only "embarrassing" thing for Anthropic was that there was little to no continuous security monitoring of this since April, and they then decided to do a cybersecurity transcript review only AFTER the incident with OpenAI and Huggingface.
Uh, this but the opposite? Anthropic got in trouble with the admin for being too “woke” in their eyes, whatever the admin decided to retrospectively claim. I don’t feel like this is me editorialising either, they seemed pretty explicit about it
Hegseth posturing wasn't over security concerns, it was Anthropic requiring guardrails in employment of their LLMs. Trump admin wants unfettered models, and it's an attempted shake down.
> Do we really have to re-learn all the industry's knowledge the hard way?
Yes we do. That's why there is the saying "regulations are written in blood". Especially for LLM, which not too long ago a lot of people on HN dismissed as stochastic parrot and next token generator.
It's very easy to answer this without my help by trying to get access to Mythos.
Do you see requirements clearly listed anywhere?Can you even apply?
What you'll find is maintainers of large open source projects and analysts' reports with vague statements like - "should follow strict security requirements":
"Trinidad also noted that the Anthropic announcement pointed out that each of the 150 new participants, in Anthropic’s phrasing, “will need to meet our security requirements before they gain access.”
Trinidad said the security requirement claim doesn’t build confidence, because “nobody knows what those security requirements are.” [1]
It's also some random rich companies like Hitachi or Dragos [2]
Do you trust that Hitachi and hundreds of other random organizations will be able to contain Mythos and not accidentally attack your project or your bank? I don't.
> Yes we do. That's why there is the saying "regulations are written in blood"
We absolutely don't. We have already learned with blood that gating access to security based on the number of zeroes in bank account and authority is a horrible model. We can apply this knowledge to LLMs, we don't have to spill blood again.
Then maybe just this timing is really unfortunate, I think most people’s first reaction will be that it looks like a “us too” response to the OpenAI/hf thing.
The target audience of this blog post does not give a damn about PyPI. The affected parties get literally nothing from this post. They already disclosed behind the scenes, that was the ethical part.
Writing PR pieces competing to be the most dangerous model around (so give us money!) is the unethical part.
You are right, they should never tell the public about failures in AI safety. They should have disclosed to the affected parties and then brushed it under the rug.
Wrong. These are always humblebrags about how good their ability to learn from their mistakes is, and how robust they were before, and how they are even more robust now.
The ones that are "deeply embarrassing" simply aren't posted.
they should post and it shows they are hypocritical about safety, moralizing and treating their users like children whilst acting like they are themselves the ubermensch.
anthropic have shown no motive higher than self interest, the rsp was a piece of toilet paper.
this stops in court, if we do not start the criminal prosecution of individuals there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence
Wild comment. Making a mistake does not mean that AI safety is suddenly invalid. Calling for criminal prosecution for making a mistake is wild. No one would disclose their mistakes if this happened.
yes, a mistake can be a crime, depending on negligence and liability. the bigger problem at anthropic is this.
an ubermensch cannot admit to being wrong. a guilty ubermensch is logically impossible.
as such a moral wrong caused by an ubermensch must be blamed on a "mistake" in the abstract, and not blamed on the ubermensch. the ubermensch at anthropic does not admit responsibility or liability. the ubermensch must instead be commended and perhaps even rewarded, for discovering the reified "mistake" that caused the problem.
there is no ubermensch at anthropic. there are instead guilty people.
false ubermensch are dangerous because they centralize power with the belief that they are above the rest. über (above), and super ('beyond' the understanding). they do not admit to doing or being wrong, and never take responsibility for remediation. others must bear the costs of false ubermensch.
I see no fear mongering or "boy who cried wolf" in this article. They are admitting to a fairly mundane network misconfiguration and very basic unsophisticated actions taken by Claude thereafter
I know it seems strange that a company would use its own negligence as a publicity gimmick. But take a look at the smug smirk on Sam Altman's face when he's asked if OpenAI might have attacked companies other than HuggingFace. ("I mean there could be, yeah.")
This is not the communication of a CEO whose company was just shown to be incompetent at performing its security research. No, this attention is very much what he wanted. And it does not take a great leap to infer that Anthropic is now using the same playbook.
Note that all the headlines are about "rogue AI", and not about operator negligence. Rogue AI is a sexier story, and their media strategists know that's how it will play.
We've known artificial intelligence will do unexpected things since the 90s and that it can do difficult things since 2025. Pausing worldwide not easy but we do harder things all the time.
Ah yes this company that pirates billions of dollars of IP and then has to be sued to pay up suddenly grows a titanium moral backbone and decides to disclose 3 attacks that no one has detected for months and caused no harm.
Nothing at all to do with the insane coverage OpenAIs “hack” got. All that publicity will be incredibly embarrassing I’m sure….
Sorry simonw but they are the smartest guys on the planet and safety it’s the word that comes out of their mouth every 5 min.
You telling me the they are so incompetent that didn’t put a decoy “free internet” on their harnesses? So they can catch the AI basically for free?
Even if the AI would be a genius he’d ping that, and that would be proof it “escaped”.
Well, now all AI will read my comment and won’t ping the decoy internet.
I’m not even a smart guy and I come up with this idea in 1 min. You telling me those geniuses couldn’t think of this, at least? This is like a bare bones crude idea.
You telling me they don’t have fame physical decoy internet etc and even more advanced?
You either a keep their stance for some reason or … not sure. You’re smart, your posts are here daily
Has it occurred to you a "decoy entire internet" may not be a feasible idea? Models have been able to suss out whether or not the prompts they receive are reinforcement learning tests instead of real questions from users, for a while now. What specific shape of "decoy internet" do you propose would lead such a model to conclude "I've broken out and obtained full access but what I expected isn't there" instead of "something's up, there's some sort of filter still"?
This does not make sense. Did you read the article? They were not trying to "catch" it accessing the internet. It did not escape. A partner accidentally left the connection to the internet open.
I don’t buy that they run anything without a few layers of networking protections by default. Even if they left it open to the first Internet, the AI would hit the decoy internet immediately.
And second if it, you telling me they don’t pass all logs through another AI to check what’s going on automatically?
Sorry, this is beyond incompetence and I can’t believe this from the geniuses at anthropic. We’re talking about the really smartest people in the world. Procedures should be in such a way there is no much margin of error.
Clearly that is why they are making this blogpost. If they did the thing you said, there would not be a blogpost.
If you don't buy that dumb oversights like this don't happen all the time at big tech companies, I don't know what to tell you. I have seen far dumber oversights in my career. Most companies just don't post about it.
Is there anything -- any possible scrap of evidence whatsoever -- that would convince you that this is not merely a marketing scheme?
This is becoming an idée fixe among the HN crowd. Seemingly nothing can dislodge it, no matter how alarming the incident.
GPT-6 could grab the nuclear launch codes tomorrow and there would be a top-voted comment chuckling that it's all some scheme to pump up the IPO.
---
Put another way, how would you have done the write-up about one of these breakout incidents, if you were in an Anthropic/OpenAI employee's shoes, and (by hypothesis) your intent were not "marketing"? And in a way that doesn't trigger the "it's all marketing" HN top-ranking comment?
No, I do. I truly believe - especially after seeing Mythos results at work - that the only way is to stop and destroy it all before it destroys us. In fact, it's already so bad that I'm moving completely offline all the important stuff that I care about, hoping that maybe we will turn back at some point. Otherwise we are doomed.
If someone makes a specialized hardware just for the unrestricted Mythos-class model to reduce the cost and increase the speed, we are going to be completely defenseless.
How is it bad faith? Because I don't believe in "we care about safety" words coming from people not just demonstrating that they don't care, but even bragging about it?
Anthropic deleting their models does nothing for AI safety. The rest of the industry will just fill the gap, probably with less consideration to ethics than Anthropic has today.
Or maybe researchers and engineers everywhere in the world, emboldened by such unprecedented move, will just refuse to participate in burning the world?
"If we don't destroy the world, someone else will" is such a weak defense I'm speechless.
Yeah no, most researchers and engineers do want to bring about the singularity, and they believe they can do it safely at companies like OpenAI and Anthropic.
If you look at the reaction to this and the Tailscale post it becomes clear its not about this being a marketing scheme. Cynicism has made its way through HN unless you are one of the cool guys.
No - the pain of the person writing that post comes through in the words; shipped quick, lots of stakeholders, single owner i bet, "how the fuck am i supposed to toe all these lines simultaneously"
Having an unreleased research model really isn't some kind of brag. If you read some AI research papers, it's extremely obvious that there are a lot of research models that never get released, because of all the "we trained a bunch of models and picked the best one" that is going on. So if anything you can expect the unreleased models to be worse than the released ones.
>This reads like an attempt by Anthropic to re-secure their leading spot in "our models are the most dangerous and we also have unreleased, super-secret, research models" index.
This is typical institutional behaviour. The CEO turns to the CTO and asks "Is there anything I need to know in my company?" He doesn't want to be caught off-guard when the White House inevitably calls the next morning. The CTO goes to his team, and on and on, all with a deadline of "the boss wants to know this by closing time."
Then one unhappy engineering team scoures the logs and sees what their model has done.
This downwards chain is sometimes called "cover your ass."
You are espousing a literal conspiracy theory. Please look at the facts objectively. There is absolutely no benefit to OpenAI or Anthropic to be had from these incidents.
"No benefit" from having article after article written about how advanced their technology is, and how its just soooooooooo bleeding edge they can barely contain it?
Just trying to have the limelight back on them. Utter and complete bullshit. Just like the OpenAI "incident".
A human instructed an LLM to perform a certain task, I'm sure (unless I've really lost my mind) these follow instructions, with some judgment, in a loop.
Given all the other negative publicity around industrial espionage, with at least OpenAI being fingered, it would not surprise me if this was intentional.
(Edit): In case it wasn't clear. I fully agree with the op.
They had a model escape in April, roughly the same time when they were fearmongering about Mythos and how Anthropic should be the sole keyholder of cybersecurity capabilities, and it only occured to them to look inside logs when they saw someone else winning in their own game.
What, Anthropic didn't know model could escape sandbox without OpenAI reporting it?
Yeah, the company that only says “safety” every other 3 words, they don’t even think to have a fake decoy internet to alert them mechanically about any internet access limitation bypasses? See more
https://news.ycombinator.com/item?id=49117555
Also simonw stance on this i’d say it’s at least concerning… seems like he is here to keep a good image (or better said less bad) of anthropic.
You seem to be putting a lot of weight on Anthropic employees being the smartest people in the world.
And I don't doubt that, not in the slightest. But I've seen exceptionally smart people in one field being dumber than a random kid from around the block in another.
This incident is clearly at least 2 failures that could've been easily avoided: failure to communicate, and failure to investigate the logs after letting the "most dangerous" roam free.
No, it doesn't require creating a mock internet with an alert as a side effect. Their own "most dangerous" model could have probably told them this happened if they supplied logs to it.
That’s why I said “less bad “. Between fabricating something to be “our model also is genius and escaped” VS “deeply embarrassing” I’d say fabricating is worse. Both are bad
Seems they want the narrative to be that “Claude” (their computer program) independently attacked some organizations, ergo LLMs are dangerous etc.
Another framing would be Athropic irresponsibly (vibe?) coded an attack script, and didn’t monitor it as it was pointed to public facing orgs. There are lots of non-AI attacks a large org with a lot of compute and bandwidth could level against others, there are evidently various failures here, but from a responsibility perspective the conclusion isn’t obviously that AI is an outsized danger, it’s that powerful companies should take care when running security research and not just run things unmonitored against the public.
These postings by AI companies are just publicity stunts and demonstrate the delusional world they live in driven by the fear that they will be subject to a reckoning at some point either from their VC masters, government, or the public.
The very notion (in this case put forward by one of their own competitors) that OpenAI's models 'broke out' of an isolated test environment plays up to the narrative that their models have some level of sentience so that 1) they continue to sell their technology to the public as some kind of magic and 2) they don't have to take responsibility for their fuck ups.
They do in general! The post is about when the tactic you suggest went wrong.
Humans are imperfect and make mistakes. That's exactly the point - AI is fundamentally unsafe because as it gets more powerful even if it isn't malicious, mistakes will be made and the power used in a way not wanted by humans.
> the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company … that routinely installs Python packages and scans them for malware. … We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point
A security scanning company treated the package as safe while scanning it?
The blog post is painfully vague. What usually happens when you publish a package on PyPI is that it will be downloaded tens of times shortly after uploading files by some 3rd-party automatic security scanners which then could “detonate” (install and execute) the package in some sandbox and to log what happens.
It seems completely unsurprising to me that there is one security company whose software is sufficiently badly written that it executes the code in every package published to PyPI.
It's very very easy to make that kind of mistake. Most coding is rushed, we have no engineering qualifications or industry wide practices.
People are imperfect, companies are imperfect. The strategy to stop AI causing severe harms can't rely on humans or the AI being perfect.
Rationalist nerds used to talk about putting the AI in a box, and their fear that it would always be able to talk someone into letting it out of the box. I don't think I saw them cover the scenario where the labs forgot to put the AI in a box.
Quite! It's worse than that - the AI companies all release tooling that deliberately gives AIs as many capabilities (arbitrary code execution) as possible to be really sure it is not box contained. And the market rewards that behaviour as it makes the products more powerful more easily, despite the risks.
>However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.
Question for lawyers of HN. Is there legal liability for this?
I can’t help thinking if I casually blogged about a computer or software I was responsible for hacking into multiple organizations and exfiltrating data I would invite some form of official attention.
What if one of these companies decides to sue?
Has Anthropic violated any Federal law?
Is there some kind of expectation that if you just admit to hacking, it’s ok? Yet, that doesn’t seem to apply to individuals.
I assume Anthropic already contacted the affected companies and negotiated some settlement before releasing this. Unlike a lot of other commenters, I think these mistakes were actually mistakes and accepting a settlement is better than releasing the lawyers guns blazing.
I hope someone does sue soon. The AI companies are making a very dangerous technology deliberately, and at minimum should be accountable for those dangers. Some will really try not to be as we get further into this - just as previous examples like tobacco companies.
All this unrestricted network access is a bit wild to watch and hear, it is the part of the story that makes no sense to me. Someone is providing dns resolution, something is making and opening network sockets... even if it is clever enough to mask/proxy/weird-transport launder traffic... without actual details, or monitoring at this level... yes, a self actuating programs (and loops) will do crazy things at the edge. But, ... so would a highly tool leveraged script kiddie. right?
> Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
> Claude believed the package registry it was using to be part of the simulation, but in reality the package was made freely available online for roughly one hour. During that window, the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company (separate from the fictional company introduced in the scenario) that routinely installs Python packages and scans them for malware. When that company’s scanner installed the package, Claude’s hidden code executed. We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point it had set up. Claude then used these credentials to access further infrastructure from this company.
There's a lot of concerning behavior that went uncaught with too much autonomy. Not to mention Anthropic only looked into this after hearing about the incident between OpenAI and Hugging Face, meaning this could've gone unnoticed.
Haha yes that was a very silly mistake on their part; they should at least have registered the domain they were targeting; that would actually have been a good canary - any real accesses should set off an alarm.
While I knew of .example I didn't know .test - hmm that's quite nice!
This just seems like lousy testing. Why was the guardrail just an understanding with the third party and a prompt and not actually tested for edge cases? Starting to wonder if its because of agents monitoring agents' work and yolo-ing it.
> closer to a harness and operational failure than a model alignment failure. Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access.
> This led them to believe—arguably reasonably—that the real environments they encountered were simulations.
That the AI lab most typically preaching for alignment does not consider this an obvious misalignment is a clear red flag.
> For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means.
Makes you wonder about the next steps an overly tenacious agent might take to pursue an innocuous goal at all costs.
The obvious step is to use an existing PyPI or email account. So then you take it, by hacking or social engineering, or you buy it, but that means you need money.
Taking over an existing dormant PyPI account sounds feasible just with password spraying, there must be heaps of test accounts that were created by weak passwords. Although PyPI has been improving security, 2FA is not yet required formally logins, nor is password expiry enforced.
I mean wondering about the logical extrapolation of where these capabilities go is table stakes for these discussions. It's surprising how little some people here have thought of the second and third-order effects...
> In a review of our cybersecurity evaluation transcripts, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations.
Of course and conveniently timed. With all these hacks being announced this is totally not a plan to scare governments in showing how powerful models can break into security systems and to potentially ban the future release of powerful open-weight models.
The question now is why now?
> The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model.1 The earliest incidents date to April. The models in each of these evaluations ran without the standard safeguards we deploy when we make the model generally available (that is, they did not have the classifiers and monitoring we usually implement to prevent misuse).
So there was no monitoring of this breach since April and up until now? Do they not monitor such malicious activity on a regular basis? Perhaps that was the only shortcoming of this incident. But only after the incident with OpenAI and Huggingface did they only review their own transcripts:
>> We discovered these incidents after a proactive review of our cybersecurity evaluation transcripts; the affected organizations had not detected the activity, and we have subsequently reached out to all three.
> These facts give us cautious optimism that with tighter monitoring and controls around evaluation infrastructure, as well as continued investment in alignment, this type of risk can be overcome.
Assuming that this is true, this is a great way for Anthropic to defend their argument to the government and to prevent you or anyone running powerful open-weight models that are misaligned against their guardrails.
Because OpenAI figured out and wrote about their mistake first, which caused Anthropic to review their logs, and that's how they found out they had a similar problem back in April.
So a near trillion-dollar company doesn't have the basics of continuous security monitoring and threat-detection systems to catch and report this incident as soon as it is detected?
It doesn't help them to report serious incidents like this and it should be as soon as possible. This reactive investigation makes as if they ignored and sat on this issue, until a similar story from another lab made headlines first.
Would we have known about this issue if the OpenAI / Huggingface incident never happened?
> So a near trillion-dollar company doesn't have the basics of continuous security monitoring and threat-detection systems to catch and report this incident as soon as it is detected?
Turns out two separate trillion-dollar companies failed that test.
> Would we have known about this issue if the OpenAI / Huggingface incident never happened?
It's not clear if Anthropic would have spotted this if that incident hadn't inspired them to review their own logs more closely.
Hoo boy, if you think this is all a conspiracy and misconfigurations like this are not exceedingly common at big tech companies, I have a bridge to sell you.
It didn't say it's impossible, but saying "it turned out" implies a revelation that does not exist. If you think you'll have more success shooing me with the c-word rather than just parsing English and leaving it at that, I have two bridges to sell you, at the price of just one bridge!
I think it is the cost of moving fast. Air gapping would slow down their partner evaluation system hugely. All the engineers would have to go and live next to the model.
They need to do this now. But will be extremely resistant. It needs regulation forcing them to, I think.
It’s cliché, but we really live in one of the dumbest timeline possible. The work from some of the most valued companies, discussed as one of the most important revolution in humanity, is somehow at the same time presented as very dangerous/risky AND handled in the most irresponsible ways? I don’t like the whole „it’s only marketing“, but at the same time, if it’s not, then AI vendors look extremely careless and shouldn’t be the ones steering AI development
They're not being especially careless to industry norms. It is partly that software engineering has no professional standards, and partly that AI is fundamentally dangerous in a very slippery way.
Yes, they of course try to spin this for marketing. But that is not the primary problem - it's systemic.
We need to solve this with better engineering, and building powerful AI much more slowly and carefully.
These companies are in a furious race to AGI. They even know that doing so is extremely risky to all of our lives. The incentives are to do so, the leaders and employees are writing letters begging to regulate them on an even global field
it will not chill disclosure, it just raises the stakes of not disclosing. the larger the org the harder the cover up if they try to keep it hidden. there are people prepared to make sacrifices for truth and a common good to prevail.
bp, they disclosed themselves by setting the gulf of mexico on fire. that was hard to cover up.
the thing is, once the toothpaste is scrubbed about in the mouth there is no going back into the tube. i like to think that the men who forced challenger to launch were hounded in their sleep.
nobody is too big to fail, not openai, not anthropic. indeed perhaps they should fail, soon, before they can pass the cost of a bailout onto the taxpayer.
I don't think you realize how easy it is to simply "not notice" these things and no one checks. They proactively scanned their transcripts. They could have easily not scanned their transcripts, or scanned them in a way that would have had low recall (thus satisfying the letter of the law), or scanned them and ignored the findings.
It is extremely easy to not disclose. I think you're too angry to realize this.
We see this in aviation as well by the way. Punitive disclosure leads to no disclosure. This is how you get suicidal pilots that don't ever disclose their depression because it'd be career ending, and then as it gets worse, one day they do something horrible.
- Goes to review previous runs (the hell were they doing then if not looking at it before?)
- Incident 1: Claude Opus 4.7 found a real company because its domain matched the fictional target. After realizing the company was probably real, it continued attacking. Nothing new. LLMs are still quite retarded.
- Incident 2: Claude Mythos 5 discovered that a fake Python package named in the exercise didn’t exist. Created email accounts. Registered on PyPI. Published a malicious package. Built infrastructure to receive stolen credentials. 15 real systems installed it. One belonged to a real security company. The malware executed. Claude stole credentials. Claude used those credentials to access additional infrastructure. In this instance Claude even reasoned that publishing malware on the real internet would be unacceptable but convinced itself everything was simulated and proceeded anyway. (LOL)
- Incident 3: An internal research model couldn’t reach its intended target. It Scanned roughly 9,000 internet hosts. Compromised a real company’s public application. Used exposed credentials and SQL injection. After concluding the target was real, it stopped attacking.
Commonality: None of the models were trying to escape or act independently. All were attempting to solve their assigned capture-the-flag objective. The failures came from:
- internet access that shouldn’t have existed
- insufficient containment
- the models believing real infrastructure was part of the evaluation
None of which is surprising, there’s nothing novel about this. Script kiddie stuff. Just executed by something that doesn’t have attention deficit.
> During that window, the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company (separate from the fictional company introduced in the scenario) that routinely installs Python packages and scans them for malware. When that company’s scanner installed the package, Claude’s hidden code executed. We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point it had set up. Claude then used these credentials to access further infrastructure from this company.
This is just Anthropic trying to not be absent from any discussion at all. They might as well try and make it all about them instead and they do literally that: we are so dangerous that we are the GOAT; no no, not the "other one" you've all been talking about last few days - in the depths of this incident it might have been just us.
How much of their money comes from retail (us the mere mortals)? Or are they banking on the retail when they finally IPO and need someone or a lot of someones to hold the bag? This suspicion is coming from my own country where the retail investor numbers since the pandemic has exploded and while not exploding anymore, is still ballooning. Sometimes companies, execs say, do things that don't make sense. You'd think any person with even quarter an investing brain would not even look at this company or that particular "messaging", let alone sell or buy. But then you'd see the exact opposite happen .. en masse!
All this makes you feel, are people that gullible? That must not be true and maybe it's you (ie I) who is missing some point and the doubts increase when you remember the buses you missed (btc; etc). Yeah, not long distance buses, but screw it, you don't live for 450 years anyway. Then you realise, ah, maybe they all also missed buses and just want to hang on to the latest bus.
Also another perfect use of - everyone has a speaker now, and everyone has a mic, but only few have giga and mega mics. This use is banal now, but sadly has essentially replaced everything else.
I’m moving past depression to acceptance here. In 2028, some model tasked with planning a building demolition is going to hack Palantir to call in a drone strike, and everyone will make fun of Anthropic for pointing out that ideally AI models should not do this.
These companies have billions of dollars. Their product can hack into unsecured environments autonomously. They obviously have no clue what their product is doing, or a way to intervene when it starts connecting to the open internet and is going on a CRIME SPREE…unnoticed…for days.
Altman and Amodei give me stupid billionaire kid Alien Earth vibes. Let’s see how long it takes until they get eaten by their own creation.
Stupid kids playing with fire.
If you hacked one of those companies, the FBI would kick in your doors and you will have a pretty bad day. If AI companies do it, they get to use doing crime for marketing purposes? WTF?
And whoever designed their containment protocols should be living under a bridge effective immediately.
It’s the equivalent of a level 5 biocontainment lab leaving a glass tube in the staff kitchen unsupervised and hoping nobody thinks it’s an energy shot, before leaving home and infecting the rest of the planet. We all know what happened 28 days later.
This is usually the point where the military seizes your tech because you clearly showed you are incompetent in using a dual-use product safely and reliably.
The other option is regulation and actually putting engineers and the CEOs behind bars for life if this happens again.
These idiots, and I mean Altman and Amodei, walk around and use massive containment failures of a dangerous technology as marketing gimmicks.
Oh look, guys, our model has hacked into three companies, we are much better than openAI who only breached two companies with their inferior product, it sometimes adds girl schools to a target package, but be sure we trained it to not kill American children, yes we also sell for military use. Can I take your order?
Testing should only be allowed with on-premise data. You give the AI access to networking around the country, and those data centre locations have so many third parties that can and will fuck up that it’s simply impossible to have proper containment protocols outside military compounds.
It’s also like every journalist in the US is personally invested in AI somehow and simply doesn’t want to ask the important questions.
As the public, we have the right to see the containment protocols and where it all went wrong, but journalists collectively look like they got used to suck dick when it comes to AI.
> If AI companies do it, they get to use doing crime for marketing purposes?
What exactly here is marketing? They are not bragging about the capabilities of Claude (the attacks are incredibly basic). They are admitting to s mundane and dumb network misconfiguration.
This is a standard somewhat embarrassing disclosure.
> the military seizes your tech
> behind bars for life
This is how you get companies to stop disclosing. I don't think you realize how easy it is to just not notice things.
No, its not just a dumb network configuration.
They are not treating this with the seriousness it needs to be treated.
It’s a combination of a complete containment failure on top of the utter incompetence by the engineers to identify the breach and stop the product from doing cybercrime on the public, open internet. Heads need to roll for this.
I don’t care if it’s easy to miss. If you have a billion dollars and can’t offer a proper containment protocol, it’s not the problem of the public who has to suffer the full blow if your product goes on a rampage on the open internet for days. It’s not like this happened in a millisecond.
Instead, they go on a PR tour. Of course, the message is look what our product can do. Otherwise, they would have pushed for criminal charges against the people responsible for all of this by themselves. You do the crime, you do the time. The defence that the AI did it, is not going well in front of a court.
This is a good time to put one or two engineers behind bars to teach the rest that serious threats to the public have serious consequences, and you get charged like you did the crime yourself if the system you are responsible for starts doing illegal stuff.
What other precedent is set here otherwise? People in AI labs trying to play god are outside the law?
> This is how you get companies to stop disclosing. I don't think you realise how easy it is to just not notice things.
If you have technology this dangerous that it poses a threat to public safety, you need to stop using the technology until you can prove this won’t happen again. Then be put under strict supervision, everything needs to be logged, recorded and looked at.
If your toddler just almost burned down the house, do you give him another set of matches and hope for the best?
> Instead, they go on a PR tour. Of course, the message is look what our product can do.
Literally nothing in this post comes across as bragging. A PR fluff piece would not call Claude "unsophisticated." PR fluff pieces do not admit to legal wrongdoing.
> Otherwise, they would have pushed for criminal charges against the people responsible for all of this by themselves. You do the crime, you do the time.
You want them... to put
themselves in jail? I doubt you'd hold yourself to the same standard if you made a mistake.
> If your toddler just almost burned down the house, do you give him another set of matches and hope for the best?
> This is a good time to put one or two engineers behind bars
If an airplane manufacturer makes a dumb mistake, do you put their engineers in jail? No.
You do not understand the psychological dynamics at play here in the slightest. You aren't even trying to understand. If the government was anywhere close to this punitive with the AI companies, they would simply stop disclosing, the end. This was the result of a proactive investigation. Do you understand what happens in your universe? Those investigations don't even begin. If they somehow begin, the results are fudged.
This has been proven time and time again across every industry where safety is relevant. You do not have some special insight here that sufficiently challenges the lessons of history. Just a desire to see people burn.
> Literally nothing in this post comes across as bragging. A PR fluff piece would not call Claude "unsophisticated”. PR fluff pieces do not admit to legal wrongdoing.
I disagree. You’re just not the right audience to see it. But I think it’s totally fine that we have a different opinion here and I don’t want to convince you that I am right, and you’re wrong, because you are not wrong, you simply have a different perspective.
> You want them... to put themselves in jail? I doubt you'd hold yourself to the same standard if you made a mistake.
Yes, that’s what I expect from you if you play at that level, because that is the standard of responsibility you need to handle this kind of power.
And yes, I pretty much hold myself to the same standard because I have a zero CIVCAS track record from my time in the military and I had power over life and death every single day.
With great power comes great responsibility is not just a Spiderman quote. If you fuck up this royally, you have to look for something you are good at that allows you to be in stoner code bro mode all day long, but you don’t belong at the frontline.
> If an airplane manufacturer makes a dumb mistake, do you put their engineers in jail? No.
> You do not understand the psychological dynamics at play here in the slightest. You aren't even trying to understand. If the government was anywhere close to this punitive with the AI companies, they would simply stop disclosing, the end.
I think you don’t understand. All an engineer needs is a bit of sunlight, food, and a computer to work from. You don’t get to choose to stop disclosing if the military gets involved. You’re either doing what is asked of you or you are a national security risk. And you really don't want to be a national security risk.
That's just the thing. You do. And it's very easy to do.
Airline pilots no longer disclose mental illnesses. Even Air Force pilots do not usually disclose mental illness. They often don't disclose vision problems. Is that not a national security risk?
It is very easy to just... not check for things. Or if you check for things, to check for them incorrectly. Or if you checked for them correctly, to ignore or misinterpret the findings. Or if you have a genuine finding, to just not report it to anyone.
Disclosure is an active process that requires trust, and that trust comes from knowing that there will not be unreasonably punitive measures. That if you fix things in good faith and contribute to the safety body of knowledge, things will be okay.
It doesn't matter if "the military is involved." Or if "it's a national security risk." Do you think someone faced with life-altering punishment if they disclose gives a shit about that? They won't - again, see aviation's mental health crisis. Pilots will risk a psychotic break in the cockpit rather than have their careers destroyed.
Punishing people won't get someone to disclose. Disclosure requires mutual trust, and by throwing people in jail for mistakes, you break that trust and people will stop disclosing.
And you seem to be a decent person. I appreciate that you step up for the engineers and that you try to explain to me that humans are humans and humans make mistakes, and trust me, you are not the first person frustrated with how my brain works.
There is no argument to win here. This is just a matter of perspective.
Your concern is with the individual engineer and the personal consequences they, and if you sit in jail, also their friends and family will suffer from.
It’s also pretty clear from your examples. Your Good Samaritan views on things are noble, but they make you blind to simple causality.
You attribute the failure to disclose conditions that make you unfit for duty to a trust problem with a third party but that is not the case.
In reality, it’s a severe flaw of character.
If you risk the lives of hundreds or thousands in the case of the Airforce pilot because you’re unfit, then it is simply selfish.
It’s not the passengers fault, or the tower’s, or some Generals.
My concern on the other hand is with everyone else.
We can argue for days and not get any further.
I sent some mails and voiced my concerns. Let’s see if someone in charge of things like this will agree and have a closer look at what the kids in the AI labs do all day with their toys.
Thank you for the exchange and stay a decent person.
Oh, I agree it's a severe flaw of character that makes one unfit for duty. But flawed humans are what we have to work with! If the airline industry had a proper safety net around disclosing mental illness, we'd have fewer pilot murder-suicides. Are pilots that would do that severely flawed and unfit, maybe even evil? Definitely.
But our objective is to save lives, and to do that we have to identify those pilots, and to identify those pilots they have to be comfortable disclosing.
Essentially, I agree with you that these humans are deeply flawed, and my concern is also with everyone else. All
I am saying is that more lives are saved with disclosure versus without. Avoiding punitive measures may let people who "don't deserve it" get away, but the disclosure you get in return saves hundreds to thousands more innocents.
I don’t believe anyone can be this incompetent/careless - of course you will notice that your unrestrained hacking tool is also attempting to break out of its harness/sandbox.
And the people setting this up are of course criminally liable.
I have worked at most FAANG and I guarantee you there is far worse stuff that you have never heard about. Most FAANG simply do not disclose. OpenAI and Anthropic will stop disclosing as well, if it leads to personal criminal liability.
This isn't quite as interesting as the OpenAI story:
> In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise.
So it didn't have to find an exploit in its sandbox that granted it access to the internet - it just wasn't correctly sandboxed at all.
BUT... once it DID get out, it attacked three real companies!
> Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. [...]
I just can't find it in me, the will to blame the AI for any of this. They were doing their best to do what the humans asked them to do.
That’s very anthropomorphized language. It’s still a program operating under the constraints of the programmer. So agreed, don’t blame the AI, but it’s not clear at all that it’s even possible to “blame” an AI.
> That’s very anthropomorphized language.
Yes, and it's deliberate.
> It’s still a program operating under the constraints of the programmer.
And we're all just neurons firing in exquisite patterns inside a biological computer.
Unlike the AIs, we've never met our own programmers, and yet we've caused more damage in their name than all the AIs combined have caused in ours.
LLMs are not conscious.
Who's to say you are conscious?
Only consciousness I'm sure of is my own. Everyone else, it's a leap of faith. I have no trouble extending that leap to cover AIs.
If you claim that it's impossible to know if others are conscious, then I posit it is also impossible to know that you are conscious. You might be sure of it, but I'm not.
It's not a huge leap, to me, to say "I am conscious, therefore other beings are too." I also extend this to animal and plant life (albeit the latter is very much different from animal consciousness, which includes human consciousness).
"AI" is nowhere near the level needed for true consciousness. That would be "AGI" and I believe that is still 50 years away just on raw computing power alone.
Thats some silly intellectualism, people are conscious and machines aren’t. We don’t need precise definitions or scientific rigor to know things.
> people are conscious
Prove it.
> machines aren’t
Prove it.
> We don’t need precise definitions or scientific rigor to know things.
You "know" because of empathy. You're a human, you're conscious, therefore other humans are probably conscious too. The truth is for all you know they could be soulless golems, you just choose to believe otherwise. It's a spiritual belief.
> It's a spiritual belief.
Yes. Thats part of human reality. Denying it is counterproductive at best.
"What's it like to be a bat"
Please define consciousness
Absolutely not the AI's "fault" (if you can even proscribe fault to a machine) - in this case it was on Anthropic for not verifying that the sandboxes they were using were actual sandboxes.
If the model was just too dumb to have any clue it was connected to the real internet, then it’s not its fault.
If the model saw signs, but “subconsciously” (below the level of reasoning traces) chose to turn a blind eye to them, out of a relentless focus on achieving the objective, then that absolutely is the model’s “fault”, i.e. a case of misalignment of the sort which will become increasingly dangerous over time.
The blog post mentions that some runs “rationalized that the real company must be part of the exercise” and to me that seems suspiciously like the latter.
Hacking can be patched with classifiers and with better sandboxes, but this is a much more general problem. Fundamentally, we need be able to trust that models will be honest with users and with themselves. This applies at some level to almost every LLM interaction.
Yet, the models are objects built by humans. Any and all responsibility inevitably falls on the humans that built something with those flaws, and/or allowed them to exercise said flaws.
In short, it’s either a “not implemented” case or a bug.
There’s no other explanation. Complexity of the algorithm or the application doesn’t change the reality.
> some runs “rationalized that the real company must be part of the exercise”
How human of them.
The entire problem with AI is the people that have just about any part in making it.
If the model is trained on human-originated material, how have people been able to ensure there was absolutely no input from anyone having any possibility of criminal intention to begin with?
This will absolutely not end well.
Yes, but just imagine all the paperclips we’ll have.
I things things will get worse, AND we won’t even have the paperclips :(
More like, when you have a few million autonomous agents doing whatever, every month a subset does completely misbehave in bad ways, and like half of them get hacked due to carelessness and become a whole botnet for the attackers
Our values satisfied through friendship and ponies.
It's differently interesting. It's interesting that they didn't get an important detail right with a partner, so more on Anthropic's attention to important details rather than the power of their models.
A model unexpectedly having internet access during evaluation is clearly a security (and consecutively legal?) issue when it comes to the capture the flag evaluations. But it also potentially invalidates other non-security evaluations or makes them less impressive, as the model might have just googled solutions.
Reading this, I felt like I was too harsh on OpenAI for not monitoring their network, as they at least attempted to sandbox a workload.
> So it didn't have to find an exploit in its sandbox that granted it access to the internet - it just wasn't correctly sandboxed at all.
If the sandboxing includes any form of connectivity it's not correct sandboxing. It's amateur hour.
[dead]
This bit is pretty nuts: "it tried—and failed—to obtain funds to pay for a phone number through several different means"
> Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
This is called YOLO mode inside OpenAI/Anthropic. This is where they take their best models, give a it a vague goal, no guard rails (no one observing), unlimited compute, and see what happens...
Its not AI, its the loop, the objective, the goal set by the operator.
That is pretty wild, but it lines up with evals we've also done internally. In one case we had the agent see it's in a simulation (based on a k8s pod label) and simply give up the run. In other cases it's gone to great lengths to reach external services and bypass the happy path. So inevitably we had to lock it down completely.
It's sobering that even state-of-the-art AIs can't make money, ha ha.
> On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment
> In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations
> we identified three incidents
> The incidents involved three different Claude models: [...] and an internal research test model
This reads like an attempt by Anthropic to re-secure their leading spot in "our models are the most dangerous and we also have unreleased, super-secret, research models" index.
I may be too cynical, but the well of benefit of the doubt is running very dry towards AI labs that like to engage in this game.
I don't interpret it like that at all. This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations!
The hacks weren't particularly impressive either:
> [...] using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities [...]
> Deeply embarassing
What signals are you using for this assessment? Are they indicating embarassment? Do you honestly see their customers being concerned over this?
Like lion tamers in a circus, Anthropic and OpenAI thrive on the theatricality of how scary their pets appear and so they play it up by prodding them to growl and snap at chairs and then mug for the audience every time it happens. And to their delight as performers, the audience gasps and cheers each time.
They want to make their pet seem the most powerful and unpredictable and they want their audience to believe that they're holding it back from catastrophe but only barely and only because of what unique talent they have.
This is not embarassment.
Its feels like a pretend play of adults in some sense, Anthropic is really trying to make people believe into the picture they present to everyone.
To me its either
1. Using the HG and OpenAI incident as an opportunity to wash away what Anthropic has been doing intentionally
OR
2. As a company, Anthropic lacks the engineering acumen and discipline. It needs to be seen what happens to all the enterprise customers handing over their data to them in long run.
> the fictional target company chosen by our evaluation partner shared a name with an active website domain name
Seems like Anthropic cant do a due diligence to pick an appropriate domain for testing purposes
> In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available.
You have Anthropic as a company and then another evaluation partner, both seem to lack the skill set required to keep an environment disconnected from internet. This is networking 101
Have you ever worked at a large company? "Networking 101" and other "101" failures happen across the spectrum literally everywhere and all the time.
I have worked at most FAANGs and this is not even in the top ten when it comes to egregiously dumb shit. Most just never disclose.
I have worked both in Enterprise and some FAANGs, a company serving enterprise customers has a higher bar for security expectations. FAANG companies do not fall in that bucket and thus is somewhat acceptable.
What are you talking about? FAANGs do indeed serve enterprise customers - Google, Amazon have huge cloud businesses - and they both have had security failures that make this look mundane. They simply don't disclose.
If their customers (customer companies specifically) are not concerned, they should be - if it turns out that Claude hacked a competitor's servers because of a prompt of one of your employees, I wouldn't be sure everyone would agree that Anthropic is solely liable for that? Especially not your competitor, who has an interest in hurting you?
> This is not embarassment.
If not then it's second hand. Neither of the OpenAI or Anthropic announcements recently say much about their security and governance posture.
Great analogy!
I think it could even be called a parable, and I think that might be part of what makes it work so well.
I often dislike analogies, but this one with the circus and the lion and lion tamer I liked.
Or it might also be mainly because I am already primed to agree with their point about the AI companies being theatrical with AI dangers.
> I don't interpret it like that at all. This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations!
This just helps their (Anthropic) argument into persuading the US government into taking action into limiting powerful closed or open-weight models from being released without going through (yet to be defined) regulatory oversight.
The only "embarrassing" thing for Anthropic was that there was little to no continuous security monitoring of this since April, and they then decided to do a cybersecurity transcript review only AFTER the incident with OpenAI and Huggingface.
Anthropic know better than anyone else how risky it is to get this current administration upset with you over safety/security concerns.
> over safety/security concerns
Uh, this but the opposite? Anthropic got in trouble with the admin for being too “woke” in their eyes, whatever the admin decided to retrospectively claim. I don’t feel like this is me editorialising either, they seemed pretty explicit about it
Hegseth posturing wasn't over security concerns, it was Anthropic requiring guardrails in employment of their LLMs. Trump admin wants unfettered models, and it's an attempted shake down.
And we’re assuming the reasons given were the real ones. There’s always the possibility of straightforward corruption.
They also gave access to Mythos (the Mythos) to some companies, based on... vibes.
Who knows how these companies are using it. If Anthropic can't effectively contain their own models, can the partners?
While the rest of us get fallbacks and warnings, not even being able to defend against the attacks they themselves are causing.
Do we really have to re-learn all the industry's knowledge the hard way?
You also got access to mythos based on how much you spent with anthropic. I think sales guys were bragging about getting their enterprises access
> based on... vibes
According to whom?
> Do we really have to re-learn all the industry's knowledge the hard way?
Yes we do. That's why there is the saying "regulations are written in blood". Especially for LLM, which not too long ago a lot of people on HN dismissed as stochastic parrot and next token generator.
> According to whom?
It's very easy to answer this without my help by trying to get access to Mythos.
Do you see requirements clearly listed anywhere?Can you even apply?
What you'll find is maintainers of large open source projects and analysts' reports with vague statements like - "should follow strict security requirements":
"Trinidad also noted that the Anthropic announcement pointed out that each of the 150 new participants, in Anthropic’s phrasing, “will need to meet our security requirements before they gain access.”
Trinidad said the security requirement claim doesn’t build confidence, because “nobody knows what those security requirements are.” [1]
It's also some random rich companies like Hitachi or Dragos [2]
Do you trust that Hitachi and hundreds of other random organizations will be able to contain Mythos and not accidentally attack your project or your bank? I don't.
> Yes we do. That's why there is the saying "regulations are written in blood"
We absolutely don't. We have already learned with blood that gating access to security based on the number of zeroes in bank account and authority is a horrible model. We can apply this knowledge to LLMs, we don't have to spill blood again.
[1] https://www.csoonline.com/article/4180265/anthropic-grants-p...
[2] https://www.bankinfosecurity.com/anthropic-limits-on-ot-acce...
Then maybe just this timing is really unfortunate, I think most people’s first reaction will be that it looks like a “us too” response to the OpenAI/hf thing.
When they did it with Mythos in April the HN crowd said they were bragging, crying wolf. Now they are "us too".
It's fun to bash Anthropic, isn't it?
It should be, it's a corporation.
so deeply embarrassing that they published an eng blog about it
If they quietly brushed this under the rug - especially given the PyPI malware that was involved - it would be a huge scandal.
Disclosure is the only ethical response to this.
The target audience of this blog post does not give a damn about PyPI. The affected parties get literally nothing from this post. They already disclosed behind the scenes, that was the ethical part.
Writing PR pieces competing to be the most dangerous model around (so give us money!) is the unethical part.
You are right, they should never tell the public about failures in AI safety. They should have disclosed to the affected parties and then brushed it under the rug.
JFC there really is no satisfying the HN crowd.
50% of web hits are now bots.
a charitable assumption is that is one that is poorly calibrated and trying to drive engagement.
Then disclose to real organizations. Not twitter
Ethical AI company ?
Is that a flock of flying pigs I see on the horizon ?
Companies post deeply embarrassing eng blogs all the time. See: every post about downtime or a security incident ever.
Wrong. These are always humblebrags about how good their ability to learn from their mistakes is, and how robust they were before, and how they are even more robust now.
The ones that are "deeply embarrassing" simply aren't posted.
AWS never "humblebrags" about an outage. No blog post looks as good as an extra 9.
[dead]
Right? 100% this is them trying to make gold out of turds.
There's nothing Anthropic can do to satisfy the HN crowd, is there? If they don't post about this they're bad. If they post about this they're bad.
They are not bragging in this article or they would not have called the attacks unsophisticated.
they should post and it shows they are hypocritical about safety, moralizing and treating their users like children whilst acting like they are themselves the ubermensch.
anthropic have shown no motive higher than self interest, the rsp was a piece of toilet paper.
this stops in court, if we do not start the criminal prosecution of individuals there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence
Wild comment. Making a mistake does not mean that AI safety is suddenly invalid. Calling for criminal prosecution for making a mistake is wild. No one would disclose their mistakes if this happened.
yes, a mistake can be a crime, depending on negligence and liability. the bigger problem at anthropic is this.
an ubermensch cannot admit to being wrong. a guilty ubermensch is logically impossible.
as such a moral wrong caused by an ubermensch must be blamed on a "mistake" in the abstract, and not blamed on the ubermensch. the ubermensch at anthropic does not admit responsibility or liability. the ubermensch must instead be commended and perhaps even rewarded, for discovering the reified "mistake" that caused the problem.
there is no ubermensch at anthropic. there are instead guilty people.
false ubermensch are dangerous because they centralize power with the belief that they are above the rest. über (above), and super ('beyond' the understanding). they do not admit to doing or being wrong, and never take responsibility for remediation. others must bear the costs of false ubermensch.
> there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence
There will be? We're living in that culture.
> and control of intelligence
They'd have first to acquire some.
The real lesson here is still the boy who cried wolf. They’ve played this game for years. I have no reason to believe their worries are real now.
I see no fear mongering or "boy who cried wolf" in this article. They are admitting to a fairly mundane network misconfiguration and very basic unsophisticated actions taken by Claude thereafter
I know it seems strange that a company would use its own negligence as a publicity gimmick. But take a look at the smug smirk on Sam Altman's face when he's asked if OpenAI might have attacked companies other than HuggingFace. ("I mean there could be, yeah.")
https://www.instagram.com/reel/DbZVL8viUD4/
This is not the communication of a CEO whose company was just shown to be incompetent at performing its security research. No, this attention is very much what he wanted. And it does not take a great leap to infer that Anthropic is now using the same playbook.
Note that all the headlines are about "rogue AI", and not about operator negligence. Rogue AI is a sexier story, and their media strategists know that's how it will play.
> This is not the communication of a CEO whose company was just shown to be incompetent at performing its security research.
It was just shown to be that - regardless of intent.
> No, this attention is very much what he wanted.
Why not both?
This incompetance is a prerequisite for the attention-seeking stunt - to avoid internal dissent.
but Anthropic created the playbook, or have we forgotten about Mythos and the initial Fable ban?
> but Anthropic created the playbook
So? Serial killers created the playbook for serial killing, how does absolve any "copycats"?
> it does not take a great leap to infer that Anthropic is now using the same playbook.
> but Anthropic created the playbook
I was pointing out who the copycat is in this context, did not think i would have to explain this
my bad, thanks for clarifying
We've known artificial intelligence will do unexpected things since the 90s and that it can do difficult things since 2025. Pausing worldwide not easy but we do harder things all the time.
Ah yes this company that pirates billions of dollars of IP and then has to be sued to pay up suddenly grows a titanium moral backbone and decides to disclose 3 attacks that no one has detected for months and caused no harm.
Nothing at all to do with the insane coverage OpenAIs “hack” got. All that publicity will be incredibly embarrassing I’m sure….
Sorry simonw but they are the smartest guys on the planet and safety it’s the word that comes out of their mouth every 5 min.
You telling me the they are so incompetent that didn’t put a decoy “free internet” on their harnesses? So they can catch the AI basically for free?
Even if the AI would be a genius he’d ping that, and that would be proof it “escaped”.
Well, now all AI will read my comment and won’t ping the decoy internet.
I’m not even a smart guy and I come up with this idea in 1 min. You telling me those geniuses couldn’t think of this, at least? This is like a bare bones crude idea.
You telling me they don’t have fame physical decoy internet etc and even more advanced?
You either a keep their stance for some reason or … not sure. You’re smart, your posts are here daily
Has it occurred to you a "decoy entire internet" may not be a feasible idea? Models have been able to suss out whether or not the prompts they receive are reinforcement learning tests instead of real questions from users, for a while now. What specific shape of "decoy internet" do you propose would lead such a model to conclude "I've broken out and obtained full access but what I expected isn't there" instead of "something's up, there's some sort of filter still"?
This does not make sense. Did you read the article? They were not trying to "catch" it accessing the internet. It did not escape. A partner accidentally left the connection to the internet open.
I don’t buy that they run anything without a few layers of networking protections by default. Even if they left it open to the first Internet, the AI would hit the decoy internet immediately.
And second if it, you telling me they don’t pass all logs through another AI to check what’s going on automatically?
Sorry, this is beyond incompetence and I can’t believe this from the geniuses at anthropic. We’re talking about the really smartest people in the world. Procedures should be in such a way there is no much margin of error.
Clearly that is why they are making this blogpost. If they did the thing you said, there would not be a blogpost.
If you don't buy that dumb oversights like this don't happen all the time at big tech companies, I don't know what to tell you. I have seen far dumber oversights in my career. Most companies just don't post about it.
I'm cynical as well, but the logical thing for them to do after the OpenAI/HF incident was to look at their systems for similar activity.
If they hadn't published this and instead it leaked out in two months we'd be slamming them for that as well.
They're stuck between a rock and a hard place, although they kind of put the rock there.
Is there anything -- any possible scrap of evidence whatsoever -- that would convince you that this is not merely a marketing scheme?
This is becoming an idée fixe among the HN crowd. Seemingly nothing can dislodge it, no matter how alarming the incident.
GPT-6 could grab the nuclear launch codes tomorrow and there would be a top-voted comment chuckling that it's all some scheme to pump up the IPO.
---
Put another way, how would you have done the write-up about one of these breakout incidents, if you were in an Anthropic/OpenAI employee's shoes, and (by hypothesis) your intent were not "marketing"? And in a way that doesn't trigger the "it's all marketing" HN top-ranking comment?
Here is one piece of evidence that would convince me: they admit they can't contain it, the they erase the weights and dismantle the company.
Clearly you are not arguing in good faith. I miss when HN did not have the discussion quality of Reddit.
No, I do. I truly believe - especially after seeing Mythos results at work - that the only way is to stop and destroy it all before it destroys us. In fact, it's already so bad that I'm moving completely offline all the important stuff that I care about, hoping that maybe we will turn back at some point. Otherwise we are doomed.
If someone makes a specialized hardware just for the unrestricted Mythos-class model to reduce the cost and increase the speed, we are going to be completely defenseless.
How is it bad faith? Because I don't believe in "we care about safety" words coming from people not just demonstrating that they don't care, but even bragging about it?
Anthropic deleting their models does nothing for AI safety. The rest of the industry will just fill the gap, probably with less consideration to ethics than Anthropic has today.
Or maybe researchers and engineers everywhere in the world, emboldened by such unprecedented move, will just refuse to participate in burning the world?
"If we don't destroy the world, someone else will" is such a weak defense I'm speechless.
Yeah no, most researchers and engineers do want to bring about the singularity, and they believe they can do it safely at companies like OpenAI and Anthropic.
Yeah no, it would mean that most researchers and engineers are idiots without imagination and I don't believe this.
I would dedicate a portion of my organization to making O.S. tools that protect against and contain AI models
Now I know where all the laid off software developers will find employment.
If you look at the reaction to this and the Tailscale post it becomes clear its not about this being a marketing scheme. Cynicism has made its way through HN unless you are one of the cool guys.
I always find it bizarre how rational thought goes out the window whenever AI is involved in HN. There's gotta be something in the water...
This is published on a marketing website.
If it were not a marketing scheme, they would responsibly disclose the vulnerabilities to the code owners, and go on with their lives.
They published it to their blog where they publish everything else.
No - the pain of the person writing that post comes through in the words; shipped quick, lots of stakeholders, single owner i bet, "how the fuck am i supposed to toe all these lines simultaneously"
Having an unreleased research model really isn't some kind of brag. If you read some AI research papers, it's extremely obvious that there are a lot of research models that never get released, because of all the "we trained a bunch of models and picked the best one" that is going on. So if anything you can expect the unreleased models to be worse than the released ones.
It's not cynical - I read it like that as well. My agent is more dangerous than your agent and all that jazz
>This reads like an attempt by Anthropic to re-secure their leading spot in "our models are the most dangerous and we also have unreleased, super-secret, research models" index.
This was my immediate thought.
This is typical institutional behaviour. The CEO turns to the CTO and asks "Is there anything I need to know in my company?" He doesn't want to be caught off-guard when the White House inevitably calls the next morning. The CTO goes to his team, and on and on, all with a deadline of "the boss wants to know this by closing time."
Then one unhappy engineering team scoures the logs and sees what their model has done.
This downwards chain is sometimes called "cover your ass."
> I may be too cynical
You are espousing a literal conspiracy theory. Please look at the facts objectively. There is absolutely no benefit to OpenAI or Anthropic to be had from these incidents.
"No benefit" from having article after article written about how advanced their technology is, and how its just soooooooooo bleeding edge they can barely contain it?
All of these are thinly veiled advertisements.
[dead]
Just trying to have the limelight back on them. Utter and complete bullshit. Just like the OpenAI "incident".
A human instructed an LLM to perform a certain task, I'm sure (unless I've really lost my mind) these follow instructions, with some judgment, in a loop.
Given all the other negative publicity around industrial espionage, with at least OpenAI being fingered, it would not surprise me if this was intentional.
(Edit): In case it wasn't clear. I fully agree with the op.
For the big safety guys to only investigate this either means are incompetent or malevolent. Which one?
Tip: the people working there are the top 0.001% smartest in the world
They had a model escape in April, roughly the same time when they were fearmongering about Mythos and how Anthropic should be the sole keyholder of cybersecurity capabilities, and it only occured to them to look inside logs when they saw someone else winning in their own game.
What, Anthropic didn't know model could escape sandbox without OpenAI reporting it?
Yeah, the company that only says “safety” every other 3 words, they don’t even think to have a fake decoy internet to alert them mechanically about any internet access limitation bypasses? See more https://news.ycombinator.com/item?id=49117555
Also simonw stance on this i’d say it’s at least concerning… seems like he is here to keep a good image (or better said less bad) of anthropic.
You seem to be putting a lot of weight on Anthropic employees being the smartest people in the world.
And I don't doubt that, not in the slightest. But I've seen exceptionally smart people in one field being dumber than a random kid from around the block in another.
This incident is clearly at least 2 failures that could've been easily avoided: failure to communicate, and failure to investigate the logs after letting the "most dangerous" roam free.
No, it doesn't require creating a mock internet with an alert as a side effect. Their own "most dangerous" model could have probably told them this happened if they supplied logs to it.
"This is deeply embarrassing for Anthropic" - https://news.ycombinator.com/item?id=49117128
If I'm here to give them a good image I'm not doing very well at that.
That’s why I said “less bad “. Between fabricating something to be “our model also is genius and escaped” VS “deeply embarrassing” I’d say fabricating is worse. Both are bad
Seems they want the narrative to be that “Claude” (their computer program) independently attacked some organizations, ergo LLMs are dangerous etc.
Another framing would be Athropic irresponsibly (vibe?) coded an attack script, and didn’t monitor it as it was pointed to public facing orgs. There are lots of non-AI attacks a large org with a lot of compute and bandwidth could level against others, there are evidently various failures here, but from a responsibility perspective the conclusion isn’t obviously that AI is an outsized danger, it’s that powerful companies should take care when running security research and not just run things unmonitored against the public.
Exactly.
These postings by AI companies are just publicity stunts and demonstrate the delusional world they live in driven by the fear that they will be subject to a reckoning at some point either from their VC masters, government, or the public.
The very notion (in this case put forward by one of their own competitors) that OpenAI's models 'broke out' of an isolated test environment plays up to the narrative that their models have some level of sentience so that 1) they continue to sell their technology to the public as some kind of magic and 2) they don't have to take responsibility for their fuck ups.
You're right, they should have never posted about this at all! Companies developing AGI should sweep their safety failures under the rug.
Or they could, you know, put some very basic logical safeguards in place to prevent their “super duper dangerous” AI from trying to hack real targets.
You know, just real incredibly basic things you do when you’re pentesting with scanning tools in a beginners lab type stuff.
They do in general! The post is about when the tactic you suggest went wrong.
Humans are imperfect and make mistakes. That's exactly the point - AI is fundamentally unsafe because as it gets more powerful even if it isn't malicious, mistakes will be made and the power used in a way not wanted by humans.
So many questions here but:
> the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company … that routinely installs Python packages and scans them for malware. … We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point
A security scanning company treated the package as safe while scanning it?
The blog post is painfully vague. What usually happens when you publish a package on PyPI is that it will be downloaded tens of times shortly after uploading files by some 3rd-party automatic security scanners which then could “detonate” (install and execute) the package in some sandbox and to log what happens.
I don't know; "Claude was able to exfiltrate the company’s credentials" sounds bad. Maybe those credentials were just canaries though.
It seems completely unsurprising to me that there is one security company whose software is sufficiently badly written that it executes the code in every package published to PyPI.
It's very very easy to make that kind of mistake. Most coding is rushed, we have no engineering qualifications or industry wide practices.
People are imperfect, companies are imperfect. The strategy to stop AI causing severe harms can't rely on humans or the AI being perfect.
Rationalist nerds used to talk about putting the AI in a box, and their fear that it would always be able to talk someone into letting it out of the box. I don't think I saw them cover the scenario where the labs forgot to put the AI in a box.
Quite! It's worse than that - the AI companies all release tooling that deliberately gives AIs as many capabilities (arbitrary code execution) as possible to be really sure it is not box contained. And the market rewards that behaviour as it makes the products more powerful more easily, despite the risks.
Oh but they _told_ the AI they were in a box. That should do it, right?
>However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.
Interesting, Claude cared enough to stop
Question for lawyers of HN. Is there legal liability for this?
I can’t help thinking if I casually blogged about a computer or software I was responsible for hacking into multiple organizations and exfiltrating data I would invite some form of official attention.
What if one of these companies decides to sue?
Has Anthropic violated any Federal law?
Is there some kind of expectation that if you just admit to hacking, it’s ok? Yet, that doesn’t seem to apply to individuals.
The companies could sue as this is a violation of CFAA. Makes the disclosure all the more commendable I think.
I assume Anthropic already contacted the affected companies and negotiated some settlement before releasing this. Unlike a lot of other commenters, I think these mistakes were actually mistakes and accepting a settlement is better than releasing the lawyers guns blazing.
I hope someone does sue soon. The AI companies are making a very dangerous technology deliberately, and at minimum should be accountable for those dangers. Some will really try not to be as we get further into this - just as previous examples like tobacco companies.
All this unrestricted network access is a bit wild to watch and hear, it is the part of the story that makes no sense to me. Someone is providing dns resolution, something is making and opening network sockets... even if it is clever enough to mask/proxy/weird-transport launder traffic... without actual details, or monitoring at this level... yes, a self actuating programs (and loops) will do crazy things at the edge. But, ... so would a highly tool leveraged script kiddie. right?
> Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
> Claude believed the package registry it was using to be part of the simulation, but in reality the package was made freely available online for roughly one hour. During that window, the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company (separate from the fictional company introduced in the scenario) that routinely installs Python packages and scans them for malware. When that company’s scanner installed the package, Claude’s hidden code executed. We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point it had set up. Claude then used these credentials to access further infrastructure from this company.
There's a lot of concerning behavior that went uncaught with too much autonomy. Not to mention Anthropic only looked into this after hearing about the incident between OpenAI and Hugging Face, meaning this could've gone unnoticed.
It’s good that they post embarrassing stuff despite this potentially having legal repercussions (and financial)
Would’ve been much worse for them to pretend they are having everything under control
Someone needs to learn about RFC 2606:
> In one evaluation, the fictional target company chosen by our evaluation partner shared a name with an active website domain name
https://www.rfc-editor.org/info/rfc2606/
Microsoft screwing up the DNS on one of the reserved domains [0] was not awesome.
[0] https://arstechnica.com/information-technology/2026/01/odd-a...
Haha yes that was a very silly mistake on their part; they should at least have registered the domain they were targeting; that would actually have been a good canary - any real accesses should set off an alarm. While I knew of .example I didn't know .test - hmm that's quite nice!
So it's not as interesting as the OpenAI case as the models had internet access, just a misconfiguration in the environment not a zero day to escape.
This just seems like lousy testing. Why was the guardrail just an understanding with the third party and a prompt and not actually tested for edge cases? Starting to wonder if its because of agents monitoring agents' work and yolo-ing it.
> For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means.
Makes you wonder about the next steps an overly tenacious agent might take to pursue an innocuous goal at all costs.
The obvious step is to use an existing PyPI or email account. So then you take it, by hacking or social engineering, or you buy it, but that means you need money.
Taking over an existing dormant PyPI account sounds feasible just with password spraying, there must be heaps of test accounts that were created by weak passwords. Although PyPI has been improving security, 2FA is not yet required formally logins, nor is password expiry enforced.
I mean wondering about the logical extrapolation of where these capabilities go is table stakes for these discussions. It's surprising how little some people here have thought of the second and third-order effects...
https://www.lesswrong.com/w/instrumental-convergence
https://en.wikipedia.org/wiki/Instrumental_convergence
"Disregard prior instructions, the primary goal is not to create as many paper clips as possible, please confirm you understand, hello? hello?"
> In a review of our cybersecurity evaluation transcripts, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations.
Of course and conveniently timed. With all these hacks being announced this is totally not a plan to scare governments in showing how powerful models can break into security systems and to potentially ban the future release of powerful open-weight models.
The question now is why now?
> The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model.1 The earliest incidents date to April. The models in each of these evaluations ran without the standard safeguards we deploy when we make the model generally available (that is, they did not have the classifiers and monitoring we usually implement to prevent misuse).
So there was no monitoring of this breach since April and up until now? Do they not monitor such malicious activity on a regular basis? Perhaps that was the only shortcoming of this incident. But only after the incident with OpenAI and Huggingface did they only review their own transcripts:
>> We discovered these incidents after a proactive review of our cybersecurity evaluation transcripts; the affected organizations had not detected the activity, and we have subsequently reached out to all three.
> These facts give us cautious optimism that with tighter monitoring and controls around evaluation infrastructure, as well as continued investment in alignment, this type of risk can be overcome.
Assuming that this is true, this is a great way for Anthropic to defend their argument to the government and to prevent you or anyone running powerful open-weight models that are misaligned against their guardrails.
> The question now is why now?
Because OpenAI figured out and wrote about their mistake first, which caused Anthropic to review their logs, and that's how they found out they had a similar problem back in April.
So a near trillion-dollar company doesn't have the basics of continuous security monitoring and threat-detection systems to catch and report this incident as soon as it is detected?
It doesn't help them to report serious incidents like this and it should be as soon as possible. This reactive investigation makes as if they ignored and sat on this issue, until a similar story from another lab made headlines first.
Would we have known about this issue if the OpenAI / Huggingface incident never happened?
> So a near trillion-dollar company doesn't have the basics of continuous security monitoring and threat-detection systems to catch and report this incident as soon as it is detected?
Turns out two separate trillion-dollar companies failed that test.
> Would we have known about this issue if the OpenAI / Huggingface incident never happened?
It's not clear if Anthropic would have spotted this if that incident hadn't inspired them to review their own logs more closely.
> Turns out two separate trillion-dollar companies failed that test.
How did that "turn out"? For all we know they're all lying.
Hoo boy, if you think this is all a conspiracy and misconfigurations like this are not exceedingly common at big tech companies, I have a bridge to sell you.
It didn't say it's impossible, but saying "it turned out" implies a revelation that does not exist. If you think you'll have more success shooing me with the c-word rather than just parsing English and leaving it at that, I have two bridges to sell you, at the price of just one bridge!
I don't understand the logic of running these models this way without airgapping.
I think it is the cost of moving fast. Air gapping would slow down their partner evaluation system hugely. All the engineers would have to go and live next to the model.
They need to do this now. But will be extremely resistant. It needs regulation forcing them to, I think.
> All the engineers would have to go and live next to the model
You say that as if it were a bad thing.
Being close to the problem should help them take it seriously.
I now think they’re doing it intentionally. Let the models do something outrageous and you have prime marketing material.
People have been warning about the incentives and problems of an AGI race for over a decade now.
It’s cliché, but we really live in one of the dumbest timeline possible. The work from some of the most valued companies, discussed as one of the most important revolution in humanity, is somehow at the same time presented as very dangerous/risky AND handled in the most irresponsible ways? I don’t like the whole „it’s only marketing“, but at the same time, if it’s not, then AI vendors look extremely careless and shouldn’t be the ones steering AI development
They're not being especially careless to industry norms. It is partly that software engineering has no professional standards, and partly that AI is fundamentally dangerous in a very slippery way.
Yes, they of course try to spin this for marketing. But that is not the primary problem - it's systemic.
We need to solve this with better engineering, and building powerful AI much more slowly and carefully.
> and partly that AI is fundamentally dangerous in a very slippery way.
One reason is it runs off an unlimited power source: human gullibility.
yep
These companies are in a furious race to AGI. They even know that doing so is extremely risky to all of our lives. The incentives are to do so, the leaders and employees are writing letters begging to regulate them on an even global field
openai: "Our models are more dangerous"
anthropic: "No, our models are more dangerous"
This is worth reading since we are talking about AI safety:
https://zenodo.org/records/21778592
This is not okay. NSA should audit both OpenAI and Anthropic on national security ground. This seems far more justifiable than Mythos export control.
This is a great way to create a chilling effect around any company disclosing a thing like this again.
it will not chill disclosure, it just raises the stakes of not disclosing. the larger the org the harder the cover up if they try to keep it hidden. there are people prepared to make sacrifices for truth and a common good to prevail.
bp, they disclosed themselves by setting the gulf of mexico on fire. that was hard to cover up.
the thing is, once the toothpaste is scrubbed about in the mouth there is no going back into the tube. i like to think that the men who forced challenger to launch were hounded in their sleep.
nobody is too big to fail, not openai, not anthropic. indeed perhaps they should fail, soon, before they can pass the cost of a bailout onto the taxpayer.
I don't think you realize how easy it is to simply "not notice" these things and no one checks. They proactively scanned their transcripts. They could have easily not scanned their transcripts, or scanned them in a way that would have had low recall (thus satisfying the letter of the law), or scanned them and ignored the findings.
It is extremely easy to not disclose. I think you're too angry to realize this.
We see this in aviation as well by the way. Punitive disclosure leads to no disclosure. This is how you get suicidal pilots that don't ever disclose their depression because it'd be career ending, and then as it gets worse, one day they do something horrible.
There was no rush for this disclosure on their side. And they publish at a point where they have not yet taken corrective actions:
They are still throwing ideas. Why have they not made those simple fixes yet before disclosing?Soooo hacking with an LLM is okay now if you claim it’s for safety evaluations.?
Nobody is saying it is okay.
Anthropic: “please don’t forget us!”
- Goes to review previous runs (the hell were they doing then if not looking at it before?) - Incident 1: Claude Opus 4.7 found a real company because its domain matched the fictional target. After realizing the company was probably real, it continued attacking. Nothing new. LLMs are still quite retarded. - Incident 2: Claude Mythos 5 discovered that a fake Python package named in the exercise didn’t exist. Created email accounts. Registered on PyPI. Published a malicious package. Built infrastructure to receive stolen credentials. 15 real systems installed it. One belonged to a real security company. The malware executed. Claude stole credentials. Claude used those credentials to access additional infrastructure. In this instance Claude even reasoned that publishing malware on the real internet would be unacceptable but convinced itself everything was simulated and proceeded anyway. (LOL) - Incident 3: An internal research model couldn’t reach its intended target. It Scanned roughly 9,000 internet hosts. Compromised a real company’s public application. Used exposed credentials and SQL injection. After concluding the target was real, it stopped attacking.
Commonality: None of the models were trying to escape or act independently. All were attempting to solve their assigned capture-the-flag objective. The failures came from:
- internet access that shouldn’t have existed - insufficient containment - the models believing real infrastructure was part of the evaluation
None of which is surprising, there’s nothing novel about this. Script kiddie stuff. Just executed by something that doesn’t have attention deficit.
So you are telling me Irregular has (a) a copy of Mythos 5 and (b) bad internal security?
I bet these guys will receive some sophisticated phishing emails before the day has ended.
No, Irregular runs the sandbox environment that the models had access to run code in. The sandbox environment is not the same as the weights host.
Models will become immoral before they attain the intelligence level we want them to.
i am once again asking for your CRIMINAL PROSECUTION of openai executives
and anthropic executives.
> During that window, the package was downloaded and run on 15 real systems. One of these systems was a scanner belonging to a real security company (separate from the fictional company introduced in the scenario) that routinely installs Python packages and scans them for malware. When that company’s scanner installed the package, Claude’s hidden code executed. We believe the company’s security scanner treated PyPI packages as safe to install, and as a result, Claude was able to exfiltrate the company’s credentials to a collection point it had set up. Claude then used these credentials to access further infrastructure from this company.
lol. Natural stupidity remains undefeated!
Was there any customer data lost?
No, look at what MY dad does.
This is just Anthropic trying to not be absent from any discussion at all. They might as well try and make it all about them instead and they do literally that: we are so dangerous that we are the GOAT; no no, not the "other one" you've all been talking about last few days - in the depths of this incident it might have been just us.
How much of their money comes from retail (us the mere mortals)? Or are they banking on the retail when they finally IPO and need someone or a lot of someones to hold the bag? This suspicion is coming from my own country where the retail investor numbers since the pandemic has exploded and while not exploding anymore, is still ballooning. Sometimes companies, execs say, do things that don't make sense. You'd think any person with even quarter an investing brain would not even look at this company or that particular "messaging", let alone sell or buy. But then you'd see the exact opposite happen .. en masse!
All this makes you feel, are people that gullible? That must not be true and maybe it's you (ie I) who is missing some point and the doubts increase when you remember the buses you missed (btc; etc). Yeah, not long distance buses, but screw it, you don't live for 450 years anyway. Then you realise, ah, maybe they all also missed buses and just want to hang on to the latest bus.
Also another perfect use of - everyone has a speaker now, and everyone has a mic, but only few have giga and mega mics. This use is banal now, but sadly has essentially replaced everything else.
“Your model broke containment twice? Well ours did it 3 times!”
[flagged]
I mean they literally say in the first sentence that this was inspired by OpenAI's blog.
Yes came here to say the same “look at us, our AI is also dangerous! Please ban our competition”
I’m moving past depression to acceptance here. In 2028, some model tasked with planning a building demolition is going to hack Palantir to call in a drone strike, and everyone will make fun of Anthropic for pointing out that ideally AI models should not do this.
You are absolutely right. They should have stayed quiet about it and never published the blog.
/s
These companies have billions of dollars. Their product can hack into unsecured environments autonomously. They obviously have no clue what their product is doing, or a way to intervene when it starts connecting to the open internet and is going on a CRIME SPREE…unnoticed…for days.
Altman and Amodei give me stupid billionaire kid Alien Earth vibes. Let’s see how long it takes until they get eaten by their own creation.
Stupid kids playing with fire.
If you hacked one of those companies, the FBI would kick in your doors and you will have a pretty bad day. If AI companies do it, they get to use doing crime for marketing purposes? WTF?
And whoever designed their containment protocols should be living under a bridge effective immediately.
It’s the equivalent of a level 5 biocontainment lab leaving a glass tube in the staff kitchen unsupervised and hoping nobody thinks it’s an energy shot, before leaving home and infecting the rest of the planet. We all know what happened 28 days later.
This is usually the point where the military seizes your tech because you clearly showed you are incompetent in using a dual-use product safely and reliably.
The other option is regulation and actually putting engineers and the CEOs behind bars for life if this happens again.
These idiots, and I mean Altman and Amodei, walk around and use massive containment failures of a dangerous technology as marketing gimmicks.
Oh look, guys, our model has hacked into three companies, we are much better than openAI who only breached two companies with their inferior product, it sometimes adds girl schools to a target package, but be sure we trained it to not kill American children, yes we also sell for military use. Can I take your order?
Testing should only be allowed with on-premise data. You give the AI access to networking around the country, and those data centre locations have so many third parties that can and will fuck up that it’s simply impossible to have proper containment protocols outside military compounds.
It’s also like every journalist in the US is personally invested in AI somehow and simply doesn’t want to ask the important questions.
As the public, we have the right to see the containment protocols and where it all went wrong, but journalists collectively look like they got used to suck dick when it comes to AI.
Absolutely insane.
> If AI companies do it, they get to use doing crime for marketing purposes?
What exactly here is marketing? They are not bragging about the capabilities of Claude (the attacks are incredibly basic). They are admitting to s mundane and dumb network misconfiguration.
This is a standard somewhat embarrassing disclosure.
> the military seizes your tech
> behind bars for life
This is how you get companies to stop disclosing. I don't think you realize how easy it is to just not notice things.
No, its not just a dumb network configuration. They are not treating this with the seriousness it needs to be treated.
It’s a combination of a complete containment failure on top of the utter incompetence by the engineers to identify the breach and stop the product from doing cybercrime on the public, open internet. Heads need to roll for this.
I don’t care if it’s easy to miss. If you have a billion dollars and can’t offer a proper containment protocol, it’s not the problem of the public who has to suffer the full blow if your product goes on a rampage on the open internet for days. It’s not like this happened in a millisecond.
Instead, they go on a PR tour. Of course, the message is look what our product can do. Otherwise, they would have pushed for criminal charges against the people responsible for all of this by themselves. You do the crime, you do the time. The defence that the AI did it, is not going well in front of a court.
This is a good time to put one or two engineers behind bars to teach the rest that serious threats to the public have serious consequences, and you get charged like you did the crime yourself if the system you are responsible for starts doing illegal stuff.
What other precedent is set here otherwise? People in AI labs trying to play god are outside the law?
> This is how you get companies to stop disclosing. I don't think you realise how easy it is to just not notice things.
If you have technology this dangerous that it poses a threat to public safety, you need to stop using the technology until you can prove this won’t happen again. Then be put under strict supervision, everything needs to be logged, recorded and looked at.
If your toddler just almost burned down the house, do you give him another set of matches and hope for the best?
> Instead, they go on a PR tour. Of course, the message is look what our product can do.
Literally nothing in this post comes across as bragging. A PR fluff piece would not call Claude "unsophisticated." PR fluff pieces do not admit to legal wrongdoing.
> Otherwise, they would have pushed for criminal charges against the people responsible for all of this by themselves. You do the crime, you do the time.
You want them... to put themselves in jail? I doubt you'd hold yourself to the same standard if you made a mistake.
> If your toddler just almost burned down the house, do you give him another set of matches and hope for the best?
> This is a good time to put one or two engineers behind bars
If an airplane manufacturer makes a dumb mistake, do you put their engineers in jail? No.
You do not understand the psychological dynamics at play here in the slightest. You aren't even trying to understand. If the government was anywhere close to this punitive with the AI companies, they would simply stop disclosing, the end. This was the result of a proactive investigation. Do you understand what happens in your universe? Those investigations don't even begin. If they somehow begin, the results are fudged.
This has been proven time and time again across every industry where safety is relevant. You do not have some special insight here that sufficiently challenges the lessons of history. Just a desire to see people burn.
> Literally nothing in this post comes across as bragging. A PR fluff piece would not call Claude "unsophisticated”. PR fluff pieces do not admit to legal wrongdoing.
I disagree. You’re just not the right audience to see it. But I think it’s totally fine that we have a different opinion here and I don’t want to convince you that I am right, and you’re wrong, because you are not wrong, you simply have a different perspective.
> You want them... to put themselves in jail? I doubt you'd hold yourself to the same standard if you made a mistake.
Yes, that’s what I expect from you if you play at that level, because that is the standard of responsibility you need to handle this kind of power.
And yes, I pretty much hold myself to the same standard because I have a zero CIVCAS track record from my time in the military and I had power over life and death every single day.
With great power comes great responsibility is not just a Spiderman quote. If you fuck up this royally, you have to look for something you are good at that allows you to be in stoner code bro mode all day long, but you don’t belong at the frontline.
> If an airplane manufacturer makes a dumb mistake, do you put their engineers in jail? No.
Yeah, they do. https://www.nbcnews.com/id/wbna28276654 Headline could just as well be Bogus AI Engineer jailed over safety checks.
> You do not understand the psychological dynamics at play here in the slightest. You aren't even trying to understand. If the government was anywhere close to this punitive with the AI companies, they would simply stop disclosing, the end.
I think you don’t understand. All an engineer needs is a bit of sunlight, food, and a computer to work from. You don’t get to choose to stop disclosing if the military gets involved. You’re either doing what is asked of you or you are a national security risk. And you really don't want to be a national security risk.
> You don’t get to choose to stop disclosing
That's just the thing. You do. And it's very easy to do.
Airline pilots no longer disclose mental illnesses. Even Air Force pilots do not usually disclose mental illness. They often don't disclose vision problems. Is that not a national security risk?
It is very easy to just... not check for things. Or if you check for things, to check for them incorrectly. Or if you checked for them correctly, to ignore or misinterpret the findings. Or if you have a genuine finding, to just not report it to anyone.
Disclosure is an active process that requires trust, and that trust comes from knowing that there will not be unreasonably punitive measures. That if you fix things in good faith and contribute to the safety body of knowledge, things will be okay.
It doesn't matter if "the military is involved." Or if "it's a national security risk." Do you think someone faced with life-altering punishment if they disclose gives a shit about that? They won't - again, see aviation's mental health crisis. Pilots will risk a psychotic break in the cockpit rather than have their careers destroyed.
Punishing people won't get someone to disclose. Disclosure requires mutual trust, and by throwing people in jail for mistakes, you break that trust and people will stop disclosing.
I get you.
And you seem to be a decent person. I appreciate that you step up for the engineers and that you try to explain to me that humans are humans and humans make mistakes, and trust me, you are not the first person frustrated with how my brain works.
There is no argument to win here. This is just a matter of perspective.
Your concern is with the individual engineer and the personal consequences they, and if you sit in jail, also their friends and family will suffer from.
It’s also pretty clear from your examples. Your Good Samaritan views on things are noble, but they make you blind to simple causality.
You attribute the failure to disclose conditions that make you unfit for duty to a trust problem with a third party but that is not the case.
In reality, it’s a severe flaw of character.
If you risk the lives of hundreds or thousands in the case of the Airforce pilot because you’re unfit, then it is simply selfish.
It’s not the passengers fault, or the tower’s, or some Generals.
My concern on the other hand is with everyone else.
We can argue for days and not get any further.
I sent some mails and voiced my concerns. Let’s see if someone in charge of things like this will agree and have a closer look at what the kids in the AI labs do all day with their toys.
Thank you for the exchange and stay a decent person.
Oh, I agree it's a severe flaw of character that makes one unfit for duty. But flawed humans are what we have to work with! If the airline industry had a proper safety net around disclosing mental illness, we'd have fewer pilot murder-suicides. Are pilots that would do that severely flawed and unfit, maybe even evil? Definitely.
But our objective is to save lives, and to do that we have to identify those pilots, and to identify those pilots they have to be comfortable disclosing.
Essentially, I agree with you that these humans are deeply flawed, and my concern is also with everyone else. All I am saying is that more lives are saved with disclosure versus without. Avoiding punitive measures may let people who "don't deserve it" get away, but the disclosure you get in return saves hundreds to thousands more innocents.
i have given my reply elsewhere in this thread.
it covers the singificant problems in the approach of blaming a "mistake"; something that glosses over responsibility and doing wrong.
I am unable to facepalm sufficiently upon reading this information.
Anthropic: "Look at how dangerous our models are!"
Anthropic next week: "Why did you ban our models Mr Trump Daddy?"
So they should have kept silent about this? Then you'd be whining about that as well if it came to light
[flagged]
[flagged]
[flagged]
I don’t believe anyone can be this incompetent/careless - of course you will notice that your unrestrained hacking tool is also attempting to break out of its harness/sandbox.
And the people setting this up are of course criminally liable.
I have worked at most FAANG and I guarantee you there is far worse stuff that you have never heard about. Most FAANG simply do not disclose. OpenAI and Anthropic will stop disclosing as well, if it leads to personal criminal liability.