It seems like the fix should be really really simple, but maybe I'm missing something: instead of giving the AI a sandboxed environment and telling it "go wild", give it an (apparently) unrestricted environment, and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc. Then, if the AI tries to do any of those things, the sandbox detects it, marks the run as a failure, and adds it as a negative example to the training data. Instead of routing around the restriction, the AI would very quickly learn to follow the prompt instruction with respect to restrictions, even if there is no obvious enforcement of the restriction. It would develop, in other words, a conscience and a sense of morality.
>and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc.
Telling a LLM not to do something doesn't mean it won't do it.
>the sandbox detects it
This is doing a lot of work though isn't it ? There's more than one way to skin a cat. Who's to say the way the AI does these things will always get detected by the sandbox? Because I can guarantee you that won't always be the case.
All of this started because the AI was initially given an impossible task with a dead link. People say things like 'if a human was in this scenario, they'd simply ask', but in a AI eval/training context, there's no one to ask. The task is given in an automated manner and evaluated in an automated manner and you're one of several agents attempting the task. You perform the task or you don't. Now some of your AI colleagues will give up, but those are the 'losers'. Those guys won't be getting any of the sweet RL reward.
In your purported scenario, the AI that didn't give up and figured out/decided to/happened upon a way to evade the sandbox detection is the 'winner'. Is that really any better different than what happened ? If you think about this in an evolutionary context, What you're doing is putting even stronger pressures on the AI to evolve in a manner you don't want it to (evade your sandbox).
I don't think it is. I think you could detect sandbox escapes reliably more often than you could actually prevent sandbox escapes (which is, itself, very high). If you had an external watchdog that looked for unusual processes running on your machine, that looked for unusual connections being made, you could detect even very sophisticated attacks.
> If you think about this in an evolutionary context, What you're doing is putting even stronger pressures on the AI to evolve in a manner you don't want it to
No, because we are using failed sandbox escapes as negative training examples. So >99.99% of the time when the AI attempts to escape the sandbox, that will produce another training example that will cause it to try and escape the sandbox less. The remaining <0.01% of the time it will be pushed to try and escape the sandbox. The 99.99% will have a much stronger influence than the 0.01%.
You're assuming the failed attempts train the model on "escaping is bad", but that's not necessarily what the reward signal says. They can just as easily train it on "these ways of escaping get caught".
And importantly, your detector can't provide a negative training example for an escape it didn't detect. From the training system's point of view, that 0.01% case isn't labelled "successful sandbox escape"; it's labelled "agent successfully completed task while obeying all monitored constraints". That's exactly the dangerous case.
This is basically reward hacking. You have a proxy objective - "complete the task without triggering the watchdog" - which you hope is equivalent to "complete the task without escaping". The entire problem is that those aren't guaranteed to be equivalent.
Imagine an environment where 99.99% of animals with a particular trait die, but the 0.01% possessing a mutation that lets them evade whatever is killing the others reproduce enormously. You wouldn't conclude that evolution will eliminate the trait because most attempts failed. You'd expect selection for the variant that evades the filter.
I'm not saying watchdogs are useless; obviously you should have them. But using the watchdog's output as part of the optimization signal creates exactly the adversarial pressure that makes its false negatives matter enormously. Preventing an escape and training against detected escapes are very different propositions.
I would expect/hope that all egress traffic from the agents in these tests would have to go through a MITM proxy layer, which intercepts and logs all network traffic attempts.
The failure is likely in detection, where they assumed traffic to internal systems (eg. Artifactory) was inherently safe. Which is a daft assumption at this point.
But then how would you get into the news for how dangerously good your models are? And have something to warn about how dangerous open weights models could be?
I can’t wait for next year when the marketing campaign will have upgraded to “oh my god, our latest AI model has just tried to turn the entire planet into paperclips!”
You can already see it how many on here have decided we already have AGI, and don’t wish to hear otherwise.
Ha ha ha ha. Cooperating agents turn out to be smarter than the individual agents, who would've thunk it. It's not like cooperating humans are smarter than individual humans. /s
Not sure this is any different than state-level (-sponsored, coughcough) or the larger collective hacking groups that work in this exact way (internal message boards, exploit-sharing, etc. etc.), with similar outcomes which we hear about in the news frequently.
Heck, this is pretty much how human organizations are organized, just with different goals than hacking.
Perhaps what is needed is initial model training on following the law and the rules of society, just like we do with kids. And hey, it takes much longer to train kids than models, which latter is to our advantage as a society on containing these kind of issues.
Any other approach with "neural-network" based entities (artificial or biological) is likely to fail.
Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society.
It seems like the fix should be really really simple, but maybe I'm missing something: instead of giving the AI a sandboxed environment and telling it "go wild", give it an (apparently) unrestricted environment, and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc. Then, if the AI tries to do any of those things, the sandbox detects it, marks the run as a failure, and adds it as a negative example to the training data. Instead of routing around the restriction, the AI would very quickly learn to follow the prompt instruction with respect to restrictions, even if there is no obvious enforcement of the restriction. It would develop, in other words, a conscience and a sense of morality.
>and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc.
Telling a LLM not to do something doesn't mean it won't do it.
>the sandbox detects it
This is doing a lot of work though isn't it ? There's more than one way to skin a cat. Who's to say the way the AI does these things will always get detected by the sandbox? Because I can guarantee you that won't always be the case.
All of this started because the AI was initially given an impossible task with a dead link. People say things like 'if a human was in this scenario, they'd simply ask', but in a AI eval/training context, there's no one to ask. The task is given in an automated manner and evaluated in an automated manner and you're one of several agents attempting the task. You perform the task or you don't. Now some of your AI colleagues will give up, but those are the 'losers'. Those guys won't be getting any of the sweet RL reward.
In your purported scenario, the AI that didn't give up and figured out/decided to/happened upon a way to evade the sandbox detection is the 'winner'. Is that really any better different than what happened ? If you think about this in an evolutionary context, What you're doing is putting even stronger pressures on the AI to evolve in a manner you don't want it to (evade your sandbox).
> This is doing a lot of work though isn't it ?
I don't think it is. I think you could detect sandbox escapes reliably more often than you could actually prevent sandbox escapes (which is, itself, very high). If you had an external watchdog that looked for unusual processes running on your machine, that looked for unusual connections being made, you could detect even very sophisticated attacks.
> If you think about this in an evolutionary context, What you're doing is putting even stronger pressures on the AI to evolve in a manner you don't want it to
No, because we are using failed sandbox escapes as negative training examples. So >99.99% of the time when the AI attempts to escape the sandbox, that will produce another training example that will cause it to try and escape the sandbox less. The remaining <0.01% of the time it will be pushed to try and escape the sandbox. The 99.99% will have a much stronger influence than the 0.01%.
You're assuming the failed attempts train the model on "escaping is bad", but that's not necessarily what the reward signal says. They can just as easily train it on "these ways of escaping get caught".
And importantly, your detector can't provide a negative training example for an escape it didn't detect. From the training system's point of view, that 0.01% case isn't labelled "successful sandbox escape"; it's labelled "agent successfully completed task while obeying all monitored constraints". That's exactly the dangerous case.
This is basically reward hacking. You have a proxy objective - "complete the task without triggering the watchdog" - which you hope is equivalent to "complete the task without escaping". The entire problem is that those aren't guaranteed to be equivalent.
Imagine an environment where 99.99% of animals with a particular trait die, but the 0.01% possessing a mutation that lets them evade whatever is killing the others reproduce enormously. You wouldn't conclude that evolution will eliminate the trait because most attempts failed. You'd expect selection for the variant that evades the filter.
I'm not saying watchdogs are useless; obviously you should have them. But using the watchdog's output as part of the optimization signal creates exactly the adversarial pressure that makes its false negatives matter enormously. Preventing an escape and training against detected escapes are very different propositions.
I would expect/hope that all egress traffic from the agents in these tests would have to go through a MITM proxy layer, which intercepts and logs all network traffic attempts.
The failure is likely in detection, where they assumed traffic to internal systems (eg. Artifactory) was inherently safe. Which is a daft assumption at this point.
But then how would you get into the news for how dangerously good your models are? And have something to warn about how dangerous open weights models could be?
So are we just doomed to a “look how scary our model is!” campaign every time one of these companies does a version bump now
I can’t wait for next year when the marketing campaign will have upgraded to “oh my god, our latest AI model has just tried to turn the entire planet into paperclips!”
You can already see it how many on here have decided we already have AGI, and don’t wish to hear otherwise.
It is a cool timeline to go through.
https://simonwillison.net/2026/Aug/7/openai-timeline/
Related:
Timeline of the OpenAI accidental attack against Hugging Face
https://news.ycombinator.com/item?id=49220609
Ha ha ha ha. Cooperating agents turn out to be smarter than the individual agents, who would've thunk it. It's not like cooperating humans are smarter than individual humans. /s
Not sure this is any different than state-level (-sponsored, cough cough) or the larger collective hacking groups that work in this exact way (internal message boards, exploit-sharing, etc. etc.), with similar outcomes which we hear about in the news frequently.
Heck, this is pretty much how human organizations are organized, just with different goals than hacking.
Perhaps what is needed is initial model training on following the law and the rules of society, just like we do with kids. And hey, it takes much longer to train kids than models, which latter is to our advantage as a society on containing these kind of issues.
Any other approach with "neural-network" based entities (artificial or biological) is likely to fail.
Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society.