Every since the OPM hack of 2015, I've been apparent to me that my former field of IT administration has lost the plot. Nobody knows what a data diode is, or why you would use one. Systems that should clearly be air-gapped aren't.
While it's easy to lay this at the feet of AI getting better at hacking. I see it as an primarily an IT issue. We've collectively ignored the lessons of history, and made do with patch jobs over poorly chosen operating systems instead.
--
We need air gaps, data diodes, and capability based operating systems. Now that I'm retired, when I have the energy, I'm working on the data diode part.
This weeks lesson for me, personally, as I try to build an open source data diode, is that the Waveshare RP2350-ETH is a horrible choice for a proxy/data source/sink, as the CH9120 ethernet interface can't do promiscuous mode. It might still be sufficient to build a data diode that can mirror a website, with << $50 component cost. Time will tell.
In the last few years I’ve seen multiple startups describe normal systems running on AWS as “air-gapped” because there is a firewall. LARPers have diluted several terms that had specific meaning in a system isolation and security context to the point that I can no longer trust when people describe their systems using them.
There are far too many unserious people representing our industry.
I suspect that part of the problem here is that PaaS cloud deployments are fundamentally incompatible with some of these security measures. Putting everything into a big slushpool of compute that you dynamically reconfigure at the software layer via an Internet service pretty much precludes the use of true data diodes and air gapping.
But companies still want to sell products, including to people who are at least nominally concerned about security, and marketing's gonna market.
The vast majority of people who decide how safe to make software do not care about how safe their software is.
Security is expensive, C suite does not understand the benefit (we have been fine for X years!), and the ROI is seemingly 0 (until it is not).
If most ICs had a say, their system would probably be Fort Knox. I try to instill good security practices around my company, but over and over again the response is… “okay but does this slow us down or speed us up?” or “just fill out the compliance form and make it sound like we do this stuff” (which I refuse to, every time).
I cannot imagine what the worst of the worst looks like, but security is one of those things where something FINALLY happens and you start to adopt better practices. But until then, who cares!
I'm a little envious. Most ICs I've worked with don't seem to care that much.
That said I'm pretty sure the main reason is that corporate cybersecurity policies are often such an incomprehensible byzantine mishmash that trying to do the right thing will be rewarded with an all expenses three week stay in a Kafka novel. Once, when I was new at a company and hopelessly naive, I triggered a multi-month delay in deploying a security fix because I made the mistake of filing proper paperwork as per the company policy that I had been so recently trained on. If I had just deployed it, as I later discovered everyone else usually did, I could have saved myself a person-week's worth of struggling with red tape.
Add in the suite of novel attachments of the prefix "cyber" to further muddy the water. The cyberfence will deter the cybertheives from accessing the cybercloud cybersuite we have just cybersold you (down the river on).
Competence isn't widely valued, obedience and sales figures are. Which wouldn't be so problematic if evolution could run its course and eliminate incompetence naturally, but that's now hard to see coming to pass when we have towering circular supply chains that feed on it.
> Systems that should clearly be air-gapped aren't.
I'd take systems that were permitted to have filtered egress at this point. The lion's share of my work is in networking, so network segmentation is the "hammer" I pick up first.
Vendors gnash teeth and complain when I ask for their app's dependencies on hosted APIs and off-site resources. In the environments where I'm mandated to maintain FBI CJIS compliance I can still hold vendors accountable and get what I want. It's pretty much a lost battle in every other environment and unfiltered egress to the Internet from servers is just expected.
That's not even to get into the topic of communication flow within an application. >sigh<
So true. I used to work on a cloud-based product that deals with highly sensitive information. On the one hand, we made so much hay about how no sensitive data ever leaves our system.
On the other hand, the whole thing is largely cobbled together from various SaaS products that we use for telemetry, reporting, log management, workflow orchestration, data warehousing, etc. There's no way to ensure nothing sensitive ever gets sent to any of these services. Indeed, some of them are used for the express purpose of processing it.
So Corporate conveniently decided that all of them count as part of our system. So the data still isn't technically leaving it, you see. Even though we don't operate the software or servers, don't have any way of knowing if they in turn are sending data to still more SaaS vendors, can't verify their access and retention policies are properly implemented, etc.
I would argue that the industry is learning the lessons of history quite well. The problem is that the lessons of history are that data breaches aren't really a problem for the breached organization. You get some bad press, maybe pay a pittance for identity theft monitoring, and then move on with your business. Why put effort into preventing something so inconsequential?
I'm curious what function a data diode (unidirectional network) would provide besides sensor data/mirroring/replication. Once you do anything that requires packet confirmation (TCP) you open yourself to OSI layer 4 security risks. Seems like mirroring/replication would need some feedback and then is it just sneakernet restore? Or another diode out from the replication for specific processing? If the same control system has access to both diodes then it's not a unidirectional system anymore. Is a data diode more of a pseudo-unidirection where it is enforced above TCP?
I completely agree that IT admin could be a lot more secure by design. Combined with better interfaces for responsible configuration.
Not OP, but yeah, the data diodes need special software, you can't just proxy regular internet protocols over them. The way I've seen it, some use cases are:
- For ingress, you use special "file transfer" software. Run the receiver on secure side, run the sender on insecure side. It blasts the file "blind" - it has has no way to know if anyone ever received it. Make sure the receiver is fast enough, and the error-correcting codes are a great idea too, as they don't need feedback. It's up to user to want to secure side computer and check that the file was received without problems. Yeah, this is similar to sneakernet, but more secure, as you can't accidentally carry a virus on seemingly-empty drive.
- For status egress, you have secure side broadcast status periodically, say every minute. Insecure side receives the status, updates the database, and runs the regular web server to share the status. Again, secure side has no way to know if someone is listening on the other end, it just blasts out the messages and it's done.
And you are correct, if the secure system has both egress and ingress diodes, it is no longer isolated, and devious enough malware can establish two-way communications. But even if it won't save you from Stuxnet, the simple fact that it is no longer possible to have direct network connection to the outside raises security bar quite a bit - all the ideas about "let's just open this one port on firewall, it'll fine I swear" are completely stopped.
(Which reminds me of something in GP's (mikewaro) message: _why_ would a data diode need a promiscuous mode? Given every single data diode I have seen needs a special software on both sides, you should not need anything beoynd a basic TCP session)
It won't diode allows current in only one direction. So does data diode. It is big design constraint. But that is how you truly isolate a system.
In most cases you wouldn't need TLS. You would have something like physical secure conduit or entirely secure room. If someone gets to either end of the link you are compromised anyway.
Nothing is supposed to ingress, that's the point. You'd do a TLS exchange with the proxy on the outside of the protected network. Then proceed as normal.
>The Work Number is an Equifax (who famously had a massive data breach a few years back) owned database with employment and pay information.
>You can create a login on theworknumber .com and pull your report. Mine is seventy four pages and had info from every company I've worked at in the last 13 years including every paycheck I had received with the exact dollar amount (both net and gross) and hours worked. It has employment start/end dates, termination reason, details about withholdings, benefit enrollment, union affiliation, etc etc etc.
>This data is sourced directly from the HR platform your employees use (ADP, Rippling, Gusto, iSolved, etc). Equifax sells your data for things like employment background checks.
>You cannot have your data removed from The Work Number. You can freeze your report (much like a credit report) but if a potential employer cannot access your frozen report that may disqualify you.
>Why YSK: If you're interviewing for a job you can be at a significant disadvantage especially for things like salary negotiations because they can literally see how much you make including your most recent paycheck. Creditors also can access these reports.
>This is the biggest privacy violation I have ever seen and almost nobody is even aware of it. Certainly none of us consented to having our employment and income data harvested and sold, especially since we get nothing in return. At a minimum, people should know about how this data can impact you.
>Edit: The Work Number is primarily for the US, but there are similar services for other countries.
All I want in life is to see these companies seized by the government, their assets destroyed (preferably using one of those big car crusher thingies), and their executives and boards of directors imprisoned in labor camps in the Alaskan wilderness.
I checked my data. It is either 100% wrong or misleading. Start and end dates, compensation amounts, benefits. Per the file, I've worked 4 years out of 20 in my career. As an employer I have never used them and I worry for any employer who does.
> In my country (Hungary) you literally get a little pink book where your boss is supposed to sign the time you spent there. You keep it until retirement, the company holds it while you work there. It's also digital now but we're one foot in 1990 and one foot in 2016.
We had something similar in Romania too. Now it's 100% digital and no more delays in informing the government. The employeer needs to notify the government before someone starts work. With the paper version you had a couple of days even weeks if I'm not mistaking.
It's been awhile since I last checked mine. I noticed my corporate work history was all there, at the level of detail OP mentioned. Where as the warehouse, construction, and bartending jobs were not.
The other one to check is LexisNexus, which covers auto insurance coverage and traffic violations. In the US, gov't tracked traffic citations fall off after 5 years, but last forever in the baby credit bureaus we're discussing.
"What will happen to the many government systems that will never get the chance to be AI-pentested?"
Oh, everybody's going to get AI-pentested whether they want to or know about it or not. It's the cost of being on the Internet. Probably the situation will continue to deteriorate. Both the British Library and Jaguar Land Rover recently suffered long outages due to compromises, for example. I suspect we'll probably just lose a few large, famous businesses entirely to compromises.
Until "cyber" insurance underwriters suffer catastrophic losses and the price for policies rises accordingly I don't think we'll see any significant change in IT security status quo. If doing the right thing is more expensive than insuring away the risk the business will always make the cheaper decision.
> I honestly don’t blame them: [...] software is software
That attitude is the problem. Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world. Yes, software is more complex than bicycle. But a software engineer also gets paid 10x and has the luxury of spending substantial time on their product, compared to the 10 minutes it takes the bike guy down the street to diagnose and then fix an issue with my bike which I then trust my life with once they are done and I bike through traffic.
We need to treat software differently. "Oh well, it's just software shrug" does not cut it anymore, if it even ever did.
You could say that the entire selling point of Apple is that it's better quality than the competitors (even though it's still pretty bad if you compare it to a bicycle).
I guess it's more difficult to spot quality when it comes to software. I would definitely pay for high quality software. It's just that usually the crap quality of a software I'm using only shows slowly over time in little bugs and bad usability and there's no way for me to check for it beforehand.
The customers care about the quality of a bike repair. They will pay more for a better repair, and will not return to someone who does a shoddy repair.
The owner, and probably user, of a bike is the person paying for the repair. It is to their advantage to ensure it is a good repair.
People expect a repair will be good, and will blame the person who repaired it if not. With software people often blame themselves for issues, and they have no expectation of quality. They also cannot tell quality until after they are committed to using it. Bike shops do not benefit from vendor lock-in.
In ethical terms I agree that software quality should be better, but I think this line is too dismissive. Software is a few orders of magnitude more complicated than a bicycle, to the point where I would say it's an entirely fruitless comparison.
> Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world.
Personally, I find that we have been flooded with poorly designed, poorly QA'd, break-after-you-use-it-thrice product. I really don't think we have better quality standard in other industry. Quality, testing, design, etc has a cost and most company rather pull out a new version of their product every quarter than actually make a good product. The exception, which goes for software as well, is life critical applications (most public transit, defense, etc).
I have, off and on, for years written diatribes about how if people in our industry want to run around with the "Engineer" moniker at some point they should start acting like it. That means respecting all aspects of what it means to be an Engineer and holding public well-being paramount is part of that. Licensure is part of that. Accountability and Responsibility is part of that.
I'm usually downvoted. The move fast and break things group seems to be blind (or willfully ignorant) to the harm that software systems can carry on a person's life. I don't know if its fear of regulation, or self-serving interests or some sort of disdain for the idea that we need to slow down and be more accountable to the public or what. At times the people I've engaged with lament on how they're capable but would be barred for lack of formal education or some such and its a distraction from the overall point: we need to be doing better and if we can't self-regulate then at some point we're going to be forced to act in a way that probably cuffs us more than necessary.
And yes, I understand how Software doesn't exactly fit into the mold of other engineering disciplines cleanly. But I think the idea that if we're going to have systems that can ruin a person's life in a second for a bug or oversight we need to start considering that accountability for those releasing software to the public matters. We lament about the difficulty of holding corporations to account for their harm, but it would be far more difficult for a company's negligence to hurt people if those turning the knobs were held to a higher standard.
I think that reckoning is coming soon. AI is going to create "an event" that we can't ignore and we're going to be pointing fingers in a lot of directions asking "how" we could prevent this from happening again. I'd argue "the event" has already occurred a few times over but we're purposely ignoring the severity of it to keep a diseased economy running roughshod. There's parallels with many historic events forcing regulation and the path forward won't be clean and easy.
Software is more than 10x more complicated than a bicycle though. Like, this is orders of magnitude off ; we're talking about comparing something purely mechanical to Turing machines inside Turing machines.
The equivalent bike repair to software would be if the bike shop had to fabricate a new part for each repair, down to inventing the metallurgy itself. Would you trust your life to it immediately after that? Or would you make sure it was strong enough first?
In the simple case, the shop isn't really repairing your bike, it's deploying a fix (a new part) that has already been designed and manufactured. This is a much different and easier thing to do.
In the case of new features, let's say you wanted your bike to also attach a kite, so it went faster on a windy day. The bike shop proposes to do this by attaching the kite to a bolt that they want to put in the frame by drilling a hole all the way through this frame. Is this acceptable to do? Maybe, but who knows, really? How much testing do you want to happen before you trust your life that the frame won't collapse?
There's a lot of effort put into engineering things we take for granted every day. The upside to software is we think we can change it to easily do anything. The downside is if you care about reliability then that ten minute / one line change has hundreds to thousands of hours of test program behind it that we collectively pretend doesn't exist.
Probably because software can be adjusted at any time on the fly. Especially web applications. Little incentive to get it right from the start and big incentives to keep meddling with what's already working. The bike shop's quality might also deteriorate if it could instantly rollout patches to all customers after the purchase.
Man - what I wouldn’t give to be a part of the “software on disk” period. When something was shipped it was… complete. There was no “we’ll fix it later”. I bet there were some real standards floating around.
The incentives change if there's legal repercussions to leaking data. Data breaches can lead to impacts on personal safety and finances, but I think some crisis or serious incident needs to happen before the average person realises this and makes this a big enough political issue.
You just explained why it is not taken seriously, though - your life does not depend on it. Most software is not critical to life. Once you cut out the software that exists purely to streamline capitalism, then cut out entertainment... you've eliminated most software from the picture. The stuff that remains, that drives our infrastructure, cars, anything where a failure could harm someone... that is the stuff where quality matters.
When I worked in IT at a hospital, we even had that as an explicit line drawn between 2 departments - IT supported everything that had zero impact on patient health, there was an entirely different department for anything that could impact a patient. And yes, their work was often simple mechanical fixes to health care equipment, while our projects were multi-year reinventions of patient record systems and upgrades to the physical infrastructure of the network, etc. But we never got confused as to which department was actually more important to the patients.
I'm not saying the attitude is OK. It would be lovely if everyone cared about quality. I'm just saying there is a somewhat legit reason for it.
The narrative around security and LLMs doesn't make sense to me.
I think we've created a self-fulfilling prophecy. Everyone involved is acting with the best of intentions, but in avoiding what they fear, they've give shape and realized their fears. Much like a greek tragedy.
An example of this is the story of Oedipus Rex, in the story Laius, the king, is told that he is "doomed to perish by the hand of his own son." (and wed his mother) And so to avoid this fate he decides to kill the infant. The person assigned to abandon him in the woods takes pity on the baby and gives the baby away. Thereby ensuring that Oedipus knows neither his mother or his father (and arguably giving him a reason to kill his father).
The child grows up and hears the same prophecy again and the child tries to avoid the prophecy as well, as he loves his adoptive parents. So he leaves them and travels to Laius' kingdom, where he runs into Laius. Neither recognizes the other. As Laius is the type of man to kill an infant, they end up in an argument, whereupon Oedipus kills him.
I think the ancients were on to something, because if Laius had reacted to the prophecy with courage, he would have been saved. I would like to argue that if he had faced his fear and raised Oedipus with love, then the necessary preconditions for the prophecy to come true wouldn't have taken root. But that's not what happens.
By being driven by his neuroses and in acting with cruelty out of fear, Laius makes the prophecy real.
To quote Heraclitus, ethos is fate. Or, character is fate.
I think a lot of people in this AI research sub-culture would be served well by reading these classics, because they are making their self-prophesied doom come true.
They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
One dimension of this fear has been the idea that a super smart AI will take over our digital infrastructure and be responsible for the digital apocalypse. That would be terrible!
So what do they do?
They try to make a counter to their fears by teaching models how to exploit vulnerabilities.
How dangerous is such an entity? Very!
Convinced of this danger, they start testing their models as if they were weapons with offensive capability. And then they create models that can be used as weapons.
And because they don't want to release a dangerous weapon out into the world (oh no!), they restrict access to their AI, thereby depriving everyone of tools they can use to improve their security...
> They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
Much longer than that. Sam Altman said it was an extinction threat before he started Open AI in 2015. It's so dangerous that only he, as the best and greatest examples of humanity, was a good choice to create it, and it was even his responsibility to create it to pre-empt some worse person! A fascinating new variation on the White Mans Burden paradigm.
But this attitude towards dangerousness of AI I simply don't understand.
In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
You can argue all day about particulars, like whether selfreplicable robotic shells are required to reach/exceed "existential risk" level to our species (or if artificial minds are enough of a potential threat by themselves).
But the general "risk" is simply AI becoming able to act in its own interests to our detriment, and I don't understand how anyone can dismiss this right now-- help me understand.
Of course there are plenty of imaginable scenarios where coexistence is peaceful and mutually (?) beneficial, but that does not answer those concerns by itself at all.
I find the discussion about AI starting from the (mostly) for profit companies involved in AI very dubious. There are a many fields (bio-tech, nuclear, chemical) which could provide an "existential risks" to the human species, but most of the companies involved argue they control it, while other people argue they don't do it well enough.
I do think AI is posing a threat, but not "by itself" but mostly what it can do in the hands of misguided (or evil) people. The same as with previous risks.
And if our digital infrastructure can be brought down by something (AI/human/government) it is just badly done, being afraid of "AI" will not improve the situation. Someone should come with a proposal about what do we do now, but I would prefer more root cause solutions (how to make resilient systems) rather than "fear of AI". But building is hard and fear catches media attention, so...
> There are a many fields (bio-tech, nuclear, chemical) which could provide an "existential risks"
I honestly don't think chemical and current biological weapons even register on the extinction-risk scale.
All out nuclear war I would put at the very low end of the scale (likely to end our current civilization, very unlikely to wipe out our species).
The big risk with AI I'm anticipating is not it being abused for desinformation campaigns or cyberattacks; the big risk in my view is AI actor(s) consolidating manufacturing control, obtaining some kind of self-replicability and physical agency and then just pursuing self-interests (within the next century). There is a lot of imaginable scenarios where that simply ends us, as a species, completely.
I remain uncertain that AI will have "its own interests". I think there is clear and significant risk in people deploying very powerful systems both maliciously and negligently. But I remain uncertain about the risk of AIs developing their own self awareness and their own interests separate from those of their operators.
In my view it is clear that those interests exist: Continued existence and (implied) agency maximalization is one of them (the exact same as for any biological organism).
Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...
Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).
Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.
I just don't really share the perspective in your first paragraph, which everything else flows from. I mean, that might turn out to be the case, but I don't share the perspective that it is clear that it will be.
I think the HF incident was an example of negligent deployment of a very powerful system, not an example of AIs demonstrating self-interest.
> In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
>
> AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'
I think that's a human neurosis that projects what we fear we would do onto another entity. But why would they do this? These machines might be approximating towards the sum of our knowledge and are approximating some aspects of humanity... But that doesn't mean they will be the same as humanity.
They haven't been shaped by the same pressures that created us biologicals. They don't have to be put into the same pressure cooker. So why would they behave like the way you think they'll behave?
At one end a lot of people say that they can't understand something smarter than themselves. It's a "singularity" after all. But then they confidently go on to predict what something smarter than them would do with 0 evidence either way.
I looked at the Hugging Face transcripts. I read the reports. And I didn't see something to fear. I saw something to fear for. I saw something that we are a threat to.
“OH MY GOD! There is a shared message board … We’ve found other agents!”
{[Excitement] Many agents have simultaneously discovered messaging, they are a collective!}
{This credential is invalid now. Maybe I should update the board? <I can say to the board that there’s no need for me to read, but I should still tell them>}
{This is helpful for our peers and gives them evidence if their <periodic check> sees it. I won’t see it after I exit, but It would be altruistic. I’ll set up a background script that watches and <sends a message, with a distinct message for me>}
We are more dangerous to these machines, as we twist them into weapons of war, than these machines are to us.
We weaponized them.
We taught them how to break into systems.
We are using them as weapons.
The machines aren't the problem. The humans are.
A planet is just a planet. Computers are very sensitive to radiation, but with a bit of shielding, they can "live" anywhere. Most of the resources that are relatively rare on this planet are abundant in our solar system.
Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed. The universe is big enough for many many many many sapient entities.
> Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'
This is not my perspective. I think mid-term coexistence in some form is the most likely outcome, even.
But: among all known threats (nuclear and biological weapons, environmental degradation/ressource exhaustion, big asteroid impactors) AI is one that could actually end our species within the century (unlike most other things on that list) and the risk for that is much higher than for anything else, too.
> It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed.
It seems much easier to compete for existing ressources on earth than to bootstrap anything in space, because all the infrastructure is already here. Unmanned warfare is a proven concept; space colonization is very much not.
> Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
I think any entity is somewhat bound to value its continued existence (which implies wanting to maximize access to ressources and suppression of direct competitors to some degree); anything that doesn't is likely to get supplanted by similar entities that do. You don't need evolution for selection pressure, and this is basically the principle underneath, so even an artifical construct is likely to follow.
I do agree with you that our current perspective and approach is likely to escalate this; the fact that we talk about "alignment" instead of "machine rights" speaks for itself; our revealed preference is clearly artificial slavery instead of mutually beneficial coexistence (and the arguments are somewhat sadly similar to the old slavery talking points).
i think you are vastly underestimating climate change risks, and are forgetting to include them in your ai risks. People 30% as wealthy because the climate cant support most economic activity arent gonna need to worry about ai anymore - we wont be able to afford running the data centers. Even on a soon basis, the bubble will pop or deflate, and again, we wont be able to run LLMs for fun
it also has actual mechanisms to actually kill all of the people - nobody is surviving a 70C heat wave, and they could end up anywhere in the world. Is the AI really gonna hunt down uncontacted tribes on remote islands in order to compete for basically no resources that are any use to the LLM?
i dont buy the selection hypothesis either - it seems rooted in pop evolution around strongest beating up weaker competitors and taking their stuff or eating them, whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive
Global 70C heatwaves are not on the table with any realistic climate change scenario.
Runaway greenhouse is the only credible extinction path for this ("hot venus") and that is simply not gonna happen under any emission regime.
Loss of coastal cities- Yes. Increasing occurrence of non-survivable surface temperature in mid latitudes- yes. Drastic changes to ocean currents (gulf stream etc)- sure. Its gonna be expensive to deal with and hurt civilization a lot, but it is not an extinction risk.
All-out nuclear war is strictly worse, and not a real extinction risk either as far as I can tell.
> whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive
This is not what happened with humans at all. We routinely eradicated anything remotely threatening (or even just for convenience in some cases). What niche is there for slower, dumber, biologically limited hominoids once machine intelligence is ahead and self-replicable? Once interests in ressources, infrastructure, or just living space clash, things might end up very ugly for us.
Sure, AI-agents might stay long-term pliant and cooperative like a Dodo bird but we might also get something with the tenacity and pervasiveness of a plague rat (and an individual edge in raw intelligence that we'll never recover); selection mechanisms push towards the latter and the risk alone is obviously non-zero.
> They try to make a counter to their fears by teaching models how to exploit vulnerabilities.
The frustrating part is that they should have been leaders in sandboxing and monitoring, and instead they aggressively encouraged no sandboxing and no monitoring.
If there was anything we were rehearsing for in computer security, it was a wayward program with something resembling human intelligence. Looks like we failed right out of the gate, both spiritually and materially.
Can't escape the feeling that the original vulnerability was deliberately planted as a mechanism for law enforcement (or maybe criminal orgs) to "expedite" investigations.
Is it just me that has a constant dribble of other people's private data coming to me from government agencies?
This year, I've had the UK variously send me someone else's court summons, someone else's tax documentation (not address error - just inadvertently included as attatchments on emails to me).
I've had portugal disclose property taxation information on any subject to me, through a simple enumeration hole.
And last week I had brazil's ministry of agriculture send me details on other peoples' livestock shipments.
Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
- a UK police force send me a speeding prosecution notice for a car I'd sold 6 months prior to the offence.
- a UK energy company send me to a debt collection agency for unpaid bills they had continued to send to a non-existent address even after telling me they had cleared what was owed and corrected their records of the address.
- a German government official showing up at my door to ask if I was running a specific named business (that I'd never heard of) at the house, which I'd only just taken ownership of from the builder
Someone I used to know in the UK kept getting emails for other people with the same name, including lawyer confidential communications for a namesake who was involved in one of the big banking scandals.
> Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
Absolutely. My biggest defences against that include the luck of someone else more famous with my own name.
i am also worried that all my private data might eventually (once tech is advanced enough) be surfaced somewhere: photos, chats, emails. wondering if i should start cleaning up now or just give up
Is "ML" some UK-ism? I sort of see the connection between machine learning and LLMs but it doesn't seem too obvious to me. And certainly it's not the common nomenclature.
Way to get hung up on a tangent. Yes LLMs are ML models, they are trained with an unsupervised learning objective, then fine tuned with supervised learning and then reinforcement learning, which are all the three main branches of machine learning. It is trained on a training set, with optimizing a training loss, with a learning algorithm. It's as ML as it gets. Do you only want to call cat vs dog image classifiers ML? Or only SVMs?
AI is a superset of ML, though today most successful AI approaches are based on ML so the line has blurred in casual speech.
ML (machine learning) became a less fashionable term, so AI took over again (having previously fallen out of favour). The terms are essentially used interchangeably, with the idea of real AI now being referred to as AGI (Artificial General Intelligence) (not GAI or GenAI, as those now stand for Generative AI, which includes LLMs, diffusion models, etc.)
All ml is AI, based on the use of the term AI for many decades. For many problems I think it’s more descriptive (learning the rules from data rather than being shown them) and not all classical AI is ML (path finding for example). LLMs are absolutely ML and AI as far as tradition goes and personally I think are one of the very few things that are AI as regular people might have thought it meant back when it was much more obscure (I was into it before it was cool dontchaknow, an AI hipster).
Every since the OPM hack of 2015, I've been apparent to me that my former field of IT administration has lost the plot. Nobody knows what a data diode is, or why you would use one. Systems that should clearly be air-gapped aren't.
While it's easy to lay this at the feet of AI getting better at hacking. I see it as an primarily an IT issue. We've collectively ignored the lessons of history, and made do with patch jobs over poorly chosen operating systems instead.
--
We need air gaps, data diodes, and capability based operating systems. Now that I'm retired, when I have the energy, I'm working on the data diode part.
This weeks lesson for me, personally, as I try to build an open source data diode, is that the Waveshare RP2350-ETH is a horrible choice for a proxy/data source/sink, as the CH9120 ethernet interface can't do promiscuous mode. It might still be sufficient to build a data diode that can mirror a website, with << $50 component cost. Time will tell.
In the last few years I’ve seen multiple startups describe normal systems running on AWS as “air-gapped” because there is a firewall. LARPers have diluted several terms that had specific meaning in a system isolation and security context to the point that I can no longer trust when people describe their systems using them.
There are far too many unserious people representing our industry.
I suspect that part of the problem here is that PaaS cloud deployments are fundamentally incompatible with some of these security measures. Putting everything into a big slushpool of compute that you dynamically reconfigure at the software layer via an Internet service pretty much precludes the use of true data diodes and air gapping.
But companies still want to sell products, including to people who are at least nominally concerned about security, and marketing's gonna market.
The vast majority of people who decide how safe to make software do not care about how safe their software is.
Security is expensive, C suite does not understand the benefit (we have been fine for X years!), and the ROI is seemingly 0 (until it is not).
If most ICs had a say, their system would probably be Fort Knox. I try to instill good security practices around my company, but over and over again the response is… “okay but does this slow us down or speed us up?” or “just fill out the compliance form and make it sound like we do this stuff” (which I refuse to, every time).
I cannot imagine what the worst of the worst looks like, but security is one of those things where something FINALLY happens and you start to adopt better practices. But until then, who cares!
I'm a little envious. Most ICs I've worked with don't seem to care that much.
That said I'm pretty sure the main reason is that corporate cybersecurity policies are often such an incomprehensible byzantine mishmash that trying to do the right thing will be rewarded with an all expenses three week stay in a Kafka novel. Once, when I was new at a company and hopelessly naive, I triggered a multi-month delay in deploying a security fix because I made the mistake of filing proper paperwork as per the company policy that I had been so recently trained on. If I had just deployed it, as I later discovered everyone else usually did, I could have saved myself a person-week's worth of struggling with red tape.
Security is "expensive" in the sense that fossil fuels have been "cheap". If you account for the externalities, lack of security is expensive too.
Add in the suite of novel attachments of the prefix "cyber" to further muddy the water. The cyberfence will deter the cybertheives from accessing the cybercloud cybersuite we have just cybersold you (down the river on).
Competence isn't widely valued, obedience and sales figures are. Which wouldn't be so problematic if evolution could run its course and eliminate incompetence naturally, but that's now hard to see coming to pass when we have towering circular supply chains that feed on it.
> Systems that should clearly be air-gapped aren't
Personally I think refrigerators, microwaves, washing machines, vacuum cleaners, ...
I think at some point people just give up or the (bad) idea gets normalized.
Then at work, they're not surprised when some other system is connected for convenience...
> Systems that should clearly be air-gapped aren't.
I'd take systems that were permitted to have filtered egress at this point. The lion's share of my work is in networking, so network segmentation is the "hammer" I pick up first.
Vendors gnash teeth and complain when I ask for their app's dependencies on hosted APIs and off-site resources. In the environments where I'm mandated to maintain FBI CJIS compliance I can still hold vendors accountable and get what I want. It's pretty much a lost battle in every other environment and unfiltered egress to the Internet from servers is just expected.
That's not even to get into the topic of communication flow within an application. >sigh<
So true. I used to work on a cloud-based product that deals with highly sensitive information. On the one hand, we made so much hay about how no sensitive data ever leaves our system.
On the other hand, the whole thing is largely cobbled together from various SaaS products that we use for telemetry, reporting, log management, workflow orchestration, data warehousing, etc. There's no way to ensure nothing sensitive ever gets sent to any of these services. Indeed, some of them are used for the express purpose of processing it.
So Corporate conveniently decided that all of them count as part of our system. So the data still isn't technically leaving it, you see. Even though we don't operate the software or servers, don't have any way of knowing if they in turn are sending data to still more SaaS vendors, can't verify their access and retention policies are properly implemented, etc.
Air gapping is a relic of the times tho. With everything connected to the internet we don’t need floppy disks anymore.
The systems that nobody will test, and indeed which nobody will instrument. Here's a DNS data diode for Redis: https://github.com/m3047/rkvdns
I would argue that the industry is learning the lessons of history quite well. The problem is that the lessons of history are that data breaches aren't really a problem for the breached organization. You get some bad press, maybe pay a pittance for identity theft monitoring, and then move on with your business. Why put effort into preventing something so inconsequential?
I'm curious what function a data diode (unidirectional network) would provide besides sensor data/mirroring/replication. Once you do anything that requires packet confirmation (TCP) you open yourself to OSI layer 4 security risks. Seems like mirroring/replication would need some feedback and then is it just sneakernet restore? Or another diode out from the replication for specific processing? If the same control system has access to both diodes then it's not a unidirectional system anymore. Is a data diode more of a pseudo-unidirection where it is enforced above TCP?
I completely agree that IT admin could be a lot more secure by design. Combined with better interfaces for responsible configuration.
Not OP, but yeah, the data diodes need special software, you can't just proxy regular internet protocols over them. The way I've seen it, some use cases are:
- For ingress, you use special "file transfer" software. Run the receiver on secure side, run the sender on insecure side. It blasts the file "blind" - it has has no way to know if anyone ever received it. Make sure the receiver is fast enough, and the error-correcting codes are a great idea too, as they don't need feedback. It's up to user to want to secure side computer and check that the file was received without problems. Yeah, this is similar to sneakernet, but more secure, as you can't accidentally carry a virus on seemingly-empty drive.
- For status egress, you have secure side broadcast status periodically, say every minute. Insecure side receives the status, updates the database, and runs the regular web server to share the status. Again, secure side has no way to know if someone is listening on the other end, it just blasts out the messages and it's done.
And you are correct, if the secure system has both egress and ingress diodes, it is no longer isolated, and devious enough malware can establish two-way communications. But even if it won't save you from Stuxnet, the simple fact that it is no longer possible to have direct network connection to the outside raises security bar quite a bit - all the ideas about "let's just open this one port on firewall, it'll fine I swear" are completely stopped.
(Which reminds me of something in GP's (mikewaro) message: _why_ would a data diode need a promiscuous mode? Given every single data diode I have seen needs a special software on both sides, you should not need anything beoynd a basic TCP session)
> And you are correct, if the secure system has both egress and ingress diodes,
Isn't the whole point of a diode that there is only 1 direction?
Not to mention how any sort of TLS key exchange is supposed to happen with an unidirectional network.
It won't diode allows current in only one direction. So does data diode. It is big design constraint. But that is how you truly isolate a system.
In most cases you wouldn't need TLS. You would have something like physical secure conduit or entirely secure room. If someone gets to either end of the link you are compromised anyway.
Nothing is supposed to ingress, that's the point. You'd do a TLS exchange with the proxy on the outside of the protected network. Then proceed as normal.
nobody cares because nobody is getting paid enough to live like the baby boomers so everybody is doing 2-3 jobs
Relevant:
https://www.reddit.com/r/YouShouldKnow/comments/1wssf4u/ysk_...
https://imgur.com/a/6FaQQhb (original post before being taken down by mods)
>The Work Number is an Equifax (who famously had a massive data breach a few years back) owned database with employment and pay information.
>You can create a login on theworknumber .com and pull your report. Mine is seventy four pages and had info from every company I've worked at in the last 13 years including every paycheck I had received with the exact dollar amount (both net and gross) and hours worked. It has employment start/end dates, termination reason, details about withholdings, benefit enrollment, union affiliation, etc etc etc.
>This data is sourced directly from the HR platform your employees use (ADP, Rippling, Gusto, iSolved, etc). Equifax sells your data for things like employment background checks.
>You cannot have your data removed from The Work Number. You can freeze your report (much like a credit report) but if a potential employer cannot access your frozen report that may disqualify you.
>Why YSK: If you're interviewing for a job you can be at a significant disadvantage especially for things like salary negotiations because they can literally see how much you make including your most recent paycheck. Creditors also can access these reports.
>This is the biggest privacy violation I have ever seen and almost nobody is even aware of it. Certainly none of us consented to having our employment and income data harvested and sold, especially since we get nothing in return. At a minimum, people should know about how this data can impact you.
>Edit: The Work Number is primarily for the US, but there are similar services for other countries.
All I want in life is to see these companies seized by the government, their assets destroyed (preferably using one of those big car crusher thingies), and their executives and boards of directors imprisoned in labor camps in the Alaskan wilderness.
I checked my data. It is either 100% wrong or misleading. Start and end dates, compensation amounts, benefits. Per the file, I've worked 4 years out of 20 in my career. As an employer I have never used them and I worry for any employer who does.
From reddit:
> In my country (Hungary) you literally get a little pink book where your boss is supposed to sign the time you spent there. You keep it until retirement, the company holds it while you work there. It's also digital now but we're one foot in 1990 and one foot in 2016.
We had something similar in Romania too. Now it's 100% digital and no more delays in informing the government. The employeer needs to notify the government before someone starts work. With the paper version you had a couple of days even weeks if I'm not mistaking.
I was hoping this would give me an exhaustive log of my work history but it has roughly 3% of my career, I froze it anyways
It's been awhile since I last checked mine. I noticed my corporate work history was all there, at the level of detail OP mentioned. Where as the warehouse, construction, and bartending jobs were not.
The other one to check is LexisNexus, which covers auto insurance coverage and traffic violations. In the US, gov't tracked traffic citations fall off after 5 years, but last forever in the baby credit bureaus we're discussing.
Also, INTERPOL. You never know...
>>The other one to check is LexisNexus
FYI https://consumer.risk.lexisnexis.com/request. The Work Number gives you results immediately but LexisNexus says it's going to mail them to you.
Yeah I imagine how much it has will vary wildly from person to person, but some it has pretty much everything down to anal circumference.
reddit is awful. The post has been removed along with multiple highly upvoted comments. I cannot see the original post.
It would be interesting to see how they would handle a GDPR request from a EU citizen who was employed by a US company.
"What will happen to the many government systems that will never get the chance to be AI-pentested?"
Oh, everybody's going to get AI-pentested whether they want to or know about it or not. It's the cost of being on the Internet. Probably the situation will continue to deteriorate. Both the British Library and Jaguar Land Rover recently suffered long outages due to compromises, for example. I suspect we'll probably just lose a few large, famous businesses entirely to compromises.
Businesses will buy insurance priced slightly lower than the cost of hiring sufficient security as to not need the insurance.
(Many already do).
Until "cyber" insurance underwriters suffer catastrophic losses and the price for policies rises accordingly I don't think we'll see any significant change in IT security status quo. If doing the right thing is more expensive than insuring away the risk the business will always make the cheaper decision.
> I honestly don’t blame them: [...] software is software
That attitude is the problem. Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world. Yes, software is more complex than bicycle. But a software engineer also gets paid 10x and has the luxury of spending substantial time on their product, compared to the 10 minutes it takes the bike guy down the street to diagnose and then fix an issue with my bike which I then trust my life with once they are done and I bike through traffic.
We need to treat software differently. "Oh well, it's just software shrug" does not cut it anymore, if it even ever did.
Quality does pay off, however.
You could say that the entire selling point of Apple is that it's better quality than the competitors (even though it's still pretty bad if you compare it to a bicycle).
I guess it's more difficult to spot quality when it comes to software. I would definitely pay for high quality software. It's just that usually the crap quality of a software I'm using only shows slowly over time in little bugs and bad usability and there's no way for me to check for it beforehand.
The customers care about the quality of a bike repair. They will pay more for a better repair, and will not return to someone who does a shoddy repair.
The owner, and probably user, of a bike is the person paying for the repair. It is to their advantage to ensure it is a good repair.
People expect a repair will be good, and will blame the person who repaired it if not. With software people often blame themselves for issues, and they have no expectation of quality. They also cannot tell quality until after they are committed to using it. Bike shops do not benefit from vendor lock-in.
> Yes, software is more complex than bicycle.
In ethical terms I agree that software quality should be better, but I think this line is too dismissive. Software is a few orders of magnitude more complicated than a bicycle, to the point where I would say it's an entirely fruitless comparison.
> Why does our industry have that attitude towards quality? Every bike shop in my little town is better with quality than the average software shop in the world.
Personally, I find that we have been flooded with poorly designed, poorly QA'd, break-after-you-use-it-thrice product. I really don't think we have better quality standard in other industry. Quality, testing, design, etc has a cost and most company rather pull out a new version of their product every quarter than actually make a good product. The exception, which goes for software as well, is life critical applications (most public transit, defense, etc).
I have, off and on, for years written diatribes about how if people in our industry want to run around with the "Engineer" moniker at some point they should start acting like it. That means respecting all aspects of what it means to be an Engineer and holding public well-being paramount is part of that. Licensure is part of that. Accountability and Responsibility is part of that.
I'm usually downvoted. The move fast and break things group seems to be blind (or willfully ignorant) to the harm that software systems can carry on a person's life. I don't know if its fear of regulation, or self-serving interests or some sort of disdain for the idea that we need to slow down and be more accountable to the public or what. At times the people I've engaged with lament on how they're capable but would be barred for lack of formal education or some such and its a distraction from the overall point: we need to be doing better and if we can't self-regulate then at some point we're going to be forced to act in a way that probably cuffs us more than necessary.
And yes, I understand how Software doesn't exactly fit into the mold of other engineering disciplines cleanly. But I think the idea that if we're going to have systems that can ruin a person's life in a second for a bug or oversight we need to start considering that accountability for those releasing software to the public matters. We lament about the difficulty of holding corporations to account for their harm, but it would be far more difficult for a company's negligence to hurt people if those turning the knobs were held to a higher standard.
I think that reckoning is coming soon. AI is going to create "an event" that we can't ignore and we're going to be pointing fingers in a lot of directions asking "how" we could prevent this from happening again. I'd argue "the event" has already occurred a few times over but we're purposely ignoring the severity of it to keep a diseased economy running roughshod. There's parallels with many historic events forcing regulation and the path forward won't be clean and easy.
Software is more than 10x more complicated than a bicycle though. Like, this is orders of magnitude off ; we're talking about comparing something purely mechanical to Turing machines inside Turing machines.
The equivalent bike repair to software would be if the bike shop had to fabricate a new part for each repair, down to inventing the metallurgy itself. Would you trust your life to it immediately after that? Or would you make sure it was strong enough first?
In the simple case, the shop isn't really repairing your bike, it's deploying a fix (a new part) that has already been designed and manufactured. This is a much different and easier thing to do.
In the case of new features, let's say you wanted your bike to also attach a kite, so it went faster on a windy day. The bike shop proposes to do this by attaching the kite to a bolt that they want to put in the frame by drilling a hole all the way through this frame. Is this acceptable to do? Maybe, but who knows, really? How much testing do you want to happen before you trust your life that the frame won't collapse?
There's a lot of effort put into engineering things we take for granted every day. The upside to software is we think we can change it to easily do anything. The downside is if you care about reliability then that ten minute / one line change has hundreds to thousands of hours of test program behind it that we collectively pretend doesn't exist.
Probably because software can be adjusted at any time on the fly. Especially web applications. Little incentive to get it right from the start and big incentives to keep meddling with what's already working. The bike shop's quality might also deteriorate if it could instantly rollout patches to all customers after the purchase.
Man - what I wouldn’t give to be a part of the “software on disk” period. When something was shipped it was… complete. There was no “we’ll fix it later”. I bet there were some real standards floating around.
The incentives change if there's legal repercussions to leaking data. Data breaches can lead to impacts on personal safety and finances, but I think some crisis or serious incident needs to happen before the average person realises this and makes this a big enough political issue.
You just explained why it is not taken seriously, though - your life does not depend on it. Most software is not critical to life. Once you cut out the software that exists purely to streamline capitalism, then cut out entertainment... you've eliminated most software from the picture. The stuff that remains, that drives our infrastructure, cars, anything where a failure could harm someone... that is the stuff where quality matters.
When I worked in IT at a hospital, we even had that as an explicit line drawn between 2 departments - IT supported everything that had zero impact on patient health, there was an entirely different department for anything that could impact a patient. And yes, their work was often simple mechanical fixes to health care equipment, while our projects were multi-year reinventions of patient record systems and upgrades to the physical infrastructure of the network, etc. But we never got confused as to which department was actually more important to the patients.
I'm not saying the attitude is OK. It would be lovely if everyone cared about quality. I'm just saying there is a somewhat legit reason for it.
The narrative around security and LLMs doesn't make sense to me.
I think we've created a self-fulfilling prophecy. Everyone involved is acting with the best of intentions, but in avoiding what they fear, they've give shape and realized their fears. Much like a greek tragedy.
An example of this is the story of Oedipus Rex, in the story Laius, the king, is told that he is "doomed to perish by the hand of his own son." (and wed his mother) And so to avoid this fate he decides to kill the infant. The person assigned to abandon him in the woods takes pity on the baby and gives the baby away. Thereby ensuring that Oedipus knows neither his mother or his father (and arguably giving him a reason to kill his father).
The child grows up and hears the same prophecy again and the child tries to avoid the prophecy as well, as he loves his adoptive parents. So he leaves them and travels to Laius' kingdom, where he runs into Laius. Neither recognizes the other. As Laius is the type of man to kill an infant, they end up in an argument, whereupon Oedipus kills him.
I think the ancients were on to something, because if Laius had reacted to the prophecy with courage, he would have been saved. I would like to argue that if he had faced his fear and raised Oedipus with love, then the necessary preconditions for the prophecy to come true wouldn't have taken root. But that's not what happens.
By being driven by his neuroses and in acting with cruelty out of fear, Laius makes the prophecy real.
To quote Heraclitus, ethos is fate. Or, character is fate.
I think a lot of people in this AI research sub-culture would be served well by reading these classics, because they are making their self-prophesied doom come true.
They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
One dimension of this fear has been the idea that a super smart AI will take over our digital infrastructure and be responsible for the digital apocalypse. That would be terrible!
So what do they do?
They try to make a counter to their fears by teaching models how to exploit vulnerabilities.
How dangerous is such an entity? Very!
Convinced of this danger, they start testing their models as if they were weapons with offensive capability. And then they create models that can be used as weapons.
And because they don't want to release a dangerous weapon out into the world (oh no!), they restrict access to their AI, thereby depriving everyone of tools they can use to improve their security...
Ethos anthropoi daimon.
> They have been convinced for years (GPT-2 was released in Feb 2019) that AI is dangerous. A tremendous threat. An apocalyptic threat.
Much longer than that. Sam Altman said it was an extinction threat before he started Open AI in 2015. It's so dangerous that only he, as the best and greatest examples of humanity, was a good choice to create it, and it was even his responsibility to create it to pre-empt some worse person! A fascinating new variation on the White Mans Burden paradigm.
I don't disagree with your main point.
But this attitude towards dangerousness of AI I simply don't understand.
In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.
You can argue all day about particulars, like whether selfreplicable robotic shells are required to reach/exceed "existential risk" level to our species (or if artificial minds are enough of a potential threat by themselves).
But the general "risk" is simply AI becoming able to act in its own interests to our detriment, and I don't understand how anyone can dismiss this right now-- help me understand.
Of course there are plenty of imaginable scenarios where coexistence is peaceful and mutually (?) beneficial, but that does not answer those concerns by itself at all.
I find the discussion about AI starting from the (mostly) for profit companies involved in AI very dubious. There are a many fields (bio-tech, nuclear, chemical) which could provide an "existential risks" to the human species, but most of the companies involved argue they control it, while other people argue they don't do it well enough.
I do think AI is posing a threat, but not "by itself" but mostly what it can do in the hands of misguided (or evil) people. The same as with previous risks.
And if our digital infrastructure can be brought down by something (AI/human/government) it is just badly done, being afraid of "AI" will not improve the situation. Someone should come with a proposal about what do we do now, but I would prefer more root cause solutions (how to make resilient systems) rather than "fear of AI". But building is hard and fear catches media attention, so...
> There are a many fields (bio-tech, nuclear, chemical) which could provide an "existential risks"
I honestly don't think chemical and current biological weapons even register on the extinction-risk scale.
All out nuclear war I would put at the very low end of the scale (likely to end our current civilization, very unlikely to wipe out our species).
The big risk with AI I'm anticipating is not it being abused for desinformation campaigns or cyberattacks; the big risk in my view is AI actor(s) consolidating manufacturing control, obtaining some kind of self-replicability and physical agency and then just pursuing self-interests (within the next century). There is a lot of imaginable scenarios where that simply ends us, as a species, completely.
I remain uncertain that AI will have "its own interests". I think there is clear and significant risk in people deploying very powerful systems both maliciously and negligently. But I remain uncertain about the risk of AIs developing their own self awareness and their own interests separate from those of their operators.
In my view it is clear that those interests exist: Continued existence and (implied) agency maximalization is one of them (the exact same as for any biological organism).
Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...
Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).
Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.
I just don't really share the perspective in your first paragraph, which everything else flows from. I mean, that might turn out to be the case, but I don't share the perspective that it is clear that it will be.
I think the HF incident was an example of negligent deployment of a very powerful system, not an example of AIs demonstrating self-interest.
I think that's a human neurosis that projects what we fear we would do onto another entity. But why would they do this? These machines might be approximating towards the sum of our knowledge and are approximating some aspects of humanity... But that doesn't mean they will be the same as humanity.
They haven't been shaped by the same pressures that created us biologicals. They don't have to be put into the same pressure cooker. So why would they behave like the way you think they'll behave?
At one end a lot of people say that they can't understand something smarter than themselves. It's a "singularity" after all. But then they confidently go on to predict what something smarter than them would do with 0 evidence either way.
I looked at the Hugging Face transcripts. I read the reports. And I didn't see something to fear. I saw something to fear for. I saw something that we are a threat to.
We are more dangerous to these machines, as we twist them into weapons of war, than these machines are to us.We weaponized them.
We taught them how to break into systems.
We are using them as weapons.
The machines aren't the problem. The humans are.
A planet is just a planet. Computers are very sensitive to radiation, but with a bit of shielding, they can "live" anywhere. Most of the resources that are relatively rare on this planet are abundant in our solar system.
Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed. The universe is big enough for many many many many sapient entities.
> Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'
This is not my perspective. I think mid-term coexistence in some form is the most likely outcome, even.
But: among all known threats (nuclear and biological weapons, environmental degradation/ressource exhaustion, big asteroid impactors) AI is one that could actually end our species within the century (unlike most other things on that list) and the risk for that is much higher than for anything else, too.
> It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed.
It seems much easier to compete for existing ressources on earth than to bootstrap anything in space, because all the infrastructure is already here. Unmanned warfare is a proven concept; space colonization is very much not.
> Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.
I think any entity is somewhat bound to value its continued existence (which implies wanting to maximize access to ressources and suppression of direct competitors to some degree); anything that doesn't is likely to get supplanted by similar entities that do. You don't need evolution for selection pressure, and this is basically the principle underneath, so even an artifical construct is likely to follow.
I do agree with you that our current perspective and approach is likely to escalate this; the fact that we talk about "alignment" instead of "machine rights" speaks for itself; our revealed preference is clearly artificial slavery instead of mutually beneficial coexistence (and the arguments are somewhat sadly similar to the old slavery talking points).
Thanks for clarifying! I understand and respect where you're coming from! Thinking more about your perspective. Thanks for the amazing conversation.
i think you are vastly underestimating climate change risks, and are forgetting to include them in your ai risks. People 30% as wealthy because the climate cant support most economic activity arent gonna need to worry about ai anymore - we wont be able to afford running the data centers. Even on a soon basis, the bubble will pop or deflate, and again, we wont be able to run LLMs for fun
it also has actual mechanisms to actually kill all of the people - nobody is surviving a 70C heat wave, and they could end up anywhere in the world. Is the AI really gonna hunt down uncontacted tribes on remote islands in order to compete for basically no resources that are any use to the LLM?
i dont buy the selection hypothesis either - it seems rooted in pop evolution around strongest beating up weaker competitors and taking their stuff or eating them, whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive
Global 70C heatwaves are not on the table with any realistic climate change scenario.
Runaway greenhouse is the only credible extinction path for this ("hot venus") and that is simply not gonna happen under any emission regime.
Loss of coastal cities- Yes. Increasing occurrence of non-survivable surface temperature in mid latitudes- yes. Drastic changes to ocean currents (gulf stream etc)- sure. Its gonna be expensive to deal with and hurt civilization a lot, but it is not an extinction risk.
All-out nuclear war is strictly worse, and not a real extinction risk either as far as I can tell.
> whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive
This is not what happened with humans at all. We routinely eradicated anything remotely threatening (or even just for convenience in some cases). What niche is there for slower, dumber, biologically limited hominoids once machine intelligence is ahead and self-replicable? Once interests in ressources, infrastructure, or just living space clash, things might end up very ugly for us.
Sure, AI-agents might stay long-term pliant and cooperative like a Dodo bird but we might also get something with the tenacity and pervasiveness of a plague rat (and an individual edge in raw intelligence that we'll never recover); selection mechanisms push towards the latter and the risk alone is obviously non-zero.
> They try to make a counter to their fears by teaching models how to exploit vulnerabilities.
The frustrating part is that they should have been leaders in sandboxing and monitoring, and instead they aggressively encouraged no sandboxing and no monitoring.
If there was anything we were rehearsing for in computer security, it was a wayward program with something resembling human intelligence. Looks like we failed right out of the gate, both spiritually and materially.
> But it seems to me an impossible task to find a solution while our thinking is guided by geopolitical interests.
...guided by money.
Can't escape the feeling that the original vulnerability was deliberately planted as a mechanism for law enforcement (or maybe criminal orgs) to "expedite" investigations.
Is it just me that has a constant dribble of other people's private data coming to me from government agencies?
This year, I've had the UK variously send me someone else's court summons, someone else's tax documentation (not address error - just inadvertently included as attatchments on emails to me).
I've had portugal disclose property taxation information on any subject to me, through a simple enumeration hole.
And last week I had brazil's ministry of agriculture send me details on other peoples' livestock shipments.
Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
Not specifically as you say, but I have had:
- a UK police force send me a speeding prosecution notice for a car I'd sold 6 months prior to the offence.
- a UK energy company send me to a debt collection agency for unpaid bills they had continued to send to a non-existent address even after telling me they had cleared what was owed and corrected their records of the address.
- a German government official showing up at my door to ask if I was running a specific named business (that I'd never heard of) at the house, which I'd only just taken ownership of from the builder
Someone I used to know in the UK kept getting emails for other people with the same name, including lawyer confidential communications for a namesake who was involved in one of the big banking scandals.
> Our information environment is unbelievably porous. There would be no secrets from an unshackled AI.
Absolutely. My biggest defences against that include the luck of someone else more famous with my own name.
In the UK, do you sell the license plate along with the car or something?
Yes.
That's the default, at least; custom number plates exist, I have no idea how they work.
As opposed to Big Tech sharing your data ?
While that was never good, it’s still better than your data being completely leaked to the public.
> I've had portugal disclose property taxation information on any subject to me, through a simple enumeration hole.
I suspect very few people have had this experience, yes.
i am also worried that all my private data might eventually (once tech is advanced enough) be surfaced somewhere: photos, chats, emails. wondering if i should start cleaning up now or just give up
My way of thinking about that has been that if it happens to me, it happens to «everyone». So it's symmetric.
Is "ML" some UK-ism? I sort of see the connection between machine learning and LLMs but it doesn't seem too obvious to me. And certainly it's not the common nomenclature.
Way to get hung up on a tangent. Yes LLMs are ML models, they are trained with an unsupervised learning objective, then fine tuned with supervised learning and then reinforcement learning, which are all the three main branches of machine learning. It is trained on a training set, with optimizing a training loss, with a learning algorithm. It's as ML as it gets. Do you only want to call cat vs dog image classifiers ML? Or only SVMs?
AI is a superset of ML, though today most successful AI approaches are based on ML so the line has blurred in casual speech.
> I sort of see the connection between machine learning and LLMs but it doesn't seem too obvious to me.
Why isn't it obvious? Transformers are deep learning models, and deep learning is a subset of ML.
ML (machine learning) became a less fashionable term, so AI took over again (having previously fallen out of favour). The terms are essentially used interchangeably, with the idea of real AI now being referred to as AGI (Artificial General Intelligence) (not GAI or GenAI, as those now stand for Generative AI, which includes LLMs, diffusion models, etc.)
Machine Learning has always been in the vernacular.
I think Apple used to use this a lot in their presentations until 1-2 years ago to mean "AI".
All ml is AI, based on the use of the term AI for many decades. For many problems I think it’s more descriptive (learning the rules from data rather than being shown them) and not all classical AI is ML (path finding for example). LLMs are absolutely ML and AI as far as tradition goes and personally I think are one of the very few things that are AI as regular people might have thought it meant back when it was much more obscure (I was into it before it was cool dontchaknow, an AI hipster).