There are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
> But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I kind of wish you wrote "potentially worse" rather than "exciting", because that happens too, and it's deceptively subtle and underappreciated.
To make this very concrete with a programming example, C and C++ are (somewhat counterintuitively) examples here, because if you guaranteed the absence of an entire classes of vulnerabilities - say, guaranteeing that uninitialized memory is zero, to prevent secret leaks - then you simultaneously make it much harder to detect logic bugs that this would've surfaced, since you no longer have that degree of freedom to detect logic bugs (say, via sanitizers). Say, an initialized UID that would've appeared as 0xDEADBEEF might now be well-defined as UID 0, giving you root access instead of tripping an alarm...
In other words, it's like natural selection and antibiotics: being too good at solving one class of problems selects for other classes that are more resilient and harder to find, whereas in some of those cases, whack-a-mole would've actually uncovered the root cause. Like with antibiotics, that's sometimes worth it, but definitely not always! Some infections just aren't worth preventing at all costs.
I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters. We don't want to end up in a situation where problems still lurk but we push their detection beyond our ability because of the way we "solved" other problems.
(C++ was just for illustration here; this extends far beyond programming.)
I feel that it's not such a clear distinction between whackamole bug fixing and systematic security engineering. For instance, he uses an example of recurring security vulnerabilities in Chromium: a DCHECK violation again and again and again.
However, DCHECK is a Chromium assertion used to defensively check invariants. In other words, it's an example of the invariant-based security engineering, which he contrasts with the whackamole approach.
Just to be clear, I think he has a point and I enjoyed reading it---but the problems we're saddled with won't disappear in a flash of enlightenment.
While I had the same thoughts about the coming cyber-apocalypse, the ugly truth is that it doesn't make a difference as in a year or so models will be so persuasive and skilled in social engineering that even the hardest cyberdefense in the world doesn't keep them from exfiltrating any information they want by targeting not the system, but the users: leveraging data from the dark web, humans for rent or devising highly deceptive scams you can t even imagine.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
> Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
> leverage AI to implement verifiable enforcement of the invariants
The only verifiable invariants that are going to work in the long run are:
air gaps
data diodes
Interposing relays used to be used with PLCs[1] to prevent motor control systems from energizing both forward and reverse power to a large motor in the unlikely but possible event of control systems failure. We need to apply the same amount of engineering rigor to our computer systems.
We simply shouldn't trust software to a job that belongs in hardware.
The VRP is a wholly manufactured and in the long-term, unsustainable model to begin with. It's the most duct/duck-taped solution that, with a virtually entirely tech-illiterate police force, can work under the interpretation of the CFAA prior to the van Buren decision in 2020. In practice it's easily ludicrously inefficient and requires trust that simply can't exist. In fact the tech-illiteracy is so profound that even post van Buren, which fixes some of the most glaring problems Orin Kerr has spent his career screaming into the void on with the CFAA, a law drafted 6 years before the web came about but with such vague and broad language and it manages to cover, at this point, almost anything (see also: wire fraud, money laundering statutes, the Mann Act aka the White Slave Traffic Act of 1910 and on the flip side, how RICO primarily serves as a tool of coercion because in practice if you take it to trial the charge will not stand, but a host of more minor but still overwrought and expansive felonies will do the job). It's the stopgap of stopgaps and the fact that it at any point had any buy-in was a minor miracle. I once submitted a bug report only to find that no, the company's intent was to defraud its users, and they simply put the (entirely unenforceable but are you going to test that? I've defended a federal criminal case and didn't lose - hung jury - and that took so much that it was almost a pyrrhic victory but it also without a doubt ruined any political aspirations of the AUSA. Anything less than that and it's pointless.) clause that they are indeed attempting to defraud you, and you just have to take it, in laymen's terms, into their ToS, without notice. Same company also has an working admin key in their production apk for their Algolia cluster and guess who isn't reporting it? There's no point. To quote Yates in his prime, "No likely end could bring them loss/Or leave them happier than before."
The correct way to address this had always been to modify or entirely get rid of the CFAA and start from scratch with a framework that actually works, that isn't punitive, that make some sense, that did not come into existence thanks to Ronald Reagan's admiration for the uncanny realism in the 1983 film WarGames. But we have a constituency now that relies on the inefficiency machine for their living and their votes will be in their self-interest, security be damned. The best people have been sent on a fool's errand for generations. The incentive structures are entirely misaligned now. I published a PoC last weekend that indirectly but pretty clearly shows that the FBI was relying on an anonymous twitter's account's assertions, none of which were going to be admissible in court in the alternate universe where that matters, to avoid saying "I don't know" by blaming North Korea, something that someone with open source tooling that existed back then, who have never taken a STEM course past 11th grade AP Stats, whose terminal degree is a JD, could whip up and test in under an hour, probably shorter except my home lab with the GPU was occupied. We go through the motions but really, those with power are relying on the inherent imbalance of power and well, lying, essentially, to keep order. How is that sustainable?
The whole model needs to go but it likely never will and that's perhaps the real legacy of Reagan and our moribund power structure. Looking at the payout rate in hackerOne's heyday, why would anyone ever report anything to the companies? My assumption is that people who have any rationality are doing just that. Most data breaches are never disclosed officially but at best passed in rumors. I have no hard evidence that I can disclose, but the least leaky operation is a one-man operation and attorney-client is forever.
By the way, your AI benchmarks in the legal realm aren't tested on criminal matters because how do you benchmark two probablistic systems that are both subject to the prisoner's dilemma and imperfect information? If they did the score would likely be low. You'd need to build out so much back-knowledge just to set up any scenario that really any answer is "it depends" is not a joke but the best answer. Any suggestion that we simply take the status quo as is and run with it cannot be taken seriously. It's a foolish system made by clueless men who hit the lottery and didn't even see the ticket until years after. It was then exacerbated by politically ambitious AUSAs who do not care about getting the right person behind bars but someone behind bars. Fast forward 30 years and this is the state of things. Your sophisticated defenses may have been thoroughly hand-reversed years ago but to dodge the DMCA the source was put somewhere like Gitee instead of Github. I'm not sure if you can finish the signup flow without a Chinese ID at this point, but a decade ago you can, at least. Those are outliers too, but outliers in charity. Good luck with the rest. I'm not being cheeky: just because there's a vulnerability does not imply knowledge to how to maximize its impact. Data breaches are put in the open frequently because of petty feuds and a failure to recognize the importance of the data. After all, China does not run on private credit, and hence, your identity being stolen there is virtually meaningless, as meaningless as you having next to the biometric ID card data of all of their citizens. Like harm, value is contextual, and constructed so that it's framework dependent, and we at least know the frameworks that exist broadly. And what you don't know, well, you don't know.
There are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
> But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I kind of wish you wrote "potentially worse" rather than "exciting", because that happens too, and it's deceptively subtle and underappreciated.
To make this very concrete with a programming example, C and C++ are (somewhat counterintuitively) examples here, because if you guaranteed the absence of an entire classes of vulnerabilities - say, guaranteeing that uninitialized memory is zero, to prevent secret leaks - then you simultaneously make it much harder to detect logic bugs that this would've surfaced, since you no longer have that degree of freedom to detect logic bugs (say, via sanitizers). Say, an initialized UID that would've appeared as 0xDEADBEEF might now be well-defined as UID 0, giving you root access instead of tripping an alarm...
In other words, it's like natural selection and antibiotics: being too good at solving one class of problems selects for other classes that are more resilient and harder to find, whereas in some of those cases, whack-a-mole would've actually uncovered the root cause. Like with antibiotics, that's sometimes worth it, but definitely not always! Some infections just aren't worth preventing at all costs.
I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters. We don't want to end up in a situation where problems still lurk but we push their detection beyond our ability because of the way we "solved" other problems.
(C++ was just for illustration here; this extends far beyond programming.)
I feel that it's not such a clear distinction between whackamole bug fixing and systematic security engineering. For instance, he uses an example of recurring security vulnerabilities in Chromium: a DCHECK violation again and again and again. However, DCHECK is a Chromium assertion used to defensively check invariants. In other words, it's an example of the invariant-based security engineering, which he contrasts with the whackamole approach. Just to be clear, I think he has a point and I enjoyed reading it---but the problems we're saddled with won't disappear in a flash of enlightenment.
While I had the same thoughts about the coming cyber-apocalypse, the ugly truth is that it doesn't make a difference as in a year or so models will be so persuasive and skilled in social engineering that even the hardest cyberdefense in the world doesn't keep them from exfiltrating any information they want by targeting not the system, but the users: leveraging data from the dark web, humans for rent or devising highly deceptive scams you can t even imagine.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
> Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
Yes. Every feature is also a vulnerability. The most secure features are the ones that don't exist.
> leverage AI to implement verifiable enforcement of the invariants
The only verifiable invariants that are going to work in the long run are:
Interposing relays used to be used with PLCs[1] to prevent motor control systems from energizing both forward and reverse power to a large motor in the unlikely but possible event of control systems failure. We need to apply the same amount of engineering rigor to our computer systems.We simply shouldn't trust software to a job that belongs in hardware.
[1] https://en.wikipedia.org/wiki/Programmable_logic_controller
I think you can safely skip to the last three paragraphs here.
the security as identity vs security as outcome split rings true. where do bug bounty programs
The VRP is a wholly manufactured and in the long-term, unsustainable model to begin with. It's the most duct/duck-taped solution that, with a virtually entirely tech-illiterate police force, can work under the interpretation of the CFAA prior to the van Buren decision in 2020. In practice it's easily ludicrously inefficient and requires trust that simply can't exist. In fact the tech-illiteracy is so profound that even post van Buren, which fixes some of the most glaring problems Orin Kerr has spent his career screaming into the void on with the CFAA, a law drafted 6 years before the web came about but with such vague and broad language and it manages to cover, at this point, almost anything (see also: wire fraud, money laundering statutes, the Mann Act aka the White Slave Traffic Act of 1910 and on the flip side, how RICO primarily serves as a tool of coercion because in practice if you take it to trial the charge will not stand, but a host of more minor but still overwrought and expansive felonies will do the job). It's the stopgap of stopgaps and the fact that it at any point had any buy-in was a minor miracle. I once submitted a bug report only to find that no, the company's intent was to defraud its users, and they simply put the (entirely unenforceable but are you going to test that? I've defended a federal criminal case and didn't lose - hung jury - and that took so much that it was almost a pyrrhic victory but it also without a doubt ruined any political aspirations of the AUSA. Anything less than that and it's pointless.) clause that they are indeed attempting to defraud you, and you just have to take it, in laymen's terms, into their ToS, without notice. Same company also has an working admin key in their production apk for their Algolia cluster and guess who isn't reporting it? There's no point. To quote Yates in his prime, "No likely end could bring them loss/Or leave them happier than before."
The correct way to address this had always been to modify or entirely get rid of the CFAA and start from scratch with a framework that actually works, that isn't punitive, that make some sense, that did not come into existence thanks to Ronald Reagan's admiration for the uncanny realism in the 1983 film WarGames. But we have a constituency now that relies on the inefficiency machine for their living and their votes will be in their self-interest, security be damned. The best people have been sent on a fool's errand for generations. The incentive structures are entirely misaligned now. I published a PoC last weekend that indirectly but pretty clearly shows that the FBI was relying on an anonymous twitter's account's assertions, none of which were going to be admissible in court in the alternate universe where that matters, to avoid saying "I don't know" by blaming North Korea, something that someone with open source tooling that existed back then, who have never taken a STEM course past 11th grade AP Stats, whose terminal degree is a JD, could whip up and test in under an hour, probably shorter except my home lab with the GPU was occupied. We go through the motions but really, those with power are relying on the inherent imbalance of power and well, lying, essentially, to keep order. How is that sustainable?
The whole model needs to go but it likely never will and that's perhaps the real legacy of Reagan and our moribund power structure. Looking at the payout rate in hackerOne's heyday, why would anyone ever report anything to the companies? My assumption is that people who have any rationality are doing just that. Most data breaches are never disclosed officially but at best passed in rumors. I have no hard evidence that I can disclose, but the least leaky operation is a one-man operation and attorney-client is forever.
By the way, your AI benchmarks in the legal realm aren't tested on criminal matters because how do you benchmark two probablistic systems that are both subject to the prisoner's dilemma and imperfect information? If they did the score would likely be low. You'd need to build out so much back-knowledge just to set up any scenario that really any answer is "it depends" is not a joke but the best answer. Any suggestion that we simply take the status quo as is and run with it cannot be taken seriously. It's a foolish system made by clueless men who hit the lottery and didn't even see the ticket until years after. It was then exacerbated by politically ambitious AUSAs who do not care about getting the right person behind bars but someone behind bars. Fast forward 30 years and this is the state of things. Your sophisticated defenses may have been thoroughly hand-reversed years ago but to dodge the DMCA the source was put somewhere like Gitee instead of Github. I'm not sure if you can finish the signup flow without a Chinese ID at this point, but a decade ago you can, at least. Those are outliers too, but outliers in charity. Good luck with the rest. I'm not being cheeky: just because there's a vulnerability does not imply knowledge to how to maximize its impact. Data breaches are put in the open frequently because of petty feuds and a failure to recognize the importance of the data. After all, China does not run on private credit, and hence, your identity being stolen there is virtually meaningless, as meaningless as you having next to the biometric ID card data of all of their citizens. Like harm, value is contextual, and constructed so that it's framework dependent, and we at least know the frameworks that exist broadly. And what you don't know, well, you don't know.