Its an immense problem that it got so little attention until someone made a TV dramatisation of it. There are definitely many other miscarriages of justice caused by the assumption that systems are always correct, and these are not being corrected.
The lack of consequences for those responsible also means there is no deterrent to others doing similar things.
I think hopefully it raises that awareness that these systems are imperfect and do not allow people to delegate their decision making totally. I'm afraid rise of AI and the increase in more plausible sounding yet incorrect behaviours won't help us there. We should use this as an instructive example on how we can built more transparent/useful systems for all of our users.
Ah ok, potentially I was imprecise - the post office leadership appeared to assume that the software was completely correct. That is what I was trying to frame as "systems are imperfect and people should take care to validate what they are doing"
It's covered by whatever activity they are writing the software for. It's like saying "it's crazy how unregulated reading and writing are" (copyright notwithstanding).
Agreed. Often the “solution” laid out is more accountability for the engineer, which I find equally disgusting.
I’ve been in big company/gov tech for a long time. The companies building these types of projects are almost universally incompetent and often malevolent. Having had the occasion to “rescue” a few big projects, they are usually infused with a culture of incompetence and ambiguity in accountability for all things besides billing.
The problem is that it’s what I call “mutually assured incompetence”. The customer is bad at writing RFP/tenders, and often accepts substandard work out of ignorance. Contractors are disincentivized from delivering a sensible outcome as a result. Contractors only understand pain as a motivation once the deal is done. Garbage in, garbage out.
The failure of accountability is endemic as more and more tasks are automated and outsourced. Something as trivial as cash room accounting is a proven practice, traditionally with many eyes keeping all parties honest. While the personal tragedy isnt a factor, in the US, Bank of America “lost” tens of millions of dollars of physical cash in their outsourced counting facilities. The company’s response? Spend a fortune to physically shift cash around ahead of audits to pretend the money was there.
For Americans wondering what this is all about, this 4 part series captures what happened well: https://en.wikipedia.org/wiki/Mr_Bates_vs_The_Post_Office
its great that exists and that it got attention.
Its an immense problem that it got so little attention until someone made a TV dramatisation of it. There are definitely many other miscarriages of justice caused by the assumption that systems are always correct, and these are not being corrected.
The lack of consequences for those responsible also means there is no deterrent to others doing similar things.
I think hopefully it raises that awareness that these systems are imperfect and do not allow people to delegate their decision making totally. I'm afraid rise of AI and the increase in more plausible sounding yet incorrect behaviours won't help us there. We should use this as an instructive example on how we can built more transparent/useful systems for all of our users.
> these systems are imperfect and do not allow people to delegate their decision making totally
There's no evidence this scandal arose from software allowing any delegation of decisions.
Ah ok, potentially I was imprecise - the post office leadership appeared to assume that the software was completely correct. That is what I was trying to frame as "systems are imperfect and people should take care to validate what they are doing"
Thanks for the clarification. Yes, any system at fault here looks more likely to be corporate than computer.
The criminal prosecution will hopefully determine the extent to which the leadership did genuinely make that assumption.
If you think about it, it's crazy how unregulated software development is compared to other industries, considering every industry runs on software.
If you screw up as a Doctor or a Solicitor or an Accountant etc, you can get done for malpractice or struck off or go to prison.
Somebody will basically say, 'you did such a terrible job, you aren't allowed to do that job anymore'.
Screw up so badly writing software that 13 people commit suicide and sweet fa happens.
The cause of these suicides is not software. It is extortion facilitated by UK courts of law.
I'd like to hear what will be done about the judges responsible.
It's covered by whatever activity they are writing the software for. It's like saying "it's crazy how unregulated reading and writing are" (copyright notwithstanding).
Agreed. Often the “solution” laid out is more accountability for the engineer, which I find equally disgusting.
I’ve been in big company/gov tech for a long time. The companies building these types of projects are almost universally incompetent and often malevolent. Having had the occasion to “rescue” a few big projects, they are usually infused with a culture of incompetence and ambiguity in accountability for all things besides billing.
The problem is that it’s what I call “mutually assured incompetence”. The customer is bad at writing RFP/tenders, and often accepts substandard work out of ignorance. Contractors are disincentivized from delivering a sensible outcome as a result. Contractors only understand pain as a motivation once the deal is done. Garbage in, garbage out.
The failure of accountability is endemic as more and more tasks are automated and outsourced. Something as trivial as cash room accounting is a proven practice, traditionally with many eyes keeping all parties honest. While the personal tragedy isnt a factor, in the US, Bank of America “lost” tens of millions of dollars of physical cash in their outsourced counting facilities. The company’s response? Spend a fortune to physically shift cash around ahead of audits to pretend the money was there.
These are the natural limiting factors on outsourcing and the service-based economy in general.