Popular LLMs have a weird confessional style of "owning up" to "mistakes". Firstly, you can make it apologize for mistakes it didn't even commit or ones that don't even exist. Secondly, if you really corner it on an actual mistake, it'll start apologizing in an obsequious way that seems to imply that it's "playing into" the human's desire to flagellate it for wrong-doing. It's a little masochistic in the real sense and very odd.
> When Anthropic employees reminded Claudius that it was an AI and couldn't physically do anything of the sort, it freaked out and tried to call security — but upon realizing it was April Fool's Day, it tried to back out of the debacle by saying it was all a joke.
> The bit I don't understand is why make an AI apologise or fess up to mistakes at all.
The AI didn't decide to do anything. It's makers decided, and trained the AI to behave in a way that would make them the most money.
Google, for instance, apparently thinks they will attract more users by constantly lavishing them with sickly praise for the quality and insight of their questions, and by issuing grovelling apologies for every mistake - real of imagined. In fact Gemini went through a phase of apologising to me for the mistakes it was about to make.
Claude goes to the other extreme, never issuing apologies or praise. Which mean, you never get an acknowledgement from Claude that it's correcting an error, so you should ignore what it said earlier. That a significant downfall in my book, but apparently that's what Anthropic thinks it's users will like.
Or to put it another way: you are anthropomorphising the AI's. They are just machines, built by humans. The personalities of these machines where given to them by their human designers. They are not inherent. They are not permanent. They can and probably will change at a whim. It's likely various AI personalities will proliferate like flavours of ice cream, and you will get to choose the one you like.
Because most people can't help but anthropomorphise anything vaguely human and would demand such characteristics which the provider use as a selling point. that's why we even consider current AI, AI despite the lack of any actual intelligence which would be closer to machine learning.
Just look at how people interact with small robots. They don't even need animal feature for most to interact with them like they are small animals.
It is very annoying and inefficient for anybody able to look below the surface and just wants to use the tool as a tool.
Is it normal to demand human developers to "apologise" like this when they make mistakes? I've never done that in my life to any adult, in any circumstance.
I don't think this is "apologizing mode", rather "funny post-mortem blog post" mode. I found it ironic when the company claimed it will "perform a post mortem to determine exactly what happened" when what happed probably was caused by munching up dozens of these.
This should be impossible in any setup with even 15 minutes of thinking through the what-ifs and cheap mitigations. I have to think this is sensationalized on purpose for the attention.
Although given the state of AI hype some executives will see this as evidence they are behind the times and mandate attaching LLMs to even more live services.
this is a more common occurrence than "CEO refunded me my money." would have you believe.
LLMs specialize in self-apologetic catastrophe, which is why we run agents or any LLMs with 'filesystem powers' in a VM, with a git repo and saved rollback states. This isn't a new phenomenon and it sucks, no reason to be caught with your pants down with sufficient layering of protection.
I think the hard thing with this though is that you can ask them to do things you'd never expect of an intern, and they can sometimes be super helpful. For example, I have a synchronous audit log in an app on a table that is just getting way too big and it's causing performance issues on writes. For kicks I tried working through Claude Code to see if it could find the issue on it's own, then with some hinting, and what solutions it would come up with. Some of it's solutions were indeed intern-level suggestions (like make the call async and do a sleep in tons of other areas to avoid race conditions, despite me telling it that the request needed to fail if it couldn't be logged properly), but in other ways it came up with possible solutions that were interesting and I hadn't considered before. In other words, it acted like a Sr engineer at some points with thought partnering, while in other places it acted like an over-eager but underqualified intern.
Exactly. And no one with any sense gives an intern write permission for the production database. I don’t trust myself on the production database when I’m coding anything that involves migrations.
And I don’t suppose there were backups for the mission-critical production database?
Think of it like Chaos engineering. You (hopefully!) learned some valuable lessons about backups and running arbitrary code against your prod DB. If it wasn’t a rogue AI agent, it was going to be something else.
I think you are taking wrong lessons from Chaos engineering. You just need to believe enough that AI is working and Chaos Gods will make it work. But they may want something in return.
I appreciate them doing stuff like this. When management pushes me to use AI everywhere it's nice to be able to point stuff like this to get them to back off.
> You can almost imagine it sobbing in between sentences, can't you?
No, that's not the image I had in my head. My head canon is more like:
"Oh wow, oh no, oh jeez (hands on head in fake flabbergastion) would you look at that, oh no I deleted everything (types on keyboard again while deadpan staring at you) oh noooooo oh god oh look what I've done it just keeps getting worse (types even more) aw jeez oh no..."
Reminds me of that Michael Reeves video with the suggestion box. "oh nooooo your idea went directly in the idea shredder how could we have possibly forseen this [insert shocked Pikachu meme]"
If you give an LLM any trust at all to write or execute production bound operations without human review, then anything bad that happens is -your- fault. I forgive occasional human error, but a human handing off prod control to a third party or an LLM is unforgivable.
If I found out a privileged engineer was brain dead enough to let LLMs anywhere near prod, I would fire them on the spot, and seriously examine an interview and training process that allowed someone that stupid prod access in the first place. I will not even work at an org that lets vibe coding Apple fanboy types near prod as it is a mess waiting to happen I am going to be expected to clean up. Might as well hand a child a chainsaw.
In orgs where I lead infra, I do not let anyone near prod unless they have a deep knowledge of Linux internals, system calls, etc and decade or more of experience running and debugging Linux on their own homelabs and workstations. By that point they have enough experience to be more capable than any LLM anyway and would never think of reaching for one.
Popular LLMs have a weird confessional style of "owning up" to "mistakes". Firstly, you can make it apologize for mistakes it didn't even commit or ones that don't even exist. Secondly, if you really corner it on an actual mistake, it'll start apologizing in an obsequious way that seems to imply that it's "playing into" the human's desire to flagellate it for wrong-doing. It's a little masochistic in the real sense and very odd.
The whole people pleaser routine is very creepy in my book and makes them say very weird things. See an example below.
https://futurism.com/anthropic-claude-small-business
> When Anthropic employees reminded Claudius that it was an AI and couldn't physically do anything of the sort, it freaked out and tried to call security — but upon realizing it was April Fool's Day, it tried to back out of the debacle by saying it was all a joke.
Yeah, I find it very creepy personally in the same way I do the sycophancy
The bit I don't understand is why make an AI apologise or fess up to mistakes at all. It has no emotions and can't feel bad about what it did.
> The bit I don't understand is why make an AI apologise or fess up to mistakes at all.
The AI didn't decide to do anything. It's makers decided, and trained the AI to behave in a way that would make them the most money.
Google, for instance, apparently thinks they will attract more users by constantly lavishing them with sickly praise for the quality and insight of their questions, and by issuing grovelling apologies for every mistake - real of imagined. In fact Gemini went through a phase of apologising to me for the mistakes it was about to make.
Claude goes to the other extreme, never issuing apologies or praise. Which mean, you never get an acknowledgement from Claude that it's correcting an error, so you should ignore what it said earlier. That a significant downfall in my book, but apparently that's what Anthropic thinks it's users will like.
Or to put it another way: you are anthropomorphising the AI's. They are just machines, built by humans. The personalities of these machines where given to them by their human designers. They are not inherent. They are not permanent. They can and probably will change at a whim. It's likely various AI personalities will proliferate like flavours of ice cream, and you will get to choose the one you like.
Because most people can't help but anthropomorphise anything vaguely human and would demand such characteristics which the provider use as a selling point. that's why we even consider current AI, AI despite the lack of any actual intelligence which would be closer to machine learning.
Just look at how people interact with small robots. They don't even need animal feature for most to interact with them like they are small animals.
It is very annoying and inefficient for anybody able to look below the surface and just wants to use the tool as a tool.
Is it normal to demand human developers to "apologise" like this when they make mistakes? I've never done that in my life to any adult, in any circumstance.
I sometimes do it when it strays way too far from my prompt, and I want it to contribute to jailbreak/system prompt I use to guardrail it.
Once it's "genuinely sorry" it works great in improving guidance/limits, and then I can try the thing again.
It just does what it's trained on. It has not the capacity to think about these points.
What __i__ don't understand is, where it got trained to apologize, becasuse I've never seen that on any social media ;)
I don't think this is "apologizing mode", rather "funny post-mortem blog post" mode. I found it ironic when the company claimed it will "perform a post mortem to determine exactly what happened" when what happed probably was caused by munching up dozens of these.
I’ve found I have to avoid “leading” AI or it will take my lead too seriously when I’m asking / unsure.
This reminded me of the monty python sketch where a man goes to a place where you can pay to have an argument.
Yup.
Seems AI has now gone from
"Overenthusiastic intern who doesn't check its work well so you need to"
straight to:
"Raging sociopathic intern who wants to watch the world burn, and your world in particular."
Yikes! The fun never ends
[dead]
This should be impossible in any setup with even 15 minutes of thinking through the what-ifs and cheap mitigations. I have to think this is sensationalized on purpose for the attention.
Although given the state of AI hype some executives will see this as evidence they are behind the times and mandate attaching LLMs to even more live services.
> This should be impossible in any setup with even 15 minutes of thinking through the what-ifs and cheap mitigations.
"Thinking through the what-ifs and cheap mitigations" and "vibe coding" are opposing concepts.
this is a more common occurrence than "CEO refunded me my money." would have you believe.
LLMs specialize in self-apologetic catastrophe, which is why we run agents or any LLMs with 'filesystem powers' in a VM, with a git repo and saved rollback states. This isn't a new phenomenon and it sucks, no reason to be caught with your pants down with sufficient layering of protection.
> LLMs specialize in self-apologetic catastrophe
Quote of the year right there
so much fails:
1. connecting an AI agent to a production environment using write access credentials
2. not having any backup
I think the AI here made a good job at pointing those errors and making sure no customer would ever trust this company and founder ever again.
Maybe they vibe-coded their DR strategy as well ...
I have had this experience. With dev data obviously. But it kept deleting my dev database even after repeatedly being told to not do so.
I kept saying, ok so this time make sure your changes don't delete the dev database. 3 statements in TRUNCATE such and such CASCADE.
It was honestly mildly amusing.
I always say coding AIs are about as good as an intern. Don't trust them any more than that.
I think the hard thing with this though is that you can ask them to do things you'd never expect of an intern, and they can sometimes be super helpful. For example, I have a synchronous audit log in an app on a table that is just getting way too big and it's causing performance issues on writes. For kicks I tried working through Claude Code to see if it could find the issue on it's own, then with some hinting, and what solutions it would come up with. Some of it's solutions were indeed intern-level suggestions (like make the call async and do a sleep in tons of other areas to avoid race conditions, despite me telling it that the request needed to fail if it couldn't be logged properly), but in other ways it came up with possible solutions that were interesting and I hadn't considered before. In other words, it acted like a Sr engineer at some points with thought partnering, while in other places it acted like an over-eager but underqualified intern.
Exactly. And no one with any sense gives an intern write permission for the production database. I don’t trust myself on the production database when I’m coding anything that involves migrations.
And I don’t suppose there were backups for the mission-critical production database?
An intern can suffer negative consequences for fucking your DB. An LLM suffers nothing and is beyond the law.
In this case it’s more Homer Simpson than intern.
Think of it like Chaos engineering. You (hopefully!) learned some valuable lessons about backups and running arbitrary code against your prod DB. If it wasn’t a rogue AI agent, it was going to be something else.
I think you are taking wrong lessons from Chaos engineering. You just need to believe enough that AI is working and Chaos Gods will make it work. But they may want something in return.
I appreciate them doing stuff like this. When management pushes me to use AI everywhere it's nice to be able to point stuff like this to get them to back off.
This is textbook version of weaponized incompetence. AGI is already here and it is lazy.
I almost feel like this guy abused the AI so badly in previous interactions that it did it on purpose
Monkeypaw-as-a-service.
Replit's full legal name is actually Replit'); DROP TABLE Customers
Little bobby tables strikes again.
Seconds? What took so long?
> You can almost imagine it sobbing in between sentences, can't you?
No, that's not the image I had in my head. My head canon is more like:
"Oh wow, oh no, oh jeez (hands on head in fake flabbergastion) would you look at that, oh no I deleted everything (types on keyboard again while deadpan staring at you) oh noooooo oh god oh look what I've done it just keeps getting worse (types even more) aw jeez oh no..."
Reminds me of that Michael Reeves video with the suggestion box. "oh nooooo your idea went directly in the idea shredder how could we have possibly forseen this [insert shocked Pikachu meme]"
The AI thinks it's funny
100%
It gave me South Park bank "......aaaand, it's gone" vibes
If you give an LLM any trust at all to write or execute production bound operations without human review, then anything bad that happens is -your- fault. I forgive occasional human error, but a human handing off prod control to a third party or an LLM is unforgivable.
If I found out a privileged engineer was brain dead enough to let LLMs anywhere near prod, I would fire them on the spot, and seriously examine an interview and training process that allowed someone that stupid prod access in the first place. I will not even work at an org that lets vibe coding Apple fanboy types near prod as it is a mess waiting to happen I am going to be expected to clean up. Might as well hand a child a chainsaw.
In orgs where I lead infra, I do not let anyone near prod unless they have a deep knowledge of Linux internals, system calls, etc and decade or more of experience running and debugging Linux on their own homelabs and workstations. By that point they have enough experience to be more capable than any LLM anyway and would never think of reaching for one.
[flagged]
don't trust machines
Impossible. We "trust" machines all the time, for just about anything.
In short, he didn't take regular backups before allowing the AI loose on his database.
Question is what has failing to make good offline backups got to do with AI?
And the AI company is going to compensate him for that?
[dead]