This is the second significant exploit against LiteLLM in what, a year or two ? Just last night I came across notes and a script from an older claude session to help audit all project dependencies and verify nothing used litellm.
All the sandbox-exec and all the Lulu rules are not going to protect against something like this.
At one point permission pop-up fatigue sets in we start giving any permission as if it's normal. We also become desensitised to isntalling tools: brew, npm, it becomes second nature.
"Oh, those 367 tools installed by package X? Yeah yeah, must be normal, that's how all these work now".
And the advent of agents and harnesses. It's not a matter of whether, it's when! OSes need to do something about it. Help users have complete separate "program space" at ease with better, easier, and lighter VMs if nothing else. I know, I know - Unix and Linux solved all this even before Alexander III of Macedon started wearing his nappies. Of course.
No no it's really important that I have 5-factor authentication on every website and it requires me to go through the whole login process again every time I break eye contact.
> All the sandbox-exec and all the Lulu rules are not going to protect against something like this.
No, you can block this with sandboxing tools. For my LLM agents, I use a tool called Nono, whose macOS implementation is built on the same foundation as sandbox-exec. One of the key features it supports (and it's not the only tool that does this) is proxying and the injection of credentials via the proxy. For example, when I run Codex at work, Codex does not have access to my Azure AI Foundry API credentials. It sends out requests via dummy credentials, and the proxy injects the real ones outside the sandbox.
The other thing that proxy does is let you restrict network access in various ways. So you can run something like LiteLLM so that it doesn't have access to the internet at large; it can only talk to Azure AI Foundry or AWS Bedrock or whoever your inference provider is. You sandbox LiteLLM separately from your agents, and you give it access to much less. The software firewall stuff isn't perfect but it is a significant defense against things like this.
There are also lots of other restrictions that are easy to do which mitigate this. Nono blocks access to the root of $HOME, then selectively allows access to paths within it, and also explicitly blocks access to common paths to secrets. So it's pretty easy to make sure that access to important things are blocked: SSH credentials, GnuPG socket (and local Unix sockets in general), local OS keyrings, etc.
The way I use tools like this is exclusively by writing policy, testing it, and revising where the policy is inadequate. I never get approval fatigue because I don't do approvals.
You're right that attacks like this are hard to defend against, and that we're used to tool sprawl, and also that LLMs can do things like install tools we don't expect them to themselves! But there are ways to anticipate and mitigate attacks like this. Sandboxing tools (including contained and VMs) that support credential injection are genuinely useful for this.
Until this Nono tool gets a supply chain attack and happily ships all your Azure credentials somewhere they aren’t supposed to be, then starts mining Monero or something.
LiteLLM is frequently used in roughly the same scenario - trading virtual LAN-only keys for real ones as it relays requests to the inference providers.
What are you talking about? Even dependency cooldown of 24h is enough to protect the consumers from these breaches. The attack did not have any novelty, used a newly registered look-alike domain for C2 and a PTH technique which is now known.
Sandbox comes in for the unknown ones and yes, its a pain to get the profile right.
> Unix and Linux solved all this even before Alexander III of Macedon started wearing his nappies
I mean, didn't Linux shells invent pop-up fatigue? I thought we (yes, we) all used to -y flag preemptively so the upgrades and installations/setups go through. Linux really did do it first, the agents are just more needy and their popups more verbose now, but not exactly new.
Stuff like this is highly context dependent. It is a good thing that apt asks about each step. I voluntarily add more of this by opting for things like listbugs. If I then decide to override such protections with a -y for whatever reason that's a choice on my part to prioritize convenience.
Either way it's got nothing to do with security - I was never going to catch a vulnerability in the central debian repos by reading a list of dependent packages.
"Attackers compromised 3 unique developer github credentials" is a little scary; this seems systemic - if three developers had their keys stolen, it implies the real problem is elsewhere.
They revoked all but one developers, which is probably the right approach, but I'd pin LiteLLM to "something before this version" and keep it there until further notice!
I am not following how someone snuck the files into the release.
All I read was that it creates a directory from the build file but seemingly no description how new files or added dependencies slipped by. Was `cook.pth` a dependency that got automatically downloaded from PyPI on install and then the build system put it into dependencies?
--
Leaving this for others:
> BerriAI’s private release pipeline built these wheels from an attacker-modified git source. The attacker could have used a compromised maintainer or bot credential, a private branch or mirror, or the reference the pipeline builds from. A stolen PyPI token used from the attacker’s own machines fits the evidence less well.
Makes sense. It was that a maintainer or the private pipeline was attacked and the malicious dependency was injected there by squatting on the maintainer's local copy rather than the dependency being hijacked for everyone. Very clever.
If that's right then attestations wouldn't have saved anyone here either. They prove which workflow published the wheel, not what source that workflow checked out. A private pipeline building from the same-looking repo still gives you a perfectly valid attestation.
What would catch it is someone rebuilding from the public tag and comparing the result, which almost nobody does for the python wheels. I have been treating attestations as the answer for my own project's releases, so this is a good reminder of where they stop.
IMHO defaults for uv, npm, etc should all have a cooldown for 24 - 48 hours by default, you should have to go out of your way to install something newer.
We have 7d cooldown on packages on an internal pypi proxy but that blocks security updates also, which we now have to manually exclude from the cooldown.
I’m unsure how to automate this properly, maybe an LLM is the best starting point.
They definitely wouldn't be if attackers knew that it would then bypass cooldowns!
Ultimately, you need someone to verify releases and that probably means paying for it. I know Sonatype paid solutions utilise their own internal research to allow/block new releases.
No, but there's a bigger tradeoff for not upgrading. In one case it's that you're potentially exposed to a known vulnerability, vs the other case where you might just be missing out on some new or slightly reworked feature.
We need something external that actually checks to see if source code matches built artifacts, and also possibly something Llm based that checks to see if you’re making checked in code vulnerable or directly adding something malicious.
I didn't realize they've all such added that, this was a change we made to our own projects last year when these first started - glad to see it's common.
Risk is ensuring enough security researchers are finding these and quarantining these before that default cooldown lapses.
Yea, I think all the tools have this now but its not a default. Install scripts should also be turned off by default and require some hassle to turn them on for the first time in a package.
The last time I looked for information, I remember seeing that the median supply chain attack was in place for multiple months to years before being discovered. Cooldowns shorter than that seem like snake oil.
No, because there are malware-finding mechanisms other than "let some unlucky guy get affected first", like automated scanners. If everyone implements a cooldown, everyone will in fact be safer.
Plus some random value slow curve distribution. Basically incremental rollout from the client side. But if it's not caught at the commit time, attacker will just wait for a certain point on the curve.
I wonder if in some cases it's not so much that people are updating too quickly, but rather, their deployment pipelines do not have pinned / frozen versions attached to their builds.
What's interesting here is that this will only affect those who did not migrate off LiteLLM after it was attacked the first time.
I'm not saying everyong should migrate after there's a vuln, but I'm definitely saying that. Unless it's something huge and foundational like a programming language (PHP gets a second chance), just migrate within the next couple of weeks.
I didn't even have it installed the first time, so the ideal programmers, (read, your competition), actually suffered 0 vulns from this dep, if you suffered 1, that's already pretty bad, being burnt twice should be a wake up call. If you get hit 3 times by the same dependency, it's a lost case.
As someone who does not use litellm but instead a barely maintained underpowered alternative: what did everyone migrate to?
If it matters, I am mostly interested in the aspect of having a proxy that unifies multiple ollama and openai-compatible endpoints with model-based routing and failover. Optionally with api-keys, reporting, budgets, etc. I do not particularly care about the MCP gateway, Agent-To-Agent thingy, and whatever else LiteLLM bolted on over time
Even if it plays out that way (which I doubt, we have a ton of requirements around spend tracking), the problem I'm seeing is long term maintenance. I don't want to fudge around with my proxy every time one of the model providers comes up with a new parameter that works slightly differently on Bedrock than it does on Azure.
That said, I am looking into alternatives. Two times in one year is really a terrible look.
After the last LiteLLM compromise, I took my requirement (an OpenAI compatible endpoint routing across multiple providers based on latency) and built that as a small static Rust binary - https://github.com/sturnus-dev/sturnus
Now that AI makes it cheaper to build such tools in-house, the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker.
Pydantic AI has a great set of abstractions over LLM providers. The documentation tends to focus on its agent loop more, but I find the LLM abstractions quite nice even on their own
As with many dependencies, I wholeheartedly question the requirement and relentlessly cut them out without replacement.
Taking the definition from another user on this comment-chain:
"an OpenAI compatible endpoint routing across multiple providers based on latency"
If the endpoint is truly OpenAI compatible, you can just switch out the vendor by switching out the HTTP address, we are talking potentially about a single line change.
It sounds silly, but remember that software is a winner-takes all game, 99% of devs will lose, so if you are seeing 99% of devs do the same thing, it isn't surprising that it's a grossly wrong choice. It takes skill and guts to be in the 1%, reject the mainstream trends, and just write code without grasping at external imports to do even the most basic things.
I'd like to take this opportunity to sell out, I'm open for contracts if you are looking for a remote 1% dev. Early stage startups only, email in userpage.
> OSS Users
> For the time being, please pin your version to 1.106.2 or any explicit version published prior to now.
> Enterprise Customers
> There currently exists a private LiteLLM registry for enterprise customers, in which we are distributing an embargoed release that remediates a vulnerability whose disclosure date is scheduled for October 27. Our private registry is not affected. If you are an enterprise customer pulling from our private registry you are not affected.
1. there is a second vulnerability that will be disclosed next Tuesday (can't be this one because talking about it as undisclosed in the GH issue disclosing it makes no sense)
2. the patch for that vulnerability is somewhere in the diff between $current_public_release and $current_enterprise_release
3. You have to pay to get $current_enterprise_release
They might not be selling the vulnerability directly, but they are selling access to the patch that fixes it. Which would be enough of a hint even in pre-LLM times. And unlike other vendors they don't just claim "you get faster updates and additional features" or something like that, they directly point out that if you buy now you get advance access to the fix to an unannounced vuln
I'm not saying they are advertising this to bad actors. But if they wanted to do that with a shred of plausible deniability, this would be the way to do it
There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.
We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.
I wish all the best to LiteLLM team with handling it again. I can imagine how challenging are situations like this one.
However, if anyone is looking for 20x faster and less "targeted" LiteLLM alternative - I'm work on one in Go. That's why GoModel it's called. https://github.com/ENTERPILOT/GoModel/
I myself have written a small llm proxy in Go, doing nothing but injecting the real API key to outgoing requests while itself validating virtual keys it generates, and it works great.
I have GoModel running at home, being deployed at work. There are many good reasons to run a gateway, some include
- being resilient to upstream provider downtime
- centrally managed config/controls
- visibility and allocation into usage and costs
We are deploying a gateway at work primarily for these three.
At home, GoModel's virtual models have been super helpful for testing out new open weights. I can change the model deployed in vLLM without having to change any of my downstream tools, which get a stable name.
Yes if you use LLM to analyse everything. Not really if you analyse diffs, with heuristics to select candidates for LLM/agent based analysis. But at the cost of missing out novel attacks.
To clarify a little further, you can select the title, you just can't tell that it's selected because the author made a "poor" (REALLY REALLY DUMB) decision to deliberately style away a basic browser feature (text highlighting).
What you don't know, it's actually a very clever trick by an LLM. By not highlighting the text, it makes you think harder and gives you actually some challenge. 99% of people want to be challenged when selecting text, so the AI decided to drop text highlighting.
> There currently exists a private LiteLLM registry for enterprise customers, in which we are distributing an embargoed release that remediates a vulnerability whose disclosure date is scheduled for October 27.
oh boy. Announcing that there's a private enterprise repo you should be paying for if you don't want a credential stealer in your build is quite the optic, and then announcing there's an undisclosed second vulnerability that's being distributed, just not to the poors...
That's not what's being said, at all. They are simply telling their enterprise customers that there is a different build pathway for the enterprise builds and that pathway wasn't impacted by this issue.
Also if you don't like organizations fixing vulns without releasing the open source first, you're going to be upset when you learn how every major vuln and associated patches are distributed long before the fixes hit the repos. A timely example of this is PHP, literally today.
> A timely example of this is PHP, literally today.
Sorry, there was a paid-for PHP release that fixed it I could have gotten a week ago, and that they publicly announced existed, and it was just people who don't pay for PHP who had to wait...?
Yeah, that's how many companies with open ecosystems work. If you're using e.g. RedHat, you are getting packages with embargoed fixes baked into them before those fixes hit public repos. Node.js, OpenSSL, Git itself all do this.
You can argue about whether the "publicly announced existed" part since they don't advertise for any single package, but that's the ecosystem you buy into.
> If you're using e.g. RedHat, you are getting packages with embargoed fixes baked into them before those fixes hit public git repos.
The way you've written this is substantially misleading. RedHat receives notice of the vulnerabilities so they can prepare packages to be available the moment the embargo lifts, which I guess technically could hit the public packages a tiny bit earlier, but they are not releasing fixes of embargoed vulnerabilities to their commercial clients before the embargo lifts.
This is substantially different from the policy I've quoted that I am objecting to.
There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.
We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.
Well it seems like it’s over for litellm. You’d have to be negligent to be using them at this point. If you’re managing this at your company good luck explaining why you should continue to use litellm after this
There are over 8 billion people on Earth, and some subset of the computer systems that they all use are going to have issues, every day.
It's really not as bad as the accumulated headlines and exploitative hype make it sound.
It's similar to the mass hysteria and moral panics over things like child abduction. Media reports make certain dangers seem much more important than they really are, when considered in context.
This is the second significant exploit against LiteLLM in what, a year or two ? Just last night I came across notes and a script from an older claude session to help audit all project dependencies and verify nothing used litellm.
Okay, checked, it was March of this year: https://futuresearch.ai/blog/litellm-pypi-supply-chain-attac.... What even is time anymore in the llm age?
Indeed, I don't think I'm upgrading ever again. I'll stuck with 1.04.2 for a year or two. Feels like a Russian rulette at this point.
Just going to remind you all that https://canarytokens.org/ exists.
TIL! this is cool
All the sandbox-exec and all the Lulu rules are not going to protect against something like this.
At one point permission pop-up fatigue sets in we start giving any permission as if it's normal. We also become desensitised to isntalling tools: brew, npm, it becomes second nature.
"Oh, those 367 tools installed by package X? Yeah yeah, must be normal, that's how all these work now".
"Brew shows 59 packages outdated?". "Must upgrade". "Yeah, done. Perfect".
And the advent of agents and harnesses. It's not a matter of whether, it's when! OSes need to do something about it. Help users have complete separate "program space" at ease with better, easier, and lighter VMs if nothing else. I know, I know - Unix and Linux solved all this even before Alexander III of Macedon started wearing his nappies. Of course.
No no it's really important that I have 5-factor authentication on every website and it requires me to go through the whole login process again every time I break eye contact.
Ready to add anal probes?
for what, chess.com?
> All the sandbox-exec and all the Lulu rules are not going to protect against something like this.
No, you can block this with sandboxing tools. For my LLM agents, I use a tool called Nono, whose macOS implementation is built on the same foundation as sandbox-exec. One of the key features it supports (and it's not the only tool that does this) is proxying and the injection of credentials via the proxy. For example, when I run Codex at work, Codex does not have access to my Azure AI Foundry API credentials. It sends out requests via dummy credentials, and the proxy injects the real ones outside the sandbox.
The other thing that proxy does is let you restrict network access in various ways. So you can run something like LiteLLM so that it doesn't have access to the internet at large; it can only talk to Azure AI Foundry or AWS Bedrock or whoever your inference provider is. You sandbox LiteLLM separately from your agents, and you give it access to much less. The software firewall stuff isn't perfect but it is a significant defense against things like this.
There are also lots of other restrictions that are easy to do which mitigate this. Nono blocks access to the root of $HOME, then selectively allows access to paths within it, and also explicitly blocks access to common paths to secrets. So it's pretty easy to make sure that access to important things are blocked: SSH credentials, GnuPG socket (and local Unix sockets in general), local OS keyrings, etc.
The way I use tools like this is exclusively by writing policy, testing it, and revising where the policy is inadequate. I never get approval fatigue because I don't do approvals.
You're right that attacks like this are hard to defend against, and that we're used to tool sprawl, and also that LLMs can do things like install tools we don't expect them to themselves! But there are ways to anticipate and mitigate attacks like this. Sandboxing tools (including contained and VMs) that support credential injection are genuinely useful for this.
Until this Nono tool gets a supply chain attack and happily ships all your Azure credentials somewhere they aren’t supposed to be, then starts mining Monero or something.
LiteLLM is frequently used in roughly the same scenario - trading virtual LAN-only keys for real ones as it relays requests to the inference providers.
What are you talking about? Even dependency cooldown of 24h is enough to protect the consumers from these breaches. The attack did not have any novelty, used a newly registered look-alike domain for C2 and a PTH technique which is now known.
Sandbox comes in for the unknown ones and yes, its a pain to get the profile right.
> Even dependency cooldown of 24h is enough to protect the consumers from these breaches.
From some of them perhaps, but plenty hide for -years- undetected.
Don't let perfect be the enemy of good
Fuchsia solved that
> Unix and Linux solved all this even before Alexander III of Macedon started wearing his nappies
I mean, didn't Linux shells invent pop-up fatigue? I thought we (yes, we) all used to -y flag preemptively so the upgrades and installations/setups go through. Linux really did do it first, the agents are just more needy and their popups more verbose now, but not exactly new.
VM isn't a silver bullet.
Stuff like this is highly context dependent. It is a good thing that apt asks about each step. I voluntarily add more of this by opting for things like listbugs. If I then decide to override such protections with a -y for whatever reason that's a choice on my part to prioritize convenience.
Either way it's got nothing to do with security - I was never going to catch a vulnerability in the central debian repos by reading a list of dependent packages.
How could this happen to a project with 55k commits in three years?
The compromise appears somewhere in the build and deploy system.
> No public commit in BerriAI/litellm contains data = "cook" or a cook/ folder.
> SafeDep assesses, with moderate to high confidence, that BerriAI’s private release pipeline built these wheels from an attacker-modified git source.
Every harness thinking right now… "if we want do something like this we won't even have to compromise anything."
Sarcasm?
haha, yes, sorry.
you needed more exclamation marks :)
The gentleman's /s tag.
I mean, 55k commits in 3 years, that's about 55 a day, 7 days workweek. It's reasonably implied no one can be looking at all them thoroughly.
This incident is unrelated to what was committed to the repository. More info: https://docs.litellm.ai/blog/security-incident-oct-21-2026
"Attackers compromised 3 unique developer github credentials" is a little scary; this seems systemic - if three developers had their keys stolen, it implies the real problem is elsewhere.
They revoked all but one developers, which is probably the right approach, but I'd pin LiteLLM to "something before this version" and keep it there until further notice!
I am not following how someone snuck the files into the release.
All I read was that it creates a directory from the build file but seemingly no description how new files or added dependencies slipped by. Was `cook.pth` a dependency that got automatically downloaded from PyPI on install and then the build system put it into dependencies?
--
Leaving this for others:
> BerriAI’s private release pipeline built these wheels from an attacker-modified git source. The attacker could have used a compromised maintainer or bot credential, a private branch or mirror, or the reference the pipeline builds from. A stolen PyPI token used from the attacker’s own machines fits the evidence less well.
Makes sense. It was that a maintainer or the private pipeline was attacked and the malicious dependency was injected there by squatting on the maintainer's local copy rather than the dependency being hijacked for everyone. Very clever.
If that's right then attestations wouldn't have saved anyone here either. They prove which workflow published the wheel, not what source that workflow checked out. A private pipeline building from the same-looking repo still gives you a perfectly valid attestation.
What would catch it is someone rebuilding from the public tag and comparing the result, which almost nobody does for the python wheels. I have been treating attestations as the answer for my own project's releases, so this is a good reminder of where they stop.
Official Incident Post: https://docs.litellm.ai/blog/security-incident-oct-21-2026
IMHO defaults for uv, npm, etc should all have a cooldown for 24 - 48 hours by default, you should have to go out of your way to install something newer.
I use safechain for this, because there is always gonna be another package manager configured wrong. https://github.com/AikidoSec/safe-chain
We have 7d cooldown on packages on an internal pypi proxy but that blocks security updates also, which we now have to manually exclude from the cooldown.
I’m unsure how to automate this properly, maybe an LLM is the best starting point.
Are security updates immune ?
They definitely wouldn't be if attackers knew that it would then bypass cooldowns!
Ultimately, you need someone to verify releases and that probably means paying for it. I know Sonatype paid solutions utilise their own internal research to allow/block new releases.
It’s in the name. They are secure!
No, but there's a bigger tradeoff for not upgrading. In one case it's that you're potentially exposed to a known vulnerability, vs the other case where you might just be missing out on some new or slightly reworked feature.
But then an attacker marks their malicious release as a critical security update. It's really messy unfortunately.
We need something external that actually checks to see if source code matches built artifacts, and also possibly something Llm based that checks to see if you’re making checked in code vulnerable or directly adding something malicious.
Problem is, we leave ourselves open for LLM injection attacks.
I didn't realize they've all such added that, this was a change we made to our own projects last year when these first started - glad to see it's common.
Risk is ensuring enough security researchers are finding these and quarantining these before that default cooldown lapses.
Yea, I think all the tools have this now but its not a default. Install scripts should also be turned off by default and require some hassle to turn them on for the first time in a package.
I’d imagine there were probably fixes for critical CVEs that were included …. A lot of companies are on a 24 hour timeline to get them out.
The last time I looked for information, I remember seeing that the median supply chain attack was in place for multiple months to years before being discovered. Cooldowns shorter than that seem like snake oil.
If we all implement a cooldown, nobody will find the attack. This implies the cooldown length must be the cooldown length plus one month.
No, because there are malware-finding mechanisms other than "let some unlucky guy get affected first", like automated scanners. If everyone implements a cooldown, everyone will in fact be safer.
Plus some random value slow curve distribution. Basically incremental rollout from the client side. But if it's not caught at the commit time, attacker will just wait for a certain point on the curve.
You tell people that the cooldown needs to be a day, while you implement a two month cooldown.
Something something don't need to outrun the bear
I wonder if in some cases it's not so much that people are updating too quickly, but rather, their deployment pipelines do not have pinned / frozen versions attached to their builds.
Then we’ll find out about malware 24 - 48 hours later.
Last time they had this, I just vibe coded a version of lite llm and it actually works better
Yeah, thinking that after the last supply chain attack, people would be smarter than ship things with LiteLLM.
Is that you Max?
litellm in the news again
What's interesting here is that this will only affect those who did not migrate off LiteLLM after it was attacked the first time.
I'm not saying everyong should migrate after there's a vuln, but I'm definitely saying that. Unless it's something huge and foundational like a programming language (PHP gets a second chance), just migrate within the next couple of weeks.
I didn't even have it installed the first time, so the ideal programmers, (read, your competition), actually suffered 0 vulns from this dep, if you suffered 1, that's already pretty bad, being burnt twice should be a wake up call. If you get hit 3 times by the same dependency, it's a lost case.
As someone who does not use litellm but instead a barely maintained underpowered alternative: what did everyone migrate to?
If it matters, I am mostly interested in the aspect of having a proxy that unifies multiple ollama and openai-compatible endpoints with model-based routing and failover. Optionally with api-keys, reporting, budgets, etc. I do not particularly care about the MCP gateway, Agent-To-Agent thingy, and whatever else LiteLLM bolted on over time
Claude will write you something custom that fits your exact requirements with zero supply chain risk in an afternoon.
how is there zero supply risk? claude will still package dependencies into a homemade solution at some level...
sure zero was a bit strong, but I personally feel pretty good about depending on only the go stdlib, compared to pulling in litellm.
Even if it plays out that way (which I doubt, we have a ton of requirements around spend tracking), the problem I'm seeing is long term maintenance. I don't want to fudge around with my proxy every time one of the model providers comes up with a new parameter that works slightly differently on Bedrock than it does on Azure.
That said, I am looking into alternatives. Two times in one year is really a terrible look.
I just installed it on my server in docker, then the UI was unusable and I deleted it and vibe coded a minimal alternative in golang. It works better.
After the last LiteLLM compromise, I took my requirement (an OpenAI compatible endpoint routing across multiple providers based on latency) and built that as a small static Rust binary - https://github.com/sturnus-dev/sturnus
Now that AI makes it cheaper to build such tools in-house, the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker.
> and built that as a small static Rust binary
I don't think memory safety would have helped here.
> the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker
You mean the Rust ecosystem?
Pydantic AI has a great set of abstractions over LLM providers. The documentation tends to focus on its agent loop more, but I find the LLM abstractions quite nice even on their own
As with many dependencies, I wholeheartedly question the requirement and relentlessly cut them out without replacement.
Taking the definition from another user on this comment-chain:
"an OpenAI compatible endpoint routing across multiple providers based on latency"
If the endpoint is truly OpenAI compatible, you can just switch out the vendor by switching out the HTTP address, we are talking potentially about a single line change.
- LLM_HOST = "api.openai.com" + LLM_HOST = "api.competitor.com"
It sounds silly, but remember that software is a winner-takes all game, 99% of devs will lose, so if you are seeing 99% of devs do the same thing, it isn't surprising that it's a grossly wrong choice. It takes skill and guts to be in the 1%, reject the mainstream trends, and just write code without grasping at external imports to do even the most basic things.
I'd like to take this opportunity to sell out, I'm open for contracts if you are looking for a remote 1% dev. Early stage startups only, email in userpage.
>PHP gets a second chance
second? i haven't kept count, but we have to be somewhere in the hundreds of chances now
> I'm not saying everyone should migrate after there's a vuln, but I'm definitely saying that.
Turns out Ancient Aliens was onto something all along.
its litellm again. I wonder about maintainer trustworthiness
Seriously again for litellm? I really with that things like Google ADK didn't add this to my deps.
all popular, free, ungated software will be poisoned; like kesler syndrome but for the world of open source software.
Horrifying framing, I will remember that analogy viscerally. Well done.
https://github.com/BerriAI/litellm/issues/47826#issuecomment...
> OSS Users > For the time being, please pin your version to 1.106.2 or any explicit version published prior to now.
> Enterprise Customers > There currently exists a private LiteLLM registry for enterprise customers, in which we are distributing an embargoed release that remediates a vulnerability whose disclosure date is scheduled for October 27. Our private registry is not affected. If you are an enterprise customer pulling from our private registry you are not affected.
So they sell advance access to information that will allow me to hack other people's LiteLLM instances?
Seems a little unethical
I'm not following, what did you mean?
They are saying that
1. there is a second vulnerability that will be disclosed next Tuesday (can't be this one because talking about it as undisclosed in the GH issue disclosing it makes no sense)
2. the patch for that vulnerability is somewhere in the diff between $current_public_release and $current_enterprise_release
3. You have to pay to get $current_enterprise_release
They might not be selling the vulnerability directly, but they are selling access to the patch that fixes it. Which would be enough of a hint even in pre-LLM times. And unlike other vendors they don't just claim "you get faster updates and additional features" or something like that, they directly point out that if you buy now you get advance access to the fix to an unannounced vuln
I'm not saying they are advertising this to bad actors. But if they wanted to do that with a shred of plausible deniability, this would be the way to do it
Note: I work at LiteLLM
There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.
We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.
Again this is the second time this year
Isn't that at least the second time that LiteLLM got compromised?
Déjà vu for both of us, huh?
I wish all the best to LiteLLM team with handling it again. I can imagine how challenging are situations like this one.
However, if anyone is looking for 20x faster and less "targeted" LiteLLM alternative - I'm work on one in Go. That's why GoModel it's called. https://github.com/ENTERPILOT/GoModel/
Sounds like a reasonable choice of a language.
I myself have written a small llm proxy in Go, doing nothing but injecting the real API key to outgoing requests while itself validating virtual keys it generates, and it works great.
Definitely it was a deliberated decision. Also enterprises adapt Go more and more.
If you don't use any LLM router, it's infinity times faster, and infitesimally as targetted.
I have GoModel running at home, being deployed at work. There are many good reasons to run a gateway, some include
- being resilient to upstream provider downtime
- centrally managed config/controls
- visibility and allocation into usage and costs
We are deploying a gateway at work primarily for these three.
At home, GoModel's virtual models have been super helpful for testing out new open weights. I can change the model deployed in vLLM without having to change any of my downstream tools, which get a stable name.
https://gomodel.enterpilot.io/docs/features/virtual-models
Yeah, this DIY MITM attack is entirely avoidable.
A bad day for opensource, but I guess this is how we get stronger patch by patch.
Any package hosts that scan and include cooldowns by default that operate on a consumption model? The ones I have seen all charge a lot.
Bytesafe (https://bytesafe.dev). Pure Dependency Firewall, not a package host, with usage based pricing.
This one ? https://news.ycombinator.com/item?id=46909037
I use it, I don't cooldown, proper version pinning for me.
Not sure about a service, but PMG enforces dependency cooldown and sandbox on package install. https://github.com/safedep/pmg
Socket.dev
Tell em Tom Zubiri sent ya (I don't get a commission, just good karma)
man, people are going to be sorely disappointed in what's free on the internet in a couple of years.
Agents and malware will eradicate all goodwill left.
Good by internet.
I figured LLMs nowadays would be good enough to catch issues like this. Is it because it’s too costly to do it?
Yes if you use LLM to analyse everything. Not really if you analyse diffs, with heuristics to select candidates for LLM/agent based analysis. But at the cost of missing out novel attacks.
This is the 2nd time. I'm glad I moved away from this platform.
Why on earth do these people make it impossible to select the title of their article?
it's just the poor styling of the selection highlight colour.
To clarify a little further, you can select the title, you just can't tell that it's selected because the author made a "poor" (REALLY REALLY DUMB) decision to deliberately style away a basic browser feature (text highlighting).
What you don't know, it's actually a very clever trick by an LLM. By not highlighting the text, it makes you think harder and gives you actually some challenge. 99% of people want to be challenged when selecting text, so the AI decided to drop text highlighting.
/s
"A way to provide users with a sense of pride and accomplishment."
1792590200 | LiteLLM Hijacked: Malicious 1.107.0 Wheels Steal Credentials | https://research.jfrog.com/post/litellm-hijacked-again/ | https://news.ycombinator.com/item?id=50180838 | 0 comments
1792590232 | Litellm 1.107.0 Compromised | https://github.com/BerriAI/litellm/issues/47826 | https://news.ycombinator.com/item?id=50180845 | 0 comments
Again! Yikes.
Twice? uiuiui
So in just 1 week:
1. Google Cloud Outage [0]
2. eMudhra CA private key Leak [1]
3. PingCap TiDB 10.6 TB data Leak [2]
4. PHP2Shell uncovers critical CVEs (with AI) [3]
...and now we have Round 2 of a LiteLLM compromise after the first one in [4].
[0] https://news.ycombinator.com/item?id=50163961
[1] https://news.ycombinator.com/item?id=50166579
[2] https://news.ycombinator.com/item?id=50173716
[3] https://news.ycombinator.com/item?id=50180836
[4] https://news.ycombinator.com/item?id=47501426
don't forget today's NodeJS security update addressing its own RCE bugs!
is the google outage related to a security incident?
No, it was a power event localized to one building.
> There currently exists a private LiteLLM registry for enterprise customers, in which we are distributing an embargoed release that remediates a vulnerability whose disclosure date is scheduled for October 27.
https://github.com/BerriAI/litellm/issues/47826
oh boy. Announcing that there's a private enterprise repo you should be paying for if you don't want a credential stealer in your build is quite the optic, and then announcing there's an undisclosed second vulnerability that's being distributed, just not to the poors...
That's not what's being said, at all. They are simply telling their enterprise customers that there is a different build pathway for the enterprise builds and that pathway wasn't impacted by this issue.
Also if you don't like organizations fixing vulns without releasing the open source first, you're going to be upset when you learn how every major vuln and associated patches are distributed long before the fixes hit the repos. A timely example of this is PHP, literally today.
> A timely example of this is PHP, literally today.
Sorry, there was a paid-for PHP release that fixed it I could have gotten a week ago, and that they publicly announced existed, and it was just people who don't pay for PHP who had to wait...?
Yeah, that's how many companies with open ecosystems work. If you're using e.g. RedHat, you are getting packages with embargoed fixes baked into them before those fixes hit public repos. Node.js, OpenSSL, Git itself all do this.
You can argue about whether the "publicly announced existed" part since they don't advertise for any single package, but that's the ecosystem you buy into.
> If you're using e.g. RedHat, you are getting packages with embargoed fixes baked into them before those fixes hit public git repos.
The way you've written this is substantially misleading. RedHat receives notice of the vulnerabilities so they can prepare packages to be available the moment the embargo lifts, which I guess technically could hit the public packages a tiny bit earlier, but they are not releasing fixes of embargoed vulnerabilities to their commercial clients before the embargo lifts.
This is substantially different from the policy I've quoted that I am objecting to.
Note: I work at LiteLLM
There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.
We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.
Second f**king time. Just torching their reputation at this point
Well it seems like it’s over for litellm. You’d have to be negligent to be using them at this point. If you’re managing this at your company good luck explaining why you should continue to use litellm after this
Fuck this week.
There are over 8 billion people on Earth, and some subset of the computer systems that they all use are going to have issues, every day.
It's really not as bad as the accumulated headlines and exploitative hype make it sound.
It's similar to the mass hysteria and moral panics over things like child abduction. Media reports make certain dangers seem much more important than they really are, when considered in context.
Welcome to the AI era, every week in the near future