The original vulnerability description is not worded very well, here's my understanding of what's going on:
1. Attacker crafts a malicious Chart.yaml containing arbitrary code
2. Replaces Chart.lock with a symlink pointing to a sensitive file (like .bashrc or other startup scripts)
3. When you run helm dependency update, Helm processes the malicious Chart.yaml and writes the payload to whatever file the symlink targets
4. Code executes when the targeted file is next used (e.g., opening a new shell)
Why This Works:
Helm follows the symlink during the dependency update process without validating the target, allowing arbitrary file writes outside the intended chart directory.
Can anyone explain in what setup an attacker who can create a symlink where Chart.lock was could not directly write .bashrc or similar? Is this related to how Git handles symlinks?
A symlink is just a special file that contains a string of text, it's not tightly bound to the target like a hard link. You can write anything into that string of text, including, say, "~/.bashrc". Then you can ship that symlink onto another system, and it suddenly points to your .bashrc.
Git just moves symlinks across systems as is, so yes, you can use git to deploy the exploit.
As pedantry, to the very best of my knowledge symlinks could not contain "~" and have it mean $HOME - that's a shell-ism (or os.path.expanduser equivalent in your library). I was suspecting the attack vector may have used "/home/runner" or "/home/ubuntu" as very common paths that could exist and be writable by the user
This has nothing to do with Git. A symlink can be packaged up in a tarball and shipped from one system to another. An attacker would need to create a malicious Chart.yaml file and a Chart.lock file pointing to another file. Then ship those to a system where dependencies are then updated.
This doesn't affect things like installing or upgrading a chart. Dependencies aren't updated at that time.
> A symlink can be packaged up in a tarball and shipped from one system to another.
True enough, but if you have a victim unpacking and building untrusted tarballs there's no security boundary being crossed, is there? You don't have to bother with this symlink nonsense, just update the install script to include your payload directly.
Honestly this vulnerability is dumb. I don't see any realistic scenario where it can be exploited by an unprivileged attacker.
When you do a helm pull and download a chart from a repo, I believe it's a tar-ball. So if you have a workflow where you install charts from the filesystem you could be impacted. I've done that in the past.
You're not installing the untrusted tarball; helm is merely supposed to be extracting it, and then rendering the templates contained within.
(Those templates, once rendered, might then refer to pods, etc. that might be put into a k8s cluster (or perhaps we merely render then YAML, and never `apply` it), and in that sense, one might imagine that that is an install, but that's not the security boundary being crossed here; this would presumably result in execution on the host running helm, which would definitely be surprising.)
You're quibbling over the meaning of "install" but apparently conceding the part about untrusted? OK, fair enough. I still argue that any process involving the extraction and (ahem) "rendering of contained templates" from untrusted sources is broken in ways a fix for this particular symlink issue isn't going to address.
Certainly, it would be better to trust the upstream completely, but let's not kid ourselves? See the entire current state of software supply chain in the industry.
But when I visit a website, I don't expect the website to LCE me. Why should turning a YAML adlib into YAML LCE me, regardless of the trust of the upstream. This is not a privilege I'm expecting to give the upstream ever, and this behavior is a clear security bug, to me…
Helm is a program that allows users to creates packages which other users consume. Those packages contain files that are normally generated by Helm itself, but apparently if you alter your package definition by hand you can replace Chart.lock with a symlink.
As I'm typing this it's occurring to me that you probably shouldn't be able to do that. The fix they applied was to prevent the actual write from occurring when trying to write the lockfile and determining that the lockfile is a symlink. They could (should?) also validate that like, the package itself hasn't been screwed with in this manner.
Having read the CVE multiple times I am still unsure how 2. above happens? Is it possible through the malicious chart itself or is it a dependency for the CVE to be in play at all? And if the latter - what local process would write a symlink from a helm lock file to any kind of system start up script which doesn't point to a much bigger problem than this CVE?
The attacker creates a symlink (e.g., using `ln -s`) to another file. The attacker needs to create the malicious Chart.yaml file and symlink that the Chart.lock file points to.
If being able to create files and symlinks to them is a pre-condition for this, then it's not a serious security bug. If you have that kind of access then there are a million nefarious things you can do.
This is almost becoming a joke at this point, "assuming an attacker has access to the system, they can change things on the system".
Helm is not intended to be able to write files outside of the directory you are rendering the templates to, and the directory that you have downloaded the chart to, so if there is a way to do that, it is a bug in the program and a security bug at that, particularly when the destination is controlled by someone who has written a malicious chart. That it also happens to be able to run arbitrary code makes it worse, but the primary problem is that it can write files outside of the chart directory or the directory you are rendering to at all.
This has nothing to do with whether you are running it in sudo or whatever. (and in fact on MacOs, I don't believe this requires running it with sudo permissions to overwrite ~/.zshrc for example)
I create a malicious chart or compromise one you use (with symlink to an arbitrary file and code).
You download charts either as a tarball from a helm repo or oci registry with helm and helm will create the files and links with your permissions, and send me whatever I wanted to extract from your system.
Yes, you should check things you download from the internet.
But also, that is not how a chart is supposed to work.
It is on the level of "sudo curl URL". It is an obviously stupid thing to do from a security perspective, but projects have suggested doing it to install their software.
If you are new to helm or haven't considered the security around it, it is good to know what to look out for.
Helm is an abomination, as the whole idea of using a text template engine to generate YAML is. And this vulnerability adds insult to injury ;)
Sorry, just can't really recover from trauma of counting spaces and messing up newlines, etc. when writing Helm templates. You know, Lisp "sucks" because "you need to count parenthesis" (you actually don't), yet Helm is a widely accepted technology where you need to count spaces for (n)indent ;)
I'm a dev that jumped to devops and one of my pet peeves will always be the lengths devops engineers go to avoid using a real programming language. Instead of interacting with all these APIs through python, ruby, lua, go, whatever they would rather build hodgepodge systems in bash, coreutils, curl (or wget. or both!) and jq (which is the worst). Or in the case of helm, just creating a half yaml/half Go SDK for generating YAML.
Even the helm infrastructure that I work in is completely wrapped in custom shell scripts that call all sorts of other commands to populate helm variables.
But yeah it's silly that helm templates require all sorts of {{ indent | 4 }} type incantations when the final YAML output is just sent through some kind of toJSON anyway.
You'll often find that if you write ops scripts in, say Python, it's largely calling external commands.
When that's the case, bash if often the better choice, especially if you know it well. It has an excellent REPL, is easy to trace and is already installed everywhere.
> You'll often find that if you write ops scripts in, say Python, it's largely calling external commands.
That's a totally fine trade off for actual sane array/list functionality, robust string manipulation etc. I'd rather form shell commands in a programming language than in bash. People seem to love it.
I do not think for a second though that the average person that "knows bash well" can read and comprehend a multi hundred line bash script written by someone else as fast or even correctly has a seasoned python dev reading python written by someone else.
> the lengths devops engineers go to avoid using a real programming language
OK, but, you know... those tools were created by literal devs. Not in yaml, in a "real language"! So apparently devs thought they needed that.
The argument should be towards all people - we love creating new abstractions and "simplify things", but we suck at honest evaluation of the impact of our creations.
Still: I absolutely hate Helm templating and think that the very existence of "helpers" (even in a default chart!) is an abomination.
> OK, but, you know... those tools were created by literal devs.
All `kubectl` does is create REST API calls to the control plane. So to be fair, what I'm grousing about can be accomplished just fine by a developer like me constructing API calls from python to update objects in k8s.
The problem is I work in devops where tooling written in proper languages with standard libraries that have things like useful arrays or robust string manipulation or ergonomic concurrency is a non-starter for some reason. The argument against that being mostly "I have to install the interpereter first".
Now we get to the point - you are unhappy with how your org/team handles it. Would that be better description? Don't put all DevOps in one basket. It's a messy term anyway.
Many orgs and many teams allow that. That's why alternatives to Helm exist like cdk8s, or Pulumi for rest of infra.
For Kubernetes there is such a variety of tools, it's overwhelming. The only problem to tackle is that Helm/Argo is so popular, that alternatives are hard to sell regardless of real benefits.
Look at how big is a scene of Operators for k8s - this is where the programmable part of k8s went. You configure operators with their own CRDs (usually) that can be static/plain declarative text. Then you write actual code of the operator that deals with all the complexity of enforcing that state. For me this is where typically what you are talking about goes, while high level "non programmable" code sits in yaml in a other repo for others to maintain. Maybe this is where you should also go, standardize your work for others to consume in a form of operator?
I fully get neglect in adopting more complex tooling. sh, curl, sed, awk... those things are present almost everywhere and it's not that hard to get lucky and make a script that will run on almost everything your org has. And it actually might be fine for a decade or so.
I myself could not do code (even scripting) because one of my companies literally treated scripting in anything as development which was strictly regulated (so forbidden for any non dedicated dev role). Or an org that had not a single server in whole DC that has Python 3, years after it's release. Or more recent: some damn Ubuntu LTS that can't be easily upgraded to just 2-3 minor versions that this cool k8s library uses. Maintaining python versions on VMs is a pain in the ass, especially if your org has strict controls. Internet access to pip is not granted as well. UV gets the job done nowadays, if it's allowed, but long story short: that "fear of real language" can be as much lack of knowledge/skills as pragmatism that came from painful experiences.
Yeah, vi has supported % as "jump between matching parenthesis" since it's original release in the 1970s, and vim by default will do simple parenthesis matching and highlighting, I don't see why everyone is so scared of touching lisp for these reasons with modern editors (if your editor doesn't support either of the above... maybe it's not modern enough?)
… and I think I'd argue that the parent's argument against the tooling would apply equally as well to those "other things", too.
The alternative here is something that manipulates the data structure directly. E.g., it might permit me to say:
my_config_map.data["key"] = some_string_value
(This is in some pseudo-imperative language, vs. the parent's Lisp, but that distinction isn't particular relevant to the core of their argument, I think.)
And then at the end, the thing itself takes care of converting the resulting objects to YAML, thus preventing me from inadvertently turning what is meant to be a string into something like an accidental YAML-injection that results in terrible errors because I miscounted the number of spaces to indent something.
I wrote a small terraform wrapper around helm provider that basically does what you’re saying. official kubernetes + tf support is poor, but it’s been working well for me. I rarely if ever have to touch the yaml templates that I maintain.
however, this is usually true with working with helm in general if you are using charts other people maintain. That’s one of the strengths of helm. you just shove your values into the chart and it should work. Maintaining charts is not fun though which is why I wrote the wrapper for my purposes.
The funny thing is helm is as good or bad as what you make it. When folks complain about helm they are actually complaining about their own self created charts or poor selection of charts they install.
I like wrapping with kapp for this. However kustomize still skips hooks if you inflate a chart that uses them and there is value in discoverability of helm apps installed with clear versioning
I don’t think there is a lot of overlap between people who say lisp sucks because of the parens and people who are fine with using a template to generate yaml.
You place malicious binaries outside the helm directory. Helm will now execute malicious code through the helm chart pointing outside the helm directory.
Don't I have already bigger problems if you have access to my file system to place there malicious code?
Is the danger here that one can get an execute permission? But if you can manipulate my helm chart why can you not also place the malicious code in the helm directory?
Yeah, there is a rather strong "downloading and executing arbitrary code from the Internet may lead to execution of arbitrary code" kind of vibe there.
I think that there are "legitimate" use cases for symlinks that read from outside the root, which at this point are probably looked upon even less favorably. It's likely that making the change you're proposing would be backwards incompatible.
I agree that it's not clearly explained why this isn't a concern though. A cursory search for other instances of os.WriteFile doesn't seem to surface any thorough controls...
edit: ok actually it looks like the lockfile is special because it's the only instance of helm itself directly writing a file on behalf of a package consumer
If you have a chart that has `deploy.yaml` symlinked to `/home/john/testcharts/redis/deploy.yaml`, that chart is clearly not going to work on anyone's machine except john's, so that chart is useless on anyone else's machine.
If you're saying the use-case is for charts that aren't distributed, well, I'm saying we should ban all symlinks on distribution (downloading and unpacking a chart should fail if it has symlinks outside of the root), and I just can't imagine any use-case where a distributed chart with external symlinks makes sense.
If this whole thing is about charts that aren't distributed, but local to some developer's machine, well, in that case who cares if the developer can pwn themselves by typing "ln -s ~/.bashrc Chart.lock", they could have just pwned themselves by typing "bash" even more quickly.
Ya, I mean, I put "legitimate" in quotes for a reason. I think most people agree with you. This has been a thing that they've been aware of and struggling with for a while.
Smattering an --allow-symlinks flag all over their commands seems to be the least inelegant way to handle this while still giving users an easy way to maintain compatibility. Maybe they'll come around to it after this.
I have use cases for linking Terraform lock files to keep various deployments/modules on consistent versions. I could see there being a use case for symlinking Chart.lock files although usually that's limited to an internal implementation and not something a general purpose chart would probably ship
i.e. you have 3 different charts that all depends on `cache`, `load balancer` and `database` charts and you want to only ever have 1 version deployed of those subcharts so you want the parent chart locks linked
What is the attack scenario here? Where are the security boundaries? How does the attacker gets their repository with a symlink in it to the victim? Is Helm typically run as a privileged user? How would this work? And why doesn't the vulnerability description give answers to these questions?
Given the details in the article, I think even something as simple a templating a chart from a repository might be vuln., but it likely depends on a lot of exact specifics.
> Where are the security boundaries?
I expect templating does not result in LCE.
> How does the attacker gets their repository with a symlink in it to the victim?
The attacker owns the repository. They can serve whatever maliciousness in it they want. But should templating a malicious chart result in LCE?
> Is Helm typically run as a privileged user?
Enough so, yes, because the rendered result is often pushed to a k8s cluster. "Privileged" here might not be "root", but it might be "this user has k8s API access".
Imagine, e.g., that the attacker's LCE here might be "push ~/.kube to attacker".
> And why doesn't the vulnerability description give answers to these questions?
Familiarity with the tools involved is an normal assumption.
Pretty cool and nice find. I already have a "malicious" Chart.yaml in mind for this attack just based on the description of vuln.
Fortunately, my dotfiles are managed with nix so trying to write to those files on a read only partition will raise many red flags for me.
I don't use bash, but maybe should write a dummy .bashrc (and other start up script equivalents for fish) as some sort of canary.
If I happen to overlook the malicious shell script crafted in a dependency on helm chart, I would get nasty errors that a process was trying to write to a read only file.
You use invisible characters whenever you press Enter or Space. If you're referring to Tab, many of the most popular programming languages like Go and Python use them as part of their syntax.
The reason YAML was popularized is because it was a response to XML which isn't user friendly to write. It's unfortunate that the spec got so convoluted, and uses a lot of implicit behavior, but I'd rather write YAML than XML, JSON or TOML for things like configuration files. Nowadays there might be better alternatives, but YAML is the de facto standard.
It's also unfortunate that YAML got abused by people who wanted to turn it into a DSL, so we ended up with thousands of lines of Ansible playbooks, CI workflows, and Helm charts, but here we are.
The point is that "going right ahead and write your .yml files in JSON" is not valid. You'd have to restrict yourself to a subset of JSON to not get different semantics.
Python's "PyYAML" package will not decode this to the same result as a JSON decoding.
Rust's `serde_yaml` will fail on this.
I don't know about other parsers, but I'd be curious to.
The standard itself isn't well written here, IMO.
> The content of a scalar node is an opaque datum that can be presented as a series of zero or more Unicode characters.
The example here is a "quoted scalar", which can contain the escapes you see. Those escapes represent "Unicode characters", specifically,
> Escaped 16-bit Unicode character.
But "Unicode characters" is never defined by YAML.
Most implementation seem to treat them as Unicode code points, and so thus the resulting string type in almost all cases in something like [UnicodeCodePoint]; in Rust, that means no unpaired surrogates, or we can't convert it to a Rust `String`, which is roughly speaking `[USV]`. In Python, that's workable, since that's Python's `str` datatype, but that means no surrogate decoding occurs.
The grammar also further implies that it's [UnicodeCodePoint] and not [USV], and the prose never restricts unpaired surrogates. (The JSON standard strongly implies the UTF-16 decoding should happen on escaped values, though it too waffles around unpaired surrogates. Whether unpaired surrogates are accepted is variable in JSON.)
But compare with a JSON string: a JSON string decodes to a something like a [USV], so surrogate pairs are decoded to their corresponding USV.
While YAML has all sorts of issues and disadvantages compared to XML, security is certainly not one of them. XML is a crazy source of security issues by design, especially with the crazy idea of adding built-in support for URLs that parsers are expected to follow.
I have no horse in that race but just to see people talking about XML like this a quarter of a century after the first time I saw similar comments is just funny, I don't care who you are.
NIH syndrome and "inverse second system effect". In the real second system effect, the second system is more complicated because it includes everything that could possibly be perceived as missing in the first system. In the inverse second system effect the first system was perceived as too complicated, not too simple, so the second system is much simpler and doesn't do its job well.
It is tangentially related in that yaml became normal to use as a DSL within the devops world. As another post said, everything becomes a DSL eventually because people want to be "fully configurable" not realizing that is roughly the same thing as not being complete yet. But in this case the lack of direct acknowledgement of yaml as an interpreted language with an interpreter that doesn't think of itself as such and hence doesn't have a real sandbox, is what leads us to the present. People didn't use xml as a DSL as often because it was so flexible. That would be like using c++ as a DSL instead to write the interpreter for one.
I wasn't blaming yaml at all. Our mistake is thinking we are using it as a configuration file. When we are actually using it as an interpreted language. Not yaml's fault people are writing dsl interpreters unknowingly. It's just related because people who make that mistake are picking yaml. I nearly made the mistake with toml a few years ago. You could even make the mistake with complicated environment variable usage. Whenever your configuration source is flexible enough to create executable primitives it needs to be sanitized. And really that is whenever a configurable value gets used in a conditional, which is often. Especially considering that even numeric values become conditional when they are used in operations that can result in ub or even just exceptions/panics/unhandled errors. Not a yaml exclusive.
I would argue that not sanitising strings is analogous to a form of memory unsafety. You take as an input, an opaque blob of bytes that you then pass on to a myriad of other libraries and pieces of code. Nothing is captured in the types other than "String". Mainstream programming languages need to make it easier to define new types and parse strings into them. Rust is very promising in this area, as it features algebraic data types.
The original vulnerability description is not worded very well, here's my understanding of what's going on:
1. Attacker crafts a malicious Chart.yaml containing arbitrary code
2. Replaces Chart.lock with a symlink pointing to a sensitive file (like .bashrc or other startup scripts)
3. When you run helm dependency update, Helm processes the malicious Chart.yaml and writes the payload to whatever file the symlink targets
4. Code executes when the targeted file is next used (e.g., opening a new shell)
Why This Works: Helm follows the symlink during the dependency update process without validating the target, allowing arbitrary file writes outside the intended chart directory.
Can anyone explain in what setup an attacker who can create a symlink where Chart.lock was could not directly write .bashrc or similar? Is this related to how Git handles symlinks?
A symlink is just a special file that contains a string of text, it's not tightly bound to the target like a hard link. You can write anything into that string of text, including, say, "~/.bashrc". Then you can ship that symlink onto another system, and it suddenly points to your .bashrc.
Git just moves symlinks across systems as is, so yes, you can use git to deploy the exploit.
As pedantry, to the very best of my knowledge symlinks could not contain "~" and have it mean $HOME - that's a shell-ism (or os.path.expanduser equivalent in your library). I was suspecting the attack vector may have used "/home/runner" or "/home/ubuntu" as very common paths that could exist and be writable by the user
This has nothing to do with Git. A symlink can be packaged up in a tarball and shipped from one system to another. An attacker would need to create a malicious Chart.yaml file and a Chart.lock file pointing to another file. Then ship those to a system where dependencies are then updated.
This doesn't affect things like installing or upgrading a chart. Dependencies aren't updated at that time.
> A symlink can be packaged up in a tarball and shipped from one system to another.
True enough, but if you have a victim unpacking and building untrusted tarballs there's no security boundary being crossed, is there? You don't have to bother with this symlink nonsense, just update the install script to include your payload directly.
Honestly this vulnerability is dumb. I don't see any realistic scenario where it can be exploited by an unprivileged attacker.
When you do a helm pull and download a chart from a repo, I believe it's a tar-ball. So if you have a workflow where you install charts from the filesystem you could be impacted. I've done that in the past.
I can only repeat the assertion: if you have a victim pulling and installing untrusted tarballs, there is no security boundary being crossed.
It doesn't matter whether it's "from a repo". If you can't trust the repo it can feed you whatever it wants.
You're not installing the untrusted tarball; helm is merely supposed to be extracting it, and then rendering the templates contained within.
(Those templates, once rendered, might then refer to pods, etc. that might be put into a k8s cluster (or perhaps we merely render then YAML, and never `apply` it), and in that sense, one might imagine that that is an install, but that's not the security boundary being crossed here; this would presumably result in execution on the host running helm, which would definitely be surprising.)
You're quibbling over the meaning of "install" but apparently conceding the part about untrusted? OK, fair enough. I still argue that any process involving the extraction and (ahem) "rendering of contained templates" from untrusted sources is broken in ways a fix for this particular symlink issue isn't going to address.
Yes? I don't find that that odd.
Certainly, it would be better to trust the upstream completely, but let's not kid ourselves? See the entire current state of software supply chain in the industry.
But when I visit a website, I don't expect the website to LCE me. Why should turning a YAML adlib into YAML LCE me, regardless of the trust of the upstream. This is not a privilege I'm expecting to give the upstream ever, and this behavior is a clear security bug, to me…
Helm is a program that allows users to creates packages which other users consume. Those packages contain files that are normally generated by Helm itself, but apparently if you alter your package definition by hand you can replace Chart.lock with a symlink.
As I'm typing this it's occurring to me that you probably shouldn't be able to do that. The fix they applied was to prevent the actual write from occurring when trying to write the lockfile and determining that the lockfile is a symlink. They could (should?) also validate that like, the package itself hasn't been screwed with in this manner.
Having read the CVE multiple times I am still unsure how 2. above happens? Is it possible through the malicious chart itself or is it a dependency for the CVE to be in play at all? And if the latter - what local process would write a symlink from a helm lock file to any kind of system start up script which doesn't point to a much bigger problem than this CVE?
The attacker creates a symlink (e.g., using `ln -s`) to another file. The attacker needs to create the malicious Chart.yaml file and symlink that the Chart.lock file points to.
If being able to create files and symlinks to them is a pre-condition for this, then it's not a serious security bug. If you have that kind of access then there are a million nefarious things you can do.
This is almost becoming a joke at this point, "assuming an attacker has access to the system, they can change things on the system".
Helm is not intended to be able to write files outside of the directory you are rendering the templates to, and the directory that you have downloaded the chart to, so if there is a way to do that, it is a bug in the program and a security bug at that, particularly when the destination is controlled by someone who has written a malicious chart. That it also happens to be able to run arbitrary code makes it worse, but the primary problem is that it can write files outside of the chart directory or the directory you are rendering to at all.
This has nothing to do with whether you are running it in sudo or whatever. (and in fact on MacOs, I don't believe this requires running it with sudo permissions to overwrite ~/.zshrc for example)
I create a malicious chart or compromise one you use (with symlink to an arbitrary file and code).
You download charts either as a tarball from a helm repo or oci registry with helm and helm will create the files and links with your permissions, and send me whatever I wanted to extract from your system.
Yes, you should check things you download from the internet. But also, that is not how a chart is supposed to work.
As noted in other comments, a symlink is just a text reference to a file. It does not need to be created on the host system.
It is on the level of "sudo curl URL". It is an obviously stupid thing to do from a security perspective, but projects have suggested doing it to install their software.
If you are new to helm or haven't considered the security around it, it is good to know what to look out for.
thats funny because Helm refused to allow reference of external files (there is a github issue) but they follow symlinks xD
Helm is an abomination, as the whole idea of using a text template engine to generate YAML is. And this vulnerability adds insult to injury ;)
Sorry, just can't really recover from trauma of counting spaces and messing up newlines, etc. when writing Helm templates. You know, Lisp "sucks" because "you need to count parenthesis" (you actually don't), yet Helm is a widely accepted technology where you need to count spaces for (n)indent ;)
I'm a dev that jumped to devops and one of my pet peeves will always be the lengths devops engineers go to avoid using a real programming language. Instead of interacting with all these APIs through python, ruby, lua, go, whatever they would rather build hodgepodge systems in bash, coreutils, curl (or wget. or both!) and jq (which is the worst). Or in the case of helm, just creating a half yaml/half Go SDK for generating YAML.
Even the helm infrastructure that I work in is completely wrapped in custom shell scripts that call all sorts of other commands to populate helm variables.
But yeah it's silly that helm templates require all sorts of {{ indent | 4 }} type incantations when the final YAML output is just sent through some kind of toJSON anyway.
You'll often find that if you write ops scripts in, say Python, it's largely calling external commands.
When that's the case, bash if often the better choice, especially if you know it well. It has an excellent REPL, is easy to trace and is already installed everywhere.
> You'll often find that if you write ops scripts in, say Python, it's largely calling external commands.
That's a totally fine trade off for actual sane array/list functionality, robust string manipulation etc. I'd rather form shell commands in a programming language than in bash. People seem to love it.
I do not think for a second though that the average person that "knows bash well" can read and comprehend a multi hundred line bash script written by someone else as fast or even correctly has a seasoned python dev reading python written by someone else.
> the lengths devops engineers go to avoid using a real programming language
OK, but, you know... those tools were created by literal devs. Not in yaml, in a "real language"! So apparently devs thought they needed that.
The argument should be towards all people - we love creating new abstractions and "simplify things", but we suck at honest evaluation of the impact of our creations.
Still: I absolutely hate Helm templating and think that the very existence of "helpers" (even in a default chart!) is an abomination.
> OK, but, you know... those tools were created by literal devs.
All `kubectl` does is create REST API calls to the control plane. So to be fair, what I'm grousing about can be accomplished just fine by a developer like me constructing API calls from python to update objects in k8s.
The problem is I work in devops where tooling written in proper languages with standard libraries that have things like useful arrays or robust string manipulation or ergonomic concurrency is a non-starter for some reason. The argument against that being mostly "I have to install the interpereter first".
Now we get to the point - you are unhappy with how your org/team handles it. Would that be better description? Don't put all DevOps in one basket. It's a messy term anyway.
Many orgs and many teams allow that. That's why alternatives to Helm exist like cdk8s, or Pulumi for rest of infra.
For Kubernetes there is such a variety of tools, it's overwhelming. The only problem to tackle is that Helm/Argo is so popular, that alternatives are hard to sell regardless of real benefits.
Look at how big is a scene of Operators for k8s - this is where the programmable part of k8s went. You configure operators with their own CRDs (usually) that can be static/plain declarative text. Then you write actual code of the operator that deals with all the complexity of enforcing that state. For me this is where typically what you are talking about goes, while high level "non programmable" code sits in yaml in a other repo for others to maintain. Maybe this is where you should also go, standardize your work for others to consume in a form of operator?
I fully get neglect in adopting more complex tooling. sh, curl, sed, awk... those things are present almost everywhere and it's not that hard to get lucky and make a script that will run on almost everything your org has. And it actually might be fine for a decade or so.
I myself could not do code (even scripting) because one of my companies literally treated scripting in anything as development which was strictly regulated (so forbidden for any non dedicated dev role). Or an org that had not a single server in whole DC that has Python 3, years after it's release. Or more recent: some damn Ubuntu LTS that can't be easily upgraded to just 2-3 minor versions that this cool k8s library uses. Maintaining python versions on VMs is a pain in the ass, especially if your org has strict controls. Internet access to pip is not granted as well. UV gets the job done nowadays, if it's allowed, but long story short: that "fear of real language" can be as much lack of knowledge/skills as pragmatism that came from painful experiences.
Yeah, vi has supported % as "jump between matching parenthesis" since it's original release in the 1970s, and vim by default will do simple parenthesis matching and highlighting, I don't see why everyone is so scared of touching lisp for these reasons with modern editors (if your editor doesn't support either of the above... maybe it's not modern enough?)
This isn't a uniquely helm thing though, they mostly use modified go templating. Lots of other things do this with yaml as well.
… and I think I'd argue that the parent's argument against the tooling would apply equally as well to those "other things", too.
The alternative here is something that manipulates the data structure directly. E.g., it might permit me to say:
(This is in some pseudo-imperative language, vs. the parent's Lisp, but that distinction isn't particular relevant to the core of their argument, I think.)And then at the end, the thing itself takes care of converting the resulting objects to YAML, thus preventing me from inadvertently turning what is meant to be a string into something like an accidental YAML-injection that results in terrible errors because I miscounted the number of spaces to indent something.
I wrote a small terraform wrapper around helm provider that basically does what you’re saying. official kubernetes + tf support is poor, but it’s been working well for me. I rarely if ever have to touch the yaml templates that I maintain.
however, this is usually true with working with helm in general if you are using charts other people maintain. That’s one of the strengths of helm. you just shove your values into the chart and it should work. Maintaining charts is not fun though which is why I wrote the wrapper for my purposes.
The funny thing is helm is as good or bad as what you make it. When folks complain about helm they are actually complaining about their own self created charts or poor selection of charts they install.
Something like kustomize was a better approach, where resources are templates semantically.
Though it's lacking in several ways, like good destroy functionality.
I like wrapping with kapp for this. However kustomize still skips hooks if you inflate a chart that uses them and there is value in discoverability of helm apps installed with clear versioning
I don’t think there is a lot of overlap between people who say lisp sucks because of the parens and people who are fine with using a template to generate yaml.
So the attack vector is:
- You have access to my file system
- You have access to the helm repository
You place malicious binaries outside the helm directory. Helm will now execute malicious code through the helm chart pointing outside the helm directory.
Don't I have already bigger problems if you have access to my file system to place there malicious code?
Is the danger here that one can get an execute permission? But if you can manipulate my helm chart why can you not also place the malicious code in the helm directory?
> You place malicious binaries outside the helm directory
No, helm is the one doing this part in the vuln. Chart.lock is made a symlink to some important file, and helm will happily write to it.
Yeah, there is a rather strong "downloading and executing arbitrary code from the Internet may lead to execution of arbitrary code" kind of vibe there.
Starting on the other side of the airtight hatchway: https://devblogs.microsoft.com/oldnewthing/20221004-00/?p=10...
Seems the normal mitigations apply i.e. validate with hash or save a local copy. Validate new versions before adopting
And yet you just described the behavior of many mid-size company "DevOps" departments.
> But if you can manipulate my helm chart why can you not also place the malicious code in the helm directory?
If you can manipulate my helm chart, why not just do the RCE directly in my kubernetes cluster or whatever?
That description seems really unclear, like how can `Chart.lock` be a symlink to a `.bashrc`?
Is the vulnerability that you ship a chart with `Chart.lock -> ../.bashrc`, and then helm writes to `Chart.lock`?
Why is the fix specific to Chart.lock (https://github.com/helm/helm/commit/76fdba4c8c2a4829a6b7abb4...), wouldn't the fix be instead that "A chart cannot contain any symlinks outside of its root"?
I think that there are "legitimate" use cases for symlinks that read from outside the root, which at this point are probably looked upon even less favorably. It's likely that making the change you're proposing would be backwards incompatible.
I agree that it's not clearly explained why this isn't a concern though. A cursory search for other instances of os.WriteFile doesn't seem to surface any thorough controls...
edit: ok actually it looks like the lockfile is special because it's the only instance of helm itself directly writing a file on behalf of a package consumer
What use-case?
If you have a chart that has `deploy.yaml` symlinked to `/home/john/testcharts/redis/deploy.yaml`, that chart is clearly not going to work on anyone's machine except john's, so that chart is useless on anyone else's machine.
If you're saying the use-case is for charts that aren't distributed, well, I'm saying we should ban all symlinks on distribution (downloading and unpacking a chart should fail if it has symlinks outside of the root), and I just can't imagine any use-case where a distributed chart with external symlinks makes sense.
If this whole thing is about charts that aren't distributed, but local to some developer's machine, well, in that case who cares if the developer can pwn themselves by typing "ln -s ~/.bashrc Chart.lock", they could have just pwned themselves by typing "bash" even more quickly.
Ya, I mean, I put "legitimate" in quotes for a reason. I think most people agree with you. This has been a thing that they've been aware of and struggling with for a while.
https://helm.sh/blog/2019-10-30-helm-symlink-security-notice...
Smattering an --allow-symlinks flag all over their commands seems to be the least inelegant way to handle this while still giving users an easy way to maintain compatibility. Maybe they'll come around to it after this.
I have use cases for linking Terraform lock files to keep various deployments/modules on consistent versions. I could see there being a use case for symlinking Chart.lock files although usually that's limited to an internal implementation and not something a general purpose chart would probably ship
i.e. you have 3 different charts that all depends on `cache`, `load balancer` and `database` charts and you want to only ever have 1 version deployed of those subcharts so you want the parent chart locks linked
What is the attack scenario here? Where are the security boundaries? How does the attacker gets their repository with a symlink in it to the victim? Is Helm typically run as a privileged user? How would this work? And why doesn't the vulnerability description give answers to these questions?
> What is the attack scenario here?
Given the details in the article, I think even something as simple a templating a chart from a repository might be vuln., but it likely depends on a lot of exact specifics.
> Where are the security boundaries?
I expect templating does not result in LCE.
> How does the attacker gets their repository with a symlink in it to the victim?
The attacker owns the repository. They can serve whatever maliciousness in it they want. But should templating a malicious chart result in LCE?
> Is Helm typically run as a privileged user?
Enough so, yes, because the rendered result is often pushed to a k8s cluster. "Privileged" here might not be "root", but it might be "this user has k8s API access".
Imagine, e.g., that the attacker's LCE here might be "push ~/.kube to attacker".
> And why doesn't the vulnerability description give answers to these questions?
Familiarity with the tools involved is an normal assumption.
[dead]
Questions like this make me wonder if "hacker" news needs a rebranding.
Basic tech news?
Capitalist news?
Vulture Capitalist news?
Pretty cool and nice find. I already have a "malicious" Chart.yaml in mind for this attack just based on the description of vuln.
Fortunately, my dotfiles are managed with nix so trying to write to those files on a read only partition will raise many red flags for me.
I don't use bash, but maybe should write a dummy .bashrc (and other start up script equivalents for fish) as some sort of canary.
If I happen to overlook the malicious shell script crafted in a dependency on helm chart, I would get nasty errors that a process was trying to write to a read only file.
As an aside, all these tools like aider, claude desktop ask for shell access to run codes.
Allowing LLMs to generate charts and what not via shell execution is a bad idea.
For a moment I thought it was about the synth…!
https://tytel.org/helm/
For a moment I thought it was about the Emacs package.
https://github.com/emacs-helm/helm
Likewise! Phew!
Although, the whole can of worms regarding synth/audio exploits is a pretty wild scene ..
[dead]
If we're being honest, YAML is one of the dumbest ideas of the last 20 years to have proliferated. How we got from XML to here I cannot comprehend.
This is not the first RCE involving YAML and it won't be the last.
That was not RCE. It's not in yaml, it's in Helm's logic.
But glad you vented, I guess.
Why we settled on a file format that relies on invisible characters I'll never know.
The gyrations people will go through to avoid using S-expressions...
You use invisible characters whenever you press Enter or Space. If you're referring to Tab, many of the most popular programming languages like Go and Python use them as part of their syntax.
The reason YAML was popularized is because it was a response to XML which isn't user friendly to write. It's unfortunate that the spec got so convoluted, and uses a lot of implicit behavior, but I'd rather write YAML than XML, JSON or TOML for things like configuration files. Nowadays there might be better alternatives, but YAML is the de facto standard.
It's also unfortunate that YAML got abused by people who wanted to turn it into a DSL, so we ended up with thousands of lines of Ansible playbooks, CI workflows, and Helm charts, but here we are.
> many of the most popular programming languages like Go and Python use them as part of their syntax
Go doesn't use tabs or whitespace as a part of its syntax. It's a part of the formatting, but not the syntax of the language.
Python on the other hand, one extra tab or whitespace can cause havoc.
It's unfortunate, but inevitable. Every structured text data format that sees widespread use, given enough time, will eventually be turned into a DSL.
In fact, once a structured text format is used as a data source for any process, it has already become a DSL.
i always enjoy writting json more. I feel it's easier to translate/integrate json into the code.
YAML is a superset of JSON, so go right ahead and write your .yml files in JSON.
YAML is actually not a superset of JSON.
https://john-millikin.com/json-is-not-a-yaml-subset
https://news.ycombinator.com/item?id=30052633
The NO case is not valid JSON.
So that leaves scientific notation.
The point is that "going right ahead and write your .yml files in JSON" is not valid. You'd have to restrict yourself to a subset of JSON to not get different semantics.
If you configure the parser to treat it as YAML 1.2 then you don't need to restrict yourself to a subset.
This is a valid JSON value:
Python's "PyYAML" package will not decode this to the same result as a JSON decoding.Rust's `serde_yaml` will fail on this.
I don't know about other parsers, but I'd be curious to.
The standard itself isn't well written here, IMO.
> The content of a scalar node is an opaque datum that can be presented as a series of zero or more Unicode characters.
The example here is a "quoted scalar", which can contain the escapes you see. Those escapes represent "Unicode characters", specifically,
> Escaped 16-bit Unicode character.
But "Unicode characters" is never defined by YAML.
Most implementation seem to treat them as Unicode code points, and so thus the resulting string type in almost all cases in something like [UnicodeCodePoint]; in Rust, that means no unpaired surrogates, or we can't convert it to a Rust `String`, which is roughly speaking `[USV]`. In Python, that's workable, since that's Python's `str` datatype, but that means no surrogate decoding occurs.
The grammar also further implies that it's [UnicodeCodePoint] and not [USV], and the prose never restricts unpaired surrogates. (The JSON standard strongly implies the UTF-16 decoding should happen on escaped values, though it too waffles around unpaired surrogates. Whether unpaired surrogates are accepted is variable in JSON.)
But compare with a JSON string: a JSON string decodes to a something like a [USV], so surrogate pairs are decoded to their corresponding USV.
Sometimes what makes something great is what it lacks. An automatic transmission, operator overloading, schema extensions, batteries etc.
[dead]
Exactly how I feel about Python!
While YAML has all sorts of issues and disadvantages compared to XML, security is certainly not one of them. XML is a crazy source of security issues by design, especially with the crazy idea of adding built-in support for URLs that parsers are expected to follow.
Are we going to blame the next RCE we find in some application on XML just because that application uses XML somewhere?
If so, then I agree on blaming this on YAML.
I have no horse in that race but just to see people talking about XML like this a quarter of a century after the first time I saw similar comments is just funny, I don't care who you are.
A search for XML on cve.org gives
> Showing 1 - 25 of 6,749 results for XML
Searching for YAML:
> Showing 1 - 25 of 288 results for YAML
Is that from the past two years?
NIH syndrome and "inverse second system effect". In the real second system effect, the second system is more complicated because it includes everything that could possibly be perceived as missing in the first system. In the inverse second system effect the first system was perceived as too complicated, not too simple, so the second system is much simpler and doesn't do its job well.
Also this vuln has nothing to do with YAML
It is tangentially related in that yaml became normal to use as a DSL within the devops world. As another post said, everything becomes a DSL eventually because people want to be "fully configurable" not realizing that is roughly the same thing as not being complete yet. But in this case the lack of direct acknowledgement of yaml as an interpreted language with an interpreter that doesn't think of itself as such and hence doesn't have a real sandbox, is what leads us to the present. People didn't use xml as a DSL as often because it was so flexible. That would be like using c++ as a DSL instead to write the interpreter for one.
This is like blaming python problems on yaml because someone embedded a python script in a multiline string.
I wasn't blaming yaml at all. Our mistake is thinking we are using it as a configuration file. When we are actually using it as an interpreted language. Not yaml's fault people are writing dsl interpreters unknowingly. It's just related because people who make that mistake are picking yaml. I nearly made the mistake with toml a few years ago. You could even make the mistake with complicated environment variable usage. Whenever your configuration source is flexible enough to create executable primitives it needs to be sanitized. And really that is whenever a configurable value gets used in a conditional, which is often. Especially considering that even numeric values become conditional when they are used in operations that can result in ub or even just exceptions/panics/unhandled errors. Not a yaml exclusive.
In what way is this vulnerability YAML-specific?
But I thought security vulnerabilities couldn't happen in memory-safe languages!
But I thought accidents wouldn't happen if we wear helmets! Clearly they're worthless!
Sarcasm aside: wearing a helmet causes riders to take more risks, leading to more accidents.
https://www.sciencedirect.com/science/article/pii/S136984781...
I’d still wear one, but also try to be more careful knowing that the helmet provides a false sense of security.
I do believe the analogy holds very true with programming habits.
Did you read the abstract? It says the exact opposite:
> this systematic review found little to no support for the hypothesis bicycle helmet use is associated with engaging in risky behaviour.
What! You’re lying!
I would argue that not sanitising strings is analogous to a form of memory unsafety. You take as an input, an opaque blob of bytes that you then pass on to a myriad of other libraries and pieces of code. Nothing is captured in the types other than "String". Mainstream programming languages need to make it easier to define new types and parse strings into them. Rust is very promising in this area, as it features algebraic data types.
This isn't a memory bug.
And Helm isn't written in Rust, so their snark was doubly misplaced
Fwiw, Go is also considered memory-safe although not as strict as Rust
Wondering how this kind of thing can be automatically discovered by an LLM. Anyone have any experience?
All the maintainers who get bombarded by LLM generated CVEs have a lot of experience with this.
Ask an LLM and find out