I do thank the advent of LLM's for making the task of tracing through 3 layers of YAML rendering indirection to find an actual bug in layers of insanity. They are surprisingly good at it. A purist might look at the system and be taken aback in horror, but pragmatically the crazy templating things that used to be pretty major in terms of maintainability have become a lot less problematic.
I don't even mind text interpolation in helm that much. Yes, it's horrific, but it really doesn't become problematic until you are having an external system render the text being sent to helm that might be processed again after the fact with Kustomize. Your text might hit literally 3-4 different YAML rendering pipelines before it finally hits your cluster. Insanity
Hashicorp had a similar idea to this configuration language you are mentioning. They named it HCL. We do know how that turned out, but maybe that was more questionable design rather than problems endemic to the approach.
Even if you're all in on AI, it's still not a great situation. Do you want your agent to burn 10000 tokens fixing indentation bugs every time it touches a helm chart?
What takes effort for humans, takes effort for AI too.
And yeah, HCL was a horrible design. Anything else would have been better, Cue, jsonnet, dhall, pkl, etc. (I'm not sure which of those existed when HCL was created.)
I especially enjoyed where you discovered that K8s runs on string interpolation. That feeling never goes away, FYI; every time you have to play with nindent, you lose a little piece of your sanity.
This is partially a tangent, but the whole managing of software infrastructure is a strange culture. There would of course be lots of better ways, but it's too bad the infrastructure side of the software industry lives partially in a parallel world and many of the lessons from elsewhere in the industry are not really widespread there (at least yet). Even banks seem intelligent and innovative compared to infrastructure management mainstream practices.
It's very error prone to describe a hierarchy of versioned software packages and configuration with shell scripts and yaml. Or use text files in git as a database. Or use the CI system as a bad substitute for a proper service (in order to do a simple thing, you must first create the world from scratch). And so on. Mainstream industry knows types, programming languages, relational databases, graphs and you name it, but those have not yet penetrated sufficiently into the infrastructure world. The understanding of the value of having types has been paid in sweat and blood when developing software and databases. Even a basic CRUD application with a basic SQL database will run rings around these Frankenstein monsters: in safety, speed, and extendability.
> It's very error prone to describe a hierarchy of versioned software packages and configuration with shell scripts and yaml.
We have a multitude of configuration/markup languages all doing the exact same thing.
We have solved this problem in the 60s: S-Expressions. Should have just used that for everything. Configuration, code, websites, you name it. Infinitely extendable, easy to parse.
I'm a lisp nerd, but why do you say sexps are the answer here? I think the point of a lot of configuration formats is that they don't want their configuration to also be a full-fledged programming language.
> I think the point of a lot of configuration formats is that they don't want their configuration to also be a full-fledged programming language.
That seems to be everyone's initial goal, but then it inevitably gets leaky.
I think the dominant split here that forces everything into a weird mismash is that nobody really wants to have to understand arbitrary programming language configs for third-party stuff, they want to read config and know what's going on. BUT for their own stuff, they find reasons to want more power and expressibility...
There has been some config languages, like Cue, that are not Turing complete. It is useful of have them always complete and have deterministic evaluation.
Cue also has a lot of schema validation which is better than dynamic languages.
I set up a whole startup using Kustomize. It was great, except almost no-one used the inheritance feature except me - they all copy-pasted everything. Then the company grew and hired some dedicated devops people, who immediately started switching to Helm. I threw up my hands and went back to focusing on software development.
I’ve done some NGO-related work and in that context, they talk about “appropriate technology” - technology that’s suited to the local population, among other things. I’ve started to see tools like Helm and languages like Go as appropriate technology for many teams, but that doesn’t mean I’m going to switch to a pedal-powered water pump in my own house.
The point is not how pretty the strings are. The point is there should be no strings at all. These are computer programs, so they should use normal objects. The python code `my_pod = nignx_pod()` will never throw a syntax error because the result has too many/too few spaces.
no one is stopping you from importing k8s types as protobufs and writing a program to talk to k8s api via protobuf encoding. all the types are already protos (except CRDs). it's just not the most user-friendly way so most people don't use it.
And plenty more exist today. But helm got popular first, so due to the collective action problem it's going to take the industry forever to switch off it.
Wait, but how does that help? Are text templates any better suited for creating JSON than they are for creating YAML? Trying to manipulate well-structured data as if it were syntax-free text seems downright deranged to me, regardless of the specifics of the format ...
Is it somewhat convoluted to setup? Absolutely! But it's possible to use relatively robust and well supported built in controllers with custom metrics.
Helm is a great tool, but you absolutely must keep it simple. We use Helm at work, and our charts are 95% hard-coded YAML templates, with an occasional sprinkling of a Helm variable for values which differ between dev/test/prod. It works great and is a ton easier than Kustomize for that purpose. But I would never want to maintain the super complex Helm charts you see in public repos.
> We use Helm at work, and our charts are 95% hard-coded YAML templates, with an occasional sprinkling of a Helm variable for values which differ between dev/test/prod. It works great and is a ton easier than Kustomize for that purpose.
I have come to precisely the opposite conclusion. I can count all of the helm charts that didn't make me immediately want to cry with one hand... and I'd have at least 2 fingers still free.
If you have "base, and then we change three things per environment", Kustomize is about as good as it gets using simple patches.
at $dayJob, we have templates wrapping templates and not a ton of tests on anything. Lots of implicit defaults that are not documented anywhere other than in the master/root chart defaults. As far as I can tell, this was how policy was meant to be enforced rather than something like OPA.
Using nix to generate k8s yaml is actually quite pleasant. You avoid all the string templating bullshit and can use nix functions and the nixos module system to build proper abstractions.
If we had to live in a world with YAML, that world would have been so much better had every library that emits YAML defaulted to "flow" style instead of "block" style. Well, and had the standard fixed things so that the "|" multi-line string operator worked in "flow" style. That's a very convenient operator for embedding scripts into configuration.
But no, apparently '{' characters are scary or something.
K8s is a reconciliation engine for desired state, like many other adjacent tech in the infra space. Hating on yaml, json, missing types, or templating layers that drive the engine is kind of silly, and usually missing the point.. like hating dbs because you take issue with a sql layer or an orm layer. Disliking one or the other isn’t a deal breaker that invalidates the value proposition, so you may want to consider the other one before you retreat to flat text in csvs.
Kubernetes can be driven with tofu and hcl.. make it operator and even driven.. use any of many sdks, cdks to drive it, or add a completely different layer for faas or orchestration if that’s the relevant vocab for your problem space. It all runs on k8s.. that’s kinda the point.
I've written at length about my disdain for text interpolation in helm.
I've done some experiments using Apple's Pkl and CNCF's KCL, which look to solve these problems with a simple "configuration language".
I don't think either have very widespread adoption, though I did hear a while back that Apple is using Pkl internally.
I do thank the advent of LLM's for making the task of tracing through 3 layers of YAML rendering indirection to find an actual bug in layers of insanity. They are surprisingly good at it. A purist might look at the system and be taken aback in horror, but pragmatically the crazy templating things that used to be pretty major in terms of maintainability have become a lot less problematic.
I don't even mind text interpolation in helm that much. Yes, it's horrific, but it really doesn't become problematic until you are having an external system render the text being sent to helm that might be processed again after the fact with Kustomize. Your text might hit literally 3-4 different YAML rendering pipelines before it finally hits your cluster. Insanity
Hashicorp had a similar idea to this configuration language you are mentioning. They named it HCL. We do know how that turned out, but maybe that was more questionable design rather than problems endemic to the approach.
Even if you're all in on AI, it's still not a great situation. Do you want your agent to burn 10000 tokens fixing indentation bugs every time it touches a helm chart?
What takes effort for humans, takes effort for AI too.
And yeah, HCL was a horrible design. Anything else would have been better, Cue, jsonnet, dhall, pkl, etc. (I'm not sure which of those existed when HCL was created.)
It's why, except for upstream charts, I use exclusively jsonnet (through kubecfg) when I can.
String interpolation is a pox on the ecosystem, especially given how YAML was only originally a debug helper...
I especially enjoyed where you discovered that K8s runs on string interpolation. That feeling never goes away, FYI; every time you have to play with nindent, you lose a little piece of your sanity.
This is partially a tangent, but the whole managing of software infrastructure is a strange culture. There would of course be lots of better ways, but it's too bad the infrastructure side of the software industry lives partially in a parallel world and many of the lessons from elsewhere in the industry are not really widespread there (at least yet). Even banks seem intelligent and innovative compared to infrastructure management mainstream practices.
It's very error prone to describe a hierarchy of versioned software packages and configuration with shell scripts and yaml. Or use text files in git as a database. Or use the CI system as a bad substitute for a proper service (in order to do a simple thing, you must first create the world from scratch). And so on. Mainstream industry knows types, programming languages, relational databases, graphs and you name it, but those have not yet penetrated sufficiently into the infrastructure world. The understanding of the value of having types has been paid in sweat and blood when developing software and databases. Even a basic CRUD application with a basic SQL database will run rings around these Frankenstein monsters: in safety, speed, and extendability.
> It's very error prone to describe a hierarchy of versioned software packages and configuration with shell scripts and yaml.
We have a multitude of configuration/markup languages all doing the exact same thing.
We have solved this problem in the 60s: S-Expressions. Should have just used that for everything. Configuration, code, websites, you name it. Infinitely extendable, easy to parse.
I'm a lisp nerd, but why do you say sexps are the answer here? I think the point of a lot of configuration formats is that they don't want their configuration to also be a full-fledged programming language.
> I think the point of a lot of configuration formats is that they don't want their configuration to also be a full-fledged programming language.
That seems to be everyone's initial goal, but then it inevitably gets leaky.
I think the dominant split here that forces everything into a weird mismash is that nobody really wants to have to understand arbitrary programming language configs for third-party stuff, they want to read config and know what's going on. BUT for their own stuff, they find reasons to want more power and expressibility...
> they don't want their configuration to also be a full-fledged programming language.
They say they don't want it but every configurmation format eventually ends up becoming turing complete at some point.
There has been some config languages, like Cue, that are not Turing complete. It is useful of have them always complete and have deterministic evaluation.
Cue also has a lot of schema validation which is better than dynamic languages.
I definitely prefer using kustomize over helm, where the mentioned interpolation takes place.
I set up a whole startup using Kustomize. It was great, except almost no-one used the inheritance feature except me - they all copy-pasted everything. Then the company grew and hired some dedicated devops people, who immediately started switching to Helm. I threw up my hands and went back to focusing on software development.
I’ve done some NGO-related work and in that context, they talk about “appropriate technology” - technology that’s suited to the local population, among other things. I’ve started to see tools like Helm and languages like Go as appropriate technology for many teams, but that doesn’t mean I’m going to switch to a pedal-powered water pump in my own house.
You can do JSON instead of YAML.
But no one does that because it is significantly uglier. So I guess we don't deserve sanity because we don't want our programs to be ugly?
The point is not how pretty the strings are. The point is there should be no strings at all. These are computer programs, so they should use normal objects. The python code `my_pod = nignx_pod()` will never throw a syntax error because the result has too many/too few spaces.
Of course it will. Just add a few spaces at the front of the line and it’ll throw an error.
no one is stopping you from importing k8s types as protobufs and writing a program to talk to k8s api via protobuf encoding. all the types are already protos (except CRDs). it's just not the most user-friendly way so most people don't use it.
Isn't this crossplane? https://www.crossplane.io/
no, you don't have to resort to imperative protobuf slinging.
https://tanka.dev/
https://www.pulumi.com/
And plenty more exist today. But helm got popular first, so due to the collective action problem it's going to take the industry forever to switch off it.
Wait, but how does that help? Are text templates any better suited for creating JSON than they are for creating YAML? Trying to manipulate well-structured data as if it were syntax-free text seems downright deranged to me, regardless of the specifics of the format ...
> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"
Since Kubernetes 1.23, there's support for using "custom metrics" to control HPAs: https://kubernetes.io/blog/2026/07/14/custom-metrics-exporte...
Is it somewhat convoluted to setup? Absolutely! But it's possible to use relatively robust and well supported built in controllers with custom metrics.
> Is it somewhat convoluted to setup? Absolutely!
Kubernetes™
The node equivalent, also very useful for engineering cost effective cloud environments, is called KEDA.
Using literally anything other than helm will make your life 100x better with k8s.
Helm is a great tool, but you absolutely must keep it simple. We use Helm at work, and our charts are 95% hard-coded YAML templates, with an occasional sprinkling of a Helm variable for values which differ between dev/test/prod. It works great and is a ton easier than Kustomize for that purpose. But I would never want to maintain the super complex Helm charts you see in public repos.
> We use Helm at work, and our charts are 95% hard-coded YAML templates, with an occasional sprinkling of a Helm variable for values which differ between dev/test/prod. It works great and is a ton easier than Kustomize for that purpose.
I have come to precisely the opposite conclusion. I can count all of the helm charts that didn't make me immediately want to cry with one hand... and I'd have at least 2 fingers still free.
If you have "base, and then we change three things per environment", Kustomize is about as good as it gets using simple patches.
at $dayJob, we have templates wrapping templates and not a ton of tests on anything. Lots of implicit defaults that are not documented anywhere other than in the master/root chart defaults. As far as I can tell, this was how policy was meant to be enforced rather than something like OPA.
> Given the Controller pattern, why isn't there support for "Cloud Native" architectures?
you're supposed to write your own.
> Why are the storage and networking implementations "out of tree" (CNI / CSI)?
> Given the above question, why is there explicit support for Cloud providers?
the in-tree stuff came first, and they have been extracting it in pieces for years now.
Kubernetes vs SLURM: https://slurm.schedmd.com/SC23/Slurm-and-or-vs-Kubernetes.pd...
905% about K8s just link to the overview: https://www.nvidia.com/en-us/software/slurm/
Using nix to generate k8s yaml is actually quite pleasant. You avoid all the string templating bullshit and can use nix functions and the nixos module system to build proper abstractions.
> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"
I think you found the answer to that below - KEDA
Thanks for writing this up. The prose was really pleasant to read, a treat in this day.
YAML abstracting YAML abstracting YAML abstracting..
I hate it so much. I got moved from a software role to a devops/platform engineering role after some contract shifts and it's miserable.
At least some years ago, I could cope by viewing it as "resume fodder" but that doesn't even matter anymore.
If we had to live in a world with YAML, that world would have been so much better had every library that emits YAML defaulted to "flow" style instead of "block" style. Well, and had the standard fixed things so that the "|" multi-line string operator worked in "flow" style. That's a very convenient operator for embedding scripts into configuration.
But no, apparently '{' characters are scary or something.
K8s is a reconciliation engine for desired state, like many other adjacent tech in the infra space. Hating on yaml, json, missing types, or templating layers that drive the engine is kind of silly, and usually missing the point.. like hating dbs because you take issue with a sql layer or an orm layer. Disliking one or the other isn’t a deal breaker that invalidates the value proposition, so you may want to consider the other one before you retreat to flat text in csvs.
Kubernetes can be driven with tofu and hcl.. make it operator and even driven.. use any of many sdks, cdks to drive it, or add a completely different layer for faas or orchestration if that’s the relevant vocab for your problem space. It all runs on k8s.. that’s kinda the point.