Most of the DevOps folks I’ve encountered, unfortunately, couldn’t code much. So it was mostly ops work with a fancy new title.
The idea of a single entity being responsible for development, operations, observability, and support is flawed from the start. That’s not a one-person job, and the expectation simply doesn’t scale. So DevOps often ends up being either ops folks or dev folks, and rarely a true blend of the two.
What we need are feature-focused developers, ops-savvy devs who can deploy their own work, and a strong team dedicated to observability and applying modern SRE practices.
So I think curious developers who aren’t afraid of infra, along with a solid platform engineering team, are a real improvement over the status quo.
> Most of the DevOps folks I’ve encountered, unfortunately, couldn’t code much. So it was mostly ops work with a fancy new title.
The expectations were (and are) irrealistic.
Looking for somebody that has both the software engineering skills of a software engineer and the skills of a sysadmin, willing to do two jobs for the salary of one: yeah sure, keep looking for it.
That being said, i'm one of those that jumped on the bandwagon because in practice it meant fancier job title, higher pay and getting to use newer (often better) tooling.
Ten years ago being a devops engineer rather than a sysadmin usually meant getting to work with EC2 and cloud stuff rather than administering remote physical servers and fighting with NetApp SANs.
Devops was meant to be really a methodology change, not really a job title.
In my experience whether DevOps works or not really depends on the management. I can setup all the automation you want, but management must back me when I tell you're supposed to use the automations rather than involve me into doing your work.
I can surely take feedback and improve whatever you need to be improved, but you *must* be using the automation.
Otherwise we go back and Development and Operations.
I've seen the world begore devops, and it wasn't that pretty either. We had 2 silos. Devs had a bunch of apps, ops too. Devs had no idea what the hardware did, ops had no idea what the software did. Users suffered, and assigning responsibility was impossible.
The dev in me hated throwing an app over the wall, only to hear about ops completely misconfiguring it. Or ops ignoring deployment guidelines and mucking up releases. Or getting a db with 2 records and being expected to test performance. Or getting complaints about prod crashing, but no logs, eror messages, or any kind of access, and ops couldn't notice that disk full might be a disk being full.
The ops in me hated devs being loose with passwords, asking huge mountains of resources, and getting apps thrown over the wall at me without any idea what to expect. Or devs logging sensitive stuff.
For me, devops was good for breaking up the silos. We wanted to have a group of people cooperating on a group of applications. Some people will have a dev or ops profile, and that's fine. People should respect that others know more of some aspects. But: no one should shield the applications from everybody else in such team. Everybody should strive to learn enough basics to understand both perspectives and priorities.
Starting a devops silo is just the worst possible option.
> We left product development teams without anyone focused on production. We undid everything that made DevOps work in the first place.
Very good point, that's the same I have observed for the past couple of years when working on a devops team. Product team engineers nowadays feels like spoiled kids, they had no current how server runs, and asked for things unreasonable. I still remembered someone came to my desk and asked for me to increase the mem request to 10s of GB, he claimed that's the best solution he could think of is to load everything in mem.. and very often people don't even know what status code means 500, 502, 503, 504...
I think that Big DevOps was the best thing that could have happened for systems-focused devs and SWE-like sysadmins. Their ability to straddle both domains was suddenly in demand and made them worth heaps more in the job market, even more so after gaining experience with Kubernetes and LLMs.
Good food for thought but the article assumes how currently DevOps and Dev teams (and there's also an Ops team?) operate and there's a lot of different ways in how "DevOps" is done at different organizations.
Take any combination of dev, platform engineering, devops, SRE, ops teams (existing or not) and combine with different responsibilities and how they work together and there's a company out there using that model.
All the devops I have encountered so far have been “sysadmin in a trench coat.” Most couldn’t write anything beyond basic Python and Shell scripts. It was mostly ops folks jumping into a fancy new title. So every time something would go poof, they had to pull in devs to debug the most basic stuff.
Sure you have a few people that are great at development and operations and understand (and can make complex changes) to both the infra and the codebase.
But those people are making $300k+ at FAANG companies.
Unfortunately they are also the ones who write all the blog posts and podcasts on devops
This article is really light on concrete examples and any kind of point outside of the title, which is a shame, because I think this is an extremely relevant discussion to most tech-y people's work and day-to-day, so I'll try to elaborate on why I agree and disagree with some of these points that have been repeated 50,000 times over the last ~10-15 years. disclaimer: spent most of my career in what probably would be described as "DevOps" at most places, and have led multiple devops teams.
For my tone, I apologize, but there is just so much wrong in this post that I don't want to go through it all, but here's one example of why I think this person, with all due respect, has very little idea of what they are talking about. It's irritating because it seems to devolve into some weird rant about "locked doors" which are 100% and absolutely necessary in any operation of size and scale:
> Originally, DevOps was about trusting developers with production. But modern DevOps teams operate on the belief that developers can’t be trusted with production.
I don't even know where to begin with this one. Who hurt this guy?
> And because DevOps owns the compliance checklists, they bake that mistrust into the rules.
That isn't remotely how compliance works and suggests a child-like understanding about any of this. At most they are responsible for implementing the checklists, which comes from a security team, compliance team, or some external entity, and even those checklists are not really generated or controlled by these entities, they are following established guidelines already in place that most people need to adhere to. So, red flag number one this person doesn't know what they're really talking about, but I'll digress to my main point because I think the spirit of the post is correct even though most everything in it is silly.
I have the opinion that the industry fundamentally misunderstood what DevOps was supposed to be and do. Of course this is rarely challenged, but where these arguments fall short are what are we supposed to do instead? The author comes close to recommending a solution - have them work closer to the devs and thus the product teams. Boom. Simple. That's it, and in fact, tons of places not stuck in the practices of 20 years ago have done this, and successfully! They're just not typically called "devops." Like, at meta for instance, the closest thing to that (and forgive me if I'm misremembering or this has changed, this is from a round of interviews some years ag) is a role called "systems engineer" and they work on and with the dev teams.
The existence of a team called "DevOps" to me is a massive red flag anymore that a company isn't getting it, but I have still seen this "chuck it over the wall at each other and pray" approach work fine enough. It's also created a job title that can mean practically anything and predicts very little about a person's skill set - you have "senior" devops guys and leads that basically got promoted from a Sys admin or IT support role, which for some reason companies thought was a perfectly natural progression, and then devs who did actual development for a while and now write fully functional platforms and automation for the dev teams they support. These people are not the same, which makes the hiring/interviewing process a nightmare. To make it worse, even tech people can barely understand the difference sometimes. So you'll run the gamut of having teams full of "devops" guys that can barely string together 6 functional lines of code, to full stack guys that could probably work with any of the dev teams they support as a dev.
I don't pretend to know the fix but I've increasingly gone out of my way over the last several years to avoid "devops" titles because it just seems to be a dead giveaway that the team is clueless or at best apathetic about how to do this.
> Originally, DevOps was about trusting developers with production. But modern DevOps teams operate on the belief that developers can’t be trusted with production.
This right here pretty much discredits him (or her) completely; there's no way you would survive five minutes of an audit where developers have direct access to Prod in any kind of regulated business (finance, medical, banking, nuclear, military etc). This might be acceptable for an owner operated website that sells trading cards but absolutely not for any company > 500 employees.
I'll give you a concrete example of how it goes so very wrong.
I worked for a bank for a short period of time, where the development team had gone a bit far with a shotgun approach to moving their systems to Azure. It was pretty hard to find things and as their approach evolved, the older conversions weren't revisited. Most of that team quit. New team is brought in (including me) which had some excellent engineers with years of experience in system architecture, how to make Azure work better. So we tried to homogenise things around a more reasonable approach.
This caused operational problems to worsen (understandably) but it was a short term pain, long term gain thing we couldn't be allowed time to do. So the CIO decided to take all the sysadmin/devops type work and give it exclusively to the system administrators. Who weren't developers. They fixated on one particularly narrow solution for deployment. To make it easier for themselves, but one that didn't really address the bigger picture of how to make it easy to monitor and deploy etc.
Anyway it ended up a disaster. The development team in their newly narrowed roles struggled to make their systems fit in the rigidly defined holes. Operationally it was no better and sometimes worse, but there was absolutely no compromise on how things should work or any consultation with the devs at all, ever.
I no longer work there. If you're going to do devops you have to listen to your experienced engineers, not the snotty kids who think clickops is engineering.
This is a valid story, and I have no doubt it’s real. I’ve heard and seen many stories like this that have happened.
But… I’m going to say the dirty, quiet, and unlikable thing out loud.
That had nothing to do with DevOps or its philosophies, processes, or patterns. That was bad leadership from the top down plain and simple. It’s likely not even the individual engineers faults. It’s leaderships fault for not setting clear objectives, implementing them, ensuring that the engineers had a real plan before beginning, and making sure no individual was too in charge of things.
Leadership in your case was likely career management who knew very little about technical items. Managers who were technical were probably shot down for not playing politics properly, not producing the correct “metrics” and “kpis”. So they moved on.
That’s a company culture issue that has little to do with tech.
I think the comment you are replying to is pointing out the terrible ramifications of just taking a random sysadmin team and expecting them to work in a full blown devops role. I too have seen similar situations play out, and in slight reverse - in their scenario the devops guys were calling all of the shots, ive also seen (and have lived) the opposite where devs call any and every shot on devops stuff and the devops team is treated as little more than glorified IT tech support but for devs (understandable how a business could come to a conclusion about why that makes sense, given how they perceive the sys admin -> devops progression). This sounds like it’s what you may want but it has its own sorts of problems - devs dont always know the right thing or what to ask for or what not to ask for. Really good devops guys are experts at guiding these conversations collaboratively.
I have known and worked with some really great former sys admins gone devops. I am working on mentoring one right now, but I have to be kind of insulting about it and be like “forget everything you knew before it probably won’t help now” which sucks because sys admins do form pretty decent understanding of OS’s, databases, networking, etc. however, when it comes to the code part and more importantly taking all of these concepts and applying them to reasoning about infrastructure code and complex systems is very hard for most people and you have to take a “im a total newb” mentality a lot of people dont seem easily capable of doing.
Valid assessment. The CIO in question did not have a technical background as was far more interested in fostering cliques than good engineering practices. The culture of the place was toxic as a result, it rewarded surveillance and not performance. It promoted fantasy land project managers who ignored reality.
Still, it made me very wary of the idea that devops is separate to development.
Managing relationships with project management and the difficulty that entails with most devops orgs where ive been is a symptom of the same problem I feel is being circled around. Good devops managers manage around these realities really well, can think of some specific examples in my head - it’s very much a several way conversation that should punish silos. however in practice it seems to reward it. If devops worked with and closely with devs you do not need layers of semi competent project managers in between acting like glorified note takers.
I worked in finance on the other side of the pond - developers wanted to constantly bring in and use new services but also didn't want any of the responsibility or the work needed to make compliance happy (or even in that particular company shoulder the costs). When me and other folks where brought in it to fix the "cloud strategy" it was a complete shitshow and heads actually rolled when we wrote a tool to assign costs to applications. But we had to start almost from scratch and limit usable services as we developed strategies and blueprints for each...
The complete, unapologetic desire of devs and security teams (but also many infra teams) to not have any kind of ownership was horrifying to me.
In the end there's not a single solution or strategy, it really goes back to the organization and where your weaknesses and strength are as an org. If you have a gazillion consultants following the "best practice" of the day and exceptions on top of exceptions you are dead, devops or otherwise. You will still make billions if you are the right company though regardless of your software practices, so...
Most of the DevOps folks I’ve encountered, unfortunately, couldn’t code much. So it was mostly ops work with a fancy new title.
The idea of a single entity being responsible for development, operations, observability, and support is flawed from the start. That’s not a one-person job, and the expectation simply doesn’t scale. So DevOps often ends up being either ops folks or dev folks, and rarely a true blend of the two.
What we need are feature-focused developers, ops-savvy devs who can deploy their own work, and a strong team dedicated to observability and applying modern SRE practices.
So I think curious developers who aren’t afraid of infra, along with a solid platform engineering team, are a real improvement over the status quo.
> Most of the DevOps folks I’ve encountered, unfortunately, couldn’t code much. So it was mostly ops work with a fancy new title.
The expectations were (and are) irrealistic.
Looking for somebody that has both the software engineering skills of a software engineer and the skills of a sysadmin, willing to do two jobs for the salary of one: yeah sure, keep looking for it.
That being said, i'm one of those that jumped on the bandwagon because in practice it meant fancier job title, higher pay and getting to use newer (often better) tooling.
Ten years ago being a devops engineer rather than a sysadmin usually meant getting to work with EC2 and cloud stuff rather than administering remote physical servers and fighting with NetApp SANs.
Devops was meant to be really a methodology change, not really a job title.
In my experience whether DevOps works or not really depends on the management. I can setup all the automation you want, but management must back me when I tell you're supposed to use the automations rather than involve me into doing your work.
I can surely take feedback and improve whatever you need to be improved, but you *must* be using the automation.
Otherwise we go back and Development and Operations.
I've seen the world begore devops, and it wasn't that pretty either. We had 2 silos. Devs had a bunch of apps, ops too. Devs had no idea what the hardware did, ops had no idea what the software did. Users suffered, and assigning responsibility was impossible.
The dev in me hated throwing an app over the wall, only to hear about ops completely misconfiguring it. Or ops ignoring deployment guidelines and mucking up releases. Or getting a db with 2 records and being expected to test performance. Or getting complaints about prod crashing, but no logs, eror messages, or any kind of access, and ops couldn't notice that disk full might be a disk being full.
The ops in me hated devs being loose with passwords, asking huge mountains of resources, and getting apps thrown over the wall at me without any idea what to expect. Or devs logging sensitive stuff.
For me, devops was good for breaking up the silos. We wanted to have a group of people cooperating on a group of applications. Some people will have a dev or ops profile, and that's fine. People should respect that others know more of some aspects. But: no one should shield the applications from everybody else in such team. Everybody should strive to learn enough basics to understand both perspectives and priorities.
Starting a devops silo is just the worst possible option.
> We left product development teams without anyone focused on production. We undid everything that made DevOps work in the first place.
Very good point, that's the same I have observed for the past couple of years when working on a devops team. Product team engineers nowadays feels like spoiled kids, they had no current how server runs, and asked for things unreasonable. I still remembered someone came to my desk and asked for me to increase the mem request to 10s of GB, he claimed that's the best solution he could think of is to load everything in mem.. and very often people don't even know what status code means 500, 502, 503, 504...
I think that Big DevOps was the best thing that could have happened for systems-focused devs and SWE-like sysadmins. Their ability to straddle both domains was suddenly in demand and made them worth heaps more in the job market, even more so after gaining experience with Kubernetes and LLMs.
Good food for thought but the article assumes how currently DevOps and Dev teams (and there's also an Ops team?) operate and there's a lot of different ways in how "DevOps" is done at different organizations.
Take any combination of dev, platform engineering, devops, SRE, ops teams (existing or not) and combine with different responsibilities and how they work together and there's a company out there using that model.
Heck, we don't even have a common understanding of what a "DevOps" team does https://docs.sadservers.com/blog/what-the-f-is-devops/
DevOps is a good idea. But most have no clue about the Ops part
All the devops I have encountered so far have been “sysadmin in a trench coat.” Most couldn’t write anything beyond basic Python and Shell scripts. It was mostly ops folks jumping into a fancy new title. So every time something would go poof, they had to pull in devs to debug the most basic stuff.
Sure you have a few people that are great at development and operations and understand (and can make complex changes) to both the infra and the codebase.
But those people are making $300k+ at FAANG companies.
Unfortunately they are also the ones who write all the blog posts and podcasts on devops
For years idiots pushed DevOps and I told them it was a terrible idea, so they did not hire me. I wish I could get back pay
This article is really light on concrete examples and any kind of point outside of the title, which is a shame, because I think this is an extremely relevant discussion to most tech-y people's work and day-to-day, so I'll try to elaborate on why I agree and disagree with some of these points that have been repeated 50,000 times over the last ~10-15 years. disclaimer: spent most of my career in what probably would be described as "DevOps" at most places, and have led multiple devops teams.
For my tone, I apologize, but there is just so much wrong in this post that I don't want to go through it all, but here's one example of why I think this person, with all due respect, has very little idea of what they are talking about. It's irritating because it seems to devolve into some weird rant about "locked doors" which are 100% and absolutely necessary in any operation of size and scale:
> Originally, DevOps was about trusting developers with production. But modern DevOps teams operate on the belief that developers can’t be trusted with production.
I don't even know where to begin with this one. Who hurt this guy?
> And because DevOps owns the compliance checklists, they bake that mistrust into the rules.
That isn't remotely how compliance works and suggests a child-like understanding about any of this. At most they are responsible for implementing the checklists, which comes from a security team, compliance team, or some external entity, and even those checklists are not really generated or controlled by these entities, they are following established guidelines already in place that most people need to adhere to. So, red flag number one this person doesn't know what they're really talking about, but I'll digress to my main point because I think the spirit of the post is correct even though most everything in it is silly.
I have the opinion that the industry fundamentally misunderstood what DevOps was supposed to be and do. Of course this is rarely challenged, but where these arguments fall short are what are we supposed to do instead? The author comes close to recommending a solution - have them work closer to the devs and thus the product teams. Boom. Simple. That's it, and in fact, tons of places not stuck in the practices of 20 years ago have done this, and successfully! They're just not typically called "devops." Like, at meta for instance, the closest thing to that (and forgive me if I'm misremembering or this has changed, this is from a round of interviews some years ag) is a role called "systems engineer" and they work on and with the dev teams.
The existence of a team called "DevOps" to me is a massive red flag anymore that a company isn't getting it, but I have still seen this "chuck it over the wall at each other and pray" approach work fine enough. It's also created a job title that can mean practically anything and predicts very little about a person's skill set - you have "senior" devops guys and leads that basically got promoted from a Sys admin or IT support role, which for some reason companies thought was a perfectly natural progression, and then devs who did actual development for a while and now write fully functional platforms and automation for the dev teams they support. These people are not the same, which makes the hiring/interviewing process a nightmare. To make it worse, even tech people can barely understand the difference sometimes. So you'll run the gamut of having teams full of "devops" guys that can barely string together 6 functional lines of code, to full stack guys that could probably work with any of the dev teams they support as a dev.
I don't pretend to know the fix but I've increasingly gone out of my way over the last several years to avoid "devops" titles because it just seems to be a dead giveaway that the team is clueless or at best apathetic about how to do this.
> Originally, DevOps was about trusting developers with production. But modern DevOps teams operate on the belief that developers can’t be trusted with production.
This right here pretty much discredits him (or her) completely; there's no way you would survive five minutes of an audit where developers have direct access to Prod in any kind of regulated business (finance, medical, banking, nuclear, military etc). This might be acceptable for an owner operated website that sells trading cards but absolutely not for any company > 500 employees.
I'll give you a concrete example of how it goes so very wrong.
I worked for a bank for a short period of time, where the development team had gone a bit far with a shotgun approach to moving their systems to Azure. It was pretty hard to find things and as their approach evolved, the older conversions weren't revisited. Most of that team quit. New team is brought in (including me) which had some excellent engineers with years of experience in system architecture, how to make Azure work better. So we tried to homogenise things around a more reasonable approach.
This caused operational problems to worsen (understandably) but it was a short term pain, long term gain thing we couldn't be allowed time to do. So the CIO decided to take all the sysadmin/devops type work and give it exclusively to the system administrators. Who weren't developers. They fixated on one particularly narrow solution for deployment. To make it easier for themselves, but one that didn't really address the bigger picture of how to make it easy to monitor and deploy etc.
Anyway it ended up a disaster. The development team in their newly narrowed roles struggled to make their systems fit in the rigidly defined holes. Operationally it was no better and sometimes worse, but there was absolutely no compromise on how things should work or any consultation with the devs at all, ever.
I no longer work there. If you're going to do devops you have to listen to your experienced engineers, not the snotty kids who think clickops is engineering.
This is a valid story, and I have no doubt it’s real. I’ve heard and seen many stories like this that have happened.
But… I’m going to say the dirty, quiet, and unlikable thing out loud.
That had nothing to do with DevOps or its philosophies, processes, or patterns. That was bad leadership from the top down plain and simple. It’s likely not even the individual engineers faults. It’s leaderships fault for not setting clear objectives, implementing them, ensuring that the engineers had a real plan before beginning, and making sure no individual was too in charge of things.
Leadership in your case was likely career management who knew very little about technical items. Managers who were technical were probably shot down for not playing politics properly, not producing the correct “metrics” and “kpis”. So they moved on.
That’s a company culture issue that has little to do with tech.
I think the comment you are replying to is pointing out the terrible ramifications of just taking a random sysadmin team and expecting them to work in a full blown devops role. I too have seen similar situations play out, and in slight reverse - in their scenario the devops guys were calling all of the shots, ive also seen (and have lived) the opposite where devs call any and every shot on devops stuff and the devops team is treated as little more than glorified IT tech support but for devs (understandable how a business could come to a conclusion about why that makes sense, given how they perceive the sys admin -> devops progression). This sounds like it’s what you may want but it has its own sorts of problems - devs dont always know the right thing or what to ask for or what not to ask for. Really good devops guys are experts at guiding these conversations collaboratively.
I have known and worked with some really great former sys admins gone devops. I am working on mentoring one right now, but I have to be kind of insulting about it and be like “forget everything you knew before it probably won’t help now” which sucks because sys admins do form pretty decent understanding of OS’s, databases, networking, etc. however, when it comes to the code part and more importantly taking all of these concepts and applying them to reasoning about infrastructure code and complex systems is very hard for most people and you have to take a “im a total newb” mentality a lot of people dont seem easily capable of doing.
Valid assessment. The CIO in question did not have a technical background as was far more interested in fostering cliques than good engineering practices. The culture of the place was toxic as a result, it rewarded surveillance and not performance. It promoted fantasy land project managers who ignored reality.
Still, it made me very wary of the idea that devops is separate to development.
Managing relationships with project management and the difficulty that entails with most devops orgs where ive been is a symptom of the same problem I feel is being circled around. Good devops managers manage around these realities really well, can think of some specific examples in my head - it’s very much a several way conversation that should punish silos. however in practice it seems to reward it. If devops worked with and closely with devs you do not need layers of semi competent project managers in between acting like glorified note takers.
I worked in finance on the other side of the pond - developers wanted to constantly bring in and use new services but also didn't want any of the responsibility or the work needed to make compliance happy (or even in that particular company shoulder the costs). When me and other folks where brought in it to fix the "cloud strategy" it was a complete shitshow and heads actually rolled when we wrote a tool to assign costs to applications. But we had to start almost from scratch and limit usable services as we developed strategies and blueprints for each...
The complete, unapologetic desire of devs and security teams (but also many infra teams) to not have any kind of ownership was horrifying to me.
In the end there's not a single solution or strategy, it really goes back to the organization and where your weaknesses and strength are as an org. If you have a gazillion consultants following the "best practice" of the day and exceptions on top of exceptions you are dead, devops or otherwise. You will still make billions if you are the right company though regardless of your software practices, so...