I was pro-systemd at the time of the controversial discussions. I still think it's a net positive relative to what was there before. But personally speaking, it's only the core of the software (service management) that improved things for me. The other things (timers, journald) I either ignore or don't like, but perhaps they're useful for distributions.
I've been quite happy with -boot, -networkd, -resolved. -logind too, what a powerful upshift in routing input.
I forget what but timesync is missing a lot of nice to haves. But ditching grub for something much much much more manageable & a joy to use. Networkd and resolved have both had a lot of nice clear straightforward options, and I love how systemd unit files snap together for these networking concerns.
Journald has been ok for me, and I use it a lot, but not intensively. Having adequate performance is definitely a huge concern.
I run, at home and at work, vpns and having a good way to configure split-horizon dns is incredibly useful.
At a previous company we were able to ship a binary that would automatically configure a bunch of company settings on developers' linux laptops and got VPN configurations working with split-horizon dns very easily.
prior to this, doing split-horizon dns on linux basically meant running dnsmasq (!!!) on every linux laptop
Yes, systemd's weirdly bundled system services largely suck. The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs) and resolvd has or used to have basic bugs that other systems don't have.
> The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs)
That sounds like an unequal comparison. For the same amount of log data and the exact same filter operation, journald should be strictly faster or equal to text logs. Are you sure this isn't because in your journald test you're going through a lot more logs? Because journald makes it easier/feasible to collect more logs.
No, journald shouldn't be strictly faster. It signs/encrypts/whatever the logs and stores them in some kind of structured format that needs to be converted to plaintext before processing it as such. That is not faster than just reading the plaintext. Grep and less search etc are very fast.
And no, I don't want to use the structured features. For error diagnosis, I want every occurrence of the search string in every field over a long time for comparison purposes.
No, it stores them with binary metadata, rather than ASCII metadata. "Plaintext" needs to be parsed, which is expensive. You don't really think binary RPC formats like protobuf are slower than JSON because it's "structured format" compared to "plaintext" I hope. Or SQL databases are slower than CSV files.
> And no, I don't want to use the structured features.
I don't see why you'd prefer having to figure out different time formats to parse from different software logs compared to a standard time format.
I'm searching for "service name" and I don't care if it comes from "service" or mentions "service". That's unstructured.
The rest of your comment doesn't even apply. Search is not parsing and not structured data transfer (which is what binary RPC formats are for), and unless you have a fulltext index, there is no faster way to do it than the usual algorithms, which work at 100+ MB/s. With log files, the text is right there ready to be searched. With journald, it needs to be assembled first, and it's pretty slow in practice.
I've worked as a sysadmin/devops for a double digit number of years.
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations.
Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse.
Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons.
I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
The grand-parent post was about the general consensus: sysadmins generally like systemd. So I'm not sure answering individual concerns is useful, but I'll take the bait...
> now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
No, you don't have to, because a syslog daemon is not required any more. BTW, rsyslogd is not the only syslog service, I've had to work with alternatives like rsyslog-ng.
And the daily usage is now much simpler: `journalctl -b -1 -p err` or `journalctl -u mariadb --since "24 hour ago"` were painful queries with (rotated and compressed) syslog files.
> Boot and shutdown has previously been deterministic, now things just randomly hang.
That was not my experience. IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
> Network device naming is also a fun one.
The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel. I think it's still possible to configure the kernel, udev and systemd to work with the old naming, but if it's just because "shorter is better", I don't think it's worth it.
> No, you don't have to, because a syslog daemon is not required any more.
That is "not required" for the single-computer-sysadmin. If all you know is your laptop, then journald might be fine. If your computers are cattle and have numbers, you don't want to read your logs on each individual computer, if possible you don't want to read logs at all, you want some automation to do it for you, aggregate it, warn you about things, etc. journald just makes that harder, and going through some syslog protocol implementation (of which there are a few, you are right there) is the only relevant game in town.
> IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
Dependencies and insserv were a later addition. Basically you just defined a boot order by numbering your /etc/rc[0-6].d/S* links in ascending order, either manually or in a dependency-based fashion through insserv. This means that you can have a suboptimal ordering if those dependencies are wrong or you choose your ascending order poorly. But it also means that once you have a working boot order, it can be repeated, and it will be repeated. That is what I mean by determinism. Once you have a working boot, it will keep working. Solaris SMF did something more modern, you would define dependencies in service definitions, and it would compute a static deterministic ordering from that, without those /etc/rc[0-6].d/S* symlinks. Deterministic, but dependency-based. If it has worked once, it will work again. And they even had a rollback: if after a change, the next boot wouldn't work, you could go back to the previous boot order and fix things from there. And neither had race conditions, because you usually wouldn't do things in parallel, one service would wait for the previous one in the boot order (parallel was possible but rarely used, because of the inherent fragility and possible race conditions).
Systemd does it dynamically on every boot, no precomputation, just looking at all the files and starting everything that could possibly be started at the same time, in parallel. The reason for this is that this makes Lennarts laptop boot faster, which is true in the ideal case. But it is neither deterministic, nor debuggable. And other than the previous methods, it can lead to race conditions.
Note that there is one more race condition at boot, which is races against kernel devices and systemd-udevd. That neither got worse nor better with systemd, it just lead to the kernel people giving up and handling more things like firmware loading in kernel.
> The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel.
The Linux kernel does not have a net.ifnames parameter, and the kernel interface naming scheme is still eth[0-9]+. When you stick that parameter in the kernel commandline, the kernel ignores it and only systemd and systemd-udevd interpret it later on.
Please, if you want to sing the praises of software or deflect blame from broken crap, make sure that you know what you are talking about...
It's a little weird to reply to someone complaining about administrative problems caused by systemd with "generally sysadmins seem to like it."
That kind of response is exactly why everyone gets emotional in these discussions. Technical problems are one thing but when you gaslight people experiencing and reporting legitimate problems you should not be surprised when they get mad and rip your solution out.
GP is being far kinder to the systemd devs than they deserve, I gave up on the project years ago for exactly this reason.
> consistent-net.rules nailed it to a mac address on first boot
fails even harder for
> put in a live-CD to debug something and your network devices are suddenly weird
I don't know that there is really any good answer here. Just methods that break in ways you're used to, and methods that break in ways you aren't. I appreciate the attempt at static naming based on hardware connection, even if the realities of how these things are presented by the bios make the results an inscrutable mess.
Maybe pure enx would have been better? But that looks like crap for even the simple cases, and fails if you swap out a piece of hardware or start doing weird things with MAC addresses.
The only consistent naming scheme I've ever seen working was Solaris on Sun hardware. But that only worked because they controlled both the hardware and the software side of things.
And yes, consistent-net.rules doesn't solve LiveCDs and reinstallations. But as I've said, systemd doesn't solve those either, and in most cases actively makes them worse.
My experience has not been that the systemd device names are "actively worse". The biggest problem I've had is changing around VMs where subtle shifts in PCIe IDs (specifically subdevices) end up changing changing names. Meanwhile, consistent-net.rules seems like a hack, always in the back of your mind as an arbitrary stateful part of the system. Why should I think of my primary network device as eth2 if there is no reason for it to be eth2 other than merely being the order on the first install?
But if you prefer consistent-net.rules, then more power to you. I'm guessing most of your frustrations could have been addressed by systemd having some shame about changing functionality out from under users, and providing an up-front option to switch between the different methods.
I used to be fine with it, until I started writing my own software tools, then looking at how to actually do stuff with systemd and I'm like, nope nope nope nope. At some point when I have space for a new project I'm going to migrate from Arch to Obarun, which uses a modern update on the excellent daemontools to provide init. When systemd works it's mostly fine, when it breaks, path of least resistance is often to reinstall the OS. Which is fine if you're just working on it. Painful if it's your personal machine, and those are what's most likely to break systemd.
> when it breaks, path of least resistance is often to reinstall the OS
Can you elaborate how do you even break this?
Adding a systemd unit cannot really break much on systemd itself. Unless you’re messing with internal systemd stuff, I am not sure what is even there to break.
At the moment, on this machine I'm using, audio just doesn't work. Dunno why. Tried a few different things to fix it, no dice. Trying to troubleshoot is made enormously more difficult by the complexity of systemd. If I had a simpler system I could run it all the way back to the source, but really, I can't be bothered, I don't really need audio on this machine atm.
i have the same sentiment, and for me systemd was getting so slow and hard to tshoot (the tools to analyze boot graphs and sequence showed service sequence breakage, some times there is no problem, sometimes sequence changed and problem happened) and i just changed for another distro without systemd and boom, startup in seconds no problem at all.
This doesn't match my experience at all. For the life of me, I cannot imagine what you are talking about, what any of your issues are. Did you file any tickets, or report any issues?
What about "actually doing stuff with systems" was "nope nope nope nope"? What situations did systemd break in? What did you try before giving up to reinstall the OS?
You probably use a desktop environment like GNOME or KDE. Maybe do some light coding in an IDE. I do not. I run window manager only, i3, with a really stripped down toolset. I like the idea of knowing my system in and out. My current project is writing my own shell. It can switch between command mode and ruby REPL with ctrl-z.
You mention filing tickets / reporting issues, sorry, that means contributing to a project. I'm not wholly opposed to doing that, but I have to like the project first. And there's nothing to like about systemd.
You go do that if you like it so much. My guess is you simply won't, and will just keep using other people's software without a care in the world, thinking this kind of online warriorship is actually helping anything.
Bro, I've been running wmii since before i3 forked off from it. Don't come at me with this dripping disdainful attitude, pretending like you are some high and mighty elite system whisperer.
It's so unfortunate that there is such a widespread persistence of hater-dom. It's so toxic and awful, and so rarely does it have even the faintest shred of anything to stand up on. I struggle so much to see these attractors into the negative as so compelling, so ensnaring of the human souls as they have been lately.
Systemd is amazing, provides such a rich world versus piecemeal poorly managed systems of yore. Yes you could understand init scripts with very little effort but what a sad virtue, to dare to go no further than what an afternoon of exploration might get you.
I don't fucking hate systemd. I'm marginally glad it exists. I just don't want to live under it's despotic regime. You might like it because you think trains running on time is more important than freedom.
You're the one coming at me with fucking attitude, like a 16 year old who just discovered IntelliJ. I have done the exploration, then looked online for why everything is so damn complicated now. It's because systemd colonized the world.
It's not just an init system, it's everything below the userland. And a lot of those components are very poorly designed. And it's this big ball of mush where if you use one component, thy must use them all. It's so bad Arch does not support replacing systemd, like at all. You're totally on your own.
Go ahead, try replacing systemd components in an Arch system. I'll wait. And lose the fucking attitude.
> It's not just an init system, it's everything below the userland. And a lot of those components are very poorly designed. And it's this big ball of mush where if you use one component, thy must use them all. It's so bad Arch does not support replacing systemd, like at all. You're totally on your own.
It's true, it's not only an init system anymore, it's also a service manager, a network manager, a DNS cache and resolver, a proper logging system (as in, with metadata, all in one place) and more.
Despite being opposed to it in the beginning, probably because it was immature at the time, I grew to love it even on Gentoo - my desktop and server OS of choice - which gives you OpenRC as a fully supported alternative:
- on my desktop, laptop and work VMs, it just works, with varying adoption of its components and services started, lazily activated if unneeded
- on my server, it simply makes sense for service auto-restart: OpenRC still requires you to manually enable support for service auto-restart ( https://wiki.gentoo.org/wiki/OpenRC/supervise-daemon#General... ), a feature that comes built-in in systemd and without extra complexity or components, and with predictable outcomes given the unit file
- almost all modern DEs rely on systemd, because it generally improved their session handling, sandboxing, brought user (rootless) services and other features
And generally, I think the biggest win is creating a system service management platform that can be used by all distros in the same way, without having to know 5+ init systems, having different scripts for the same service for each distro. I can basically work on my own devices, on work devices, on cloud VMs, on IoT devices, and have the same behavior, across different distros and hardware.
It's an improvement, I'll give you that much. But it's moved in the opposite direction and seeks to reduce freedom, not give more of it. I should be able to run my own DNS cache, network manager, service manager if I want to. Systemd makes that all but impossible.
I semi-respectfully disagree that I've given you undue attitude.
One of the things I really enjoyed about the Flix programming language submission today was running across their principles section. Right off the top of the list:
> Simple is not easy
> We believe in Rich Hickey's creed: simple is not easy. We prefer a language that gets things right to one that makes things easy. Such a language might take longer to learn in the short run, but its simplicity pays off in the long run.
In my view, yes, there are warts to systemd, but it is an operable understandable system which has a hard fought simplicity to it. I can go to any systemd computer and get a good idea of what's running, understand how it's wired. I use common practices and configs to change or drop in config.
My feeling is that a lot of people want easy. That systemd doesn't pass the easy test.
The past? The past was enormously complicated. It was complicated because every shop has its own way of configuring things. Every shop has their own tools to assess system state, their own way to configure subsystems like logins, dns, networking, logging, time sync, booting. There were countless countless easy answers, a path littered with the bodies of dead easy answers to this or that concern. And you needed an easy answer for each concern; easy didn't go hard to integrate the different factors.
That past was a horrible mess of easy. Systemd feels complex, but compared to the past, there is a persistent understandable simplicity that is a raw joy to use. Things make sense. When you see one thing, you sort of learn how the next works. The back to the woods rejectionism that springs up today is a bizarre weird thing; folks have not seen how wildly different and bedazzling system management used to be, within and across orgs. You might not appreciate why but there's very very good reasons, and I'm fact, the world is inherently complex, there's aot to do, and systemd has created a wonderful pervasive cross-sectional (via it's many opt-in components) "easy" that harnesses and enables the complexity that is, in a sensible glorious way. ("This has made a lot of people very angry and been widely regarded as a bad move.")
Systemd is anything but simple. And the past definitely wasn't easy. You seem to have the terms backward. For the end user, yes, it's a polished experience that you mostly don't have to mess with, and the parts of it you're most likely to interact with are, well, fine. Not great, but fine.
> operable understandable system
Every system is understandable, it's just a question of how much work you have to put into understanding it. Good design reduces the amount of this effort needed. Many many systemd components simply aren't well-designed. But they all interop with each other so there's no way to slowly wean a system off of systemd. Trust me I tried.
I don't want to understand systemd. It feels like being at work. Good on you for putting in the effort. I should be able to use software I like and want to use. Maybe even hand-write some replacements, like I'm doing with my shell. Systemd doesn't get in my way there, thankfully, it's been awhile so I don't have specifics off the top of my head for the last time I tried removing a few systemd components only to find hard dependencies on each other.
This is not how Linux needs to be. I agree the old ways sucked. I don't want to go back to them either. But neither do I want to be locked into the big ugly mess that systemd turned into. I want clean, modern, well-designed components that are independent of each other.
I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext.
Somehow journald manages to provide a database which is 40x slower to query than running grep on a compressed text file.
I'm all in favour of storing logs in an indexed structured format but journald ain't it.
Journald is an odd one. I don't think it being a binary log/database makes sense. If you have a tiny operation, with a single server, then the binary database doesn't really make sense, having plain text is just easier and faster. If you're a bigger operation, you'll have a central logging solution, in which case you need journald to store the longs as plain text as well, before you can do log shipping.
The only use case where the binary format might make sense is if you ship journald logs to another central journald instance. That's just very much an edge case.
Doesn't that still involve a conversion? I believe that rsyslog can read the journald database, but you're typically not querying syslog data directly, so there's a conversion between rsyslog and logstash, Splunk, Datalog, whatever.
- a very opaque structure: it's actually not that opaque, since configuration files are still plain text, and better yet with more structure, but it first appears as inscrutable, since the programs aren't shell scripts;
- a set of new tools to learn: Systemd doesn't make use of the existing Unix tool-set (sed, awk, etc), ie. the vocabulary most sysadmins are familiar with in this ecosystem.
It seems on the outset that Systemd is trying to get away from the traditional Unix "one program should do one and only one thing well" (which is colloquial phrasing for separation of concerns).
Still, one idea that occurred to me is that in the early days of Unix, system programs might have been quite simple, with very few options (that man pages were probably very short), but that's clearly not the case nowadays. So haven't we left the "one program should do one and only one thing well" paradigm a long time ago anyway?
Actually, I think that one crucial issue with systemd is its specialization: can it be used for anything else than service management? At first, this seems to clash with the Unix principle, but even awk and sed are meant to do one thing, yet their field of action is very general, not just confined to handle one style of files or perform one style of transformation.
Same, that was my first experience as well. It was a breath of fresh air in many ways, and after my initial learning of the new syntax, I quickly became a convert.
That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
I think the biggest problem with systemd is that they named the main init service manager systemd and also named the organization systemd! It'd be like if GNU named their initrd "GNU". Their other tools just have systemd prepended, e.g. systemd-journald, much like GNU Make or GNU libc (glibc) or similar. The "scope creep" issue doesn't get complained about for GNU because it's clear they're an organization making a bunch of tools for different purposes. Systemd-the-organization is the same, but the naming makes it unclear that many of these tools are not depended on by systemd-the-init-system.
Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example).
You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
I use journald whenever I feel my blood pressure getting too low.
It's slow, truncates lines, and doesn't work well at all with less. It's almost like Pottering created it so that pulseaudio wouldn't be his worst program anymore.
And a lot of software don't use syslog because it's easier to print to stderr/stdout or some random log file. Journald makes it easier to capture everything no matter what the software does, including the established syslog protocol, so I don't even see your point.
If everything on your machine uses syslog, journald is a drop in replacement for the dozen of possible syslogd implementations.
journald isn't drop-in because it only saves logs locally. syslog also is a protocol to send your logs to a log-server.
And the usual syslog APIs are 2 lines: Initialize with openlog(stream, process_name, flags) and after that do syslog(urgency, message). That is on par with stderr/stdout, and far simpler than handling your own logfiles. Except if you use log4$yourlanguage or something, then everything is just the same, you just configure a different destination.
And if you can't change your code to not use stdout/stderr, you can easily do yourcode | logger -t yourcode -p daemon.info 2| logger -t yourcode -p daemon.err
I architected and built our entire log-ingestion pipeline for intrusion detection on it, at Square.
I built a small Ruby wrapper around the C API. Then I used that to slurp all the logs, periodically writing the current log ID to disk. Those logs went out onto a pubsub queue, where they were ingested into both BigQuery for long-term storage / querying, and into our alerting pipeline for real-time detection.
Thanks to journald, all the logs were structured and we were able to keep a bunch of trusted metadata like timestamp, PID, UID, the binary responsible, etc. (basically anything with an underscore prefix) separate from the log message all the way to BigQuery. No parsing, and you get free isolation of the trusted bits of metadata never intermingling with user-controlled attributes.
Compared to trying to durably tail a bunch of syslog files, or having a few SPOF syslog servers that everyone forwarded to, or implementing syslog plugins, this was basically the Promised Land for us. I think we went from idea to execution in maybe a month or two (I say “we” but really this was “me”) and rolled it out as a local daemon to the entire fleet of thousands. It has received—I think—one patch release in its six+ year lifetime, and still sits there quietly collecting everything to be shipped off-host.
The only issue we’ve ever really ran into that I never figured out is a handful of times per year (across a fleet of thousands) the journald database corrupted and you couldn’t resume collecting from the saved message ID. But we were also on an absolutely ancient version of RHEL, and I suspect anything newer probably fixed that bug. We just caught the error and restarted from an earlier timestamp. We built the whole thing around at-least-once delivery so having duplicates enter the pipeline didn’t really matter.
Damn, honestly at this point I’m wishing I’d pushed to open source it.
Ironically, actually, I did write a syslog server that also forwarded into this pipeline since we had network hardware we couldn’t install custom services onto but you could point them at syslog. I also wrote this in Ruby, using the new (at the time) Fibers (“real” concurrency) feature. The main thread fired up four background threads for listening (UDP, UDP/DTLS, TCP, TCP/TLS), and each of those would hand off clients to a dedicated per-connection worker thread for message parsing. Once parsed they went onto one more background thread for collecting and sending to PubSub. Even in Ruby it could handle gazillions of messages without breaking a sweat. Fun times.
Since I’m rambling, we also made cool use of zstd’s pre-trained dictionary feature. Log messages are small and very uniform so they were perfect for it. By pre-sharing a dictionary optimized for our specific data with both ends of the pubsub queue, we got something like 90%–95% compression rates. Given the many terabytes of logs we were schlepping from our datacenters to GCP, this was a pretty nice bit of savings.
Well, everyone and noone uses journald. The usual way to log is journald -> (r)syslogd -> remote log destination(s). I've never actually seen any other way.
The preferred journald way of "fetch your logs periodically" wouldn't pass any audit and doesn't work with any kind of log processing software that people use.
You're being downvoted but you're absolutely right. The fact that Cloudwatch doesn't support journald is a major, major fail on AWS' part. It's not like this is new or obscure software.
I wouldn’t call it a success it all. Almost managed to do was moving or forcing other software to get locked in. GDM, Podman, Pipewire…just to name a few.
> The unix philosophy cries out: is this the end of Linux (or, as many are calling it, GNU plus Linux)?
I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence:
I've been a fan of systemd even if just for how much simpler creating services is. I just need to write a simple config file instead of a complex init script.
+1 - there was a misconception that initscripts were easy or simple - if you wanted _correct, bug free_ initscript you were probably looking at around 100 lines of shell script.
Or if the service didn't support pam_limits because it was legacy trash, you had to hack something into the initscript like `ulimit -n XYZ` and restart it. Now things like this are trivial and easy to solve. Using systemd makes large scale Linux systems administration much easier.
Now it has gone a bit overboard. Some of the stuff like the dns resolver or the nspawn capability seem a bit over the top, but overall, it has massively improved all Linux distributions it is used in.
Never again will I worry about trash buggy init scripts not actually stopping a service due to a stale pid file. Now it puts the service into a control group and can kill all things in the control group even if the service is bad code.
systemd has been a success for 2 groups, Windows admins and Fortune 500 Companies.
For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though.
For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.
No, it's terrible. I think what they mean by "success" here is that it functions, and is easier to use and more sturdy than init scripts. But any number of things would have been better than those. Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now.
Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.
> Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.
OpenRC is great, there's just no flashy marketing behind it or team of consultants pushing it.
EDIT:
Heh. At least I can debug a shell script, I can't debug a config file.
I think this is a fundamental disagreement many people have in what's ultimately desirable when designing software. I personally can't stand non-trivial config files and both at work and at home have wasted way too much time pulling apart software consuming them.
The problem with systemd is that, because of all the new userspace APIs it adds, we can't just agree to disagree. You either target a Systemd OS and the users get to suffer non-trivial, often broken, configs or you target a normal GNU/Linux OS and the systemd users suffer all kinds of surprises when systemd rug pulls them.
# Export some ENVs for it
export SERVER_ADDRESS="0.0.0.0:8088"
name="my-program"
description="My program"
command="/usr/local/bin/my-program"
# If it does not know how to background iself
command_background=true
pidfile="/run/${RC_SVCNAME}.pid"
# To have logs
output_log="/var/log/${RC_SVCNAME}.log"
error_log="/var/log/${RC_SVCNAME}.err"
It seems that it still uses Turing-complete shell scripts as configuration. Hard pass.
>Who exactly was going to write the systemd alternative?
And that quote proves 100% what is wrong with Linux these days.
In the 90s, if someone posted in USENET they want init to do X, at least 2 different solutions would appear with in a week, probably many more.
That happened to me when I went from Coherent OS to Slackware in the early 90s. I asked if there was a way to have virtual consoles on my VGA monitor and my B&W Hercules monitor (386sx), or to have at least 1 VC on the Hercules. Coherent 486 supported that, Linux did not. A post appeared in a couple of days with a patch.
Now, between the Linux Foundation and the Companies that owns it, that would never happen. People who have gotten into Linux in the last 20 years cannot imagine how responsive the developers were to user requests back then.
Seems these days we are dealing with some corporate behemoth drowning in red tape.
Hi, author here. The title is intentionally a little attention-grabbing hyperbole but I've appreciated the discussion and feedback about the post. Thanks for reading!
This is a nit-pick, but if you picked the title that way to BE hyperbolic, I am not sure if that is the best thing to do for a technical discussion. It sets an internal opinion before even clicking on it. It subconsciously makes people go into the article with an already preconceived notion.
You can have S66 with AntiX too. Just use the 'Init Diversity Edition' (Respin), and pick it from several other options. AFAIK the author of S66 directly collaborated with Antix, to make it perfect.
I personally love the init/service/unit-file portion of systemd and have few complaints. It has a lot of powerful features for security, cron, etc that are very simple to use. I also really like journald.
The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.
And those ancillary managers are usually half-baked or outright broken and dangerous, in that they only implement half of the stuff they are trying to replace. E.g. https://github.com/systemd/systemd/issues/25676
Hard disagree. Name resolution feels like a solved problem, so why fix it if it isn't broken?
But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better.
I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved.
Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
The older init systems were incredibly frail and often did not work. The reason is because Bash, as much as we all love it, is actually super mega ass. It's very bug-prone and actually one of the least portable languages ever invented, in practice. Because you can't, or shouldn't, call to any other binary from within Bash, otherwise you risk your portability. Now your script won't work the same on other platforms, or even the same platform with slightly different configuration.
But um... yeah... calling to other binaries is the entire ethos of Bash. It's a shell, that's supposed to be all it does. Even getting a "true" value should be a call to another binary.
Unit files, despite being a bespoke format for one singular program, are actually more portable. Like, WAY more portable.
I think even for new people the systemd configs are much more complex (and the APIs are certainly much more complex) than the libc ones which worked well (often better.)
The problem is that almost nothing in Systemd shares theory with the rest of the OS. It's entirely its own island of new ideas, commands, and configuration language that you have to learn and memorize.
> i used to think that systemd was made the default and adopted by most distros because of its ease of use and the fact it supplied a whole bunch of things in one suite and i see where the appeal is in that but after switching to artix openrc, im just lost on why they decided to use systemd when openrc is objectively better when it comes to being an init system and for managing services, and all the other components of systemd suite can just be replaced, like why would they do this?
I was following where the author was coming from until they quoted this, and rather than addressing it they just slammed it, slammed the author, and moved on.
It's weird to say that ini files are not a domain specific language. And this kind of hand waving away complexity is the crux of systemd putting a bad taste in my mouth. The whole thing is a vast kingdom of nouns (cf Yegge). Forgoing expressing semantics with the config file syntax means that expressing those semantics requires creating even more nouns. Splaying the config across a bunch of tiny files forgoes juxaposition and makes for even more nouns. Owning the aspect of whether a given config file is enabled or not creates even more nouns. Needing tools to analyze the distilled config (eg dependency graph) necessitates even more nouns.
... and this was all dumped on a community that very much focuses on a smaller number of verbs - ie command line utilities. The main problem with SysV scripts is that they are bare scripts doing anything they want with those longstanding verbs. So yes, some abstraction was necessary, and that was inevitably going to seem a bit unnecessary-complexity. But systemd just took that dynamic and went overboard with it.
Sure, I can read "type=forking" and know what that means, but no, I cannot change it without looking up the documentation. Any interaction with systemd beyond start/stop/journalctl involves looking up the documentation, re-figuring-out its implicit logical model, copying the right magical incantation, and then after I've solved the problem promptly forgetting everything I figured out because the next time I touch systemd I'm guaranteed to need to know something different.
I don't dislike it enough to desperately want to move to a different init system (although if there were an easy alternative on NixOS I'd try), and I'm certainly not going to defend SysV or BSD init scripts. But it certainly feels like there is plenty of room to implement systemd's comprehensive functionality in a more user-friendly discoverable way, especially for the main type of advanced-but-casual user that interacts with init systems.
(Also as a fellow NixOS user I think it's easy to have a blindspot about systemd because its config files align with our config format. A more user-friendly concise config format would make NixOS have to do more work to splay it out for its own composability)
I remember running Linux servers for the first time and couldn’t believe there wasn’t a straight forward way to get something to run on startup. Ask glue and duct tape and non portable.
Don’t get me started on the Python situation.
I assume it’s much better now.
With the clown services U hadn’t had to run bare Linux in a long time.
The whole post does have a somewhat hyperbolic tone to it. Personally, I agree with the content and analysis, but I could see the language could feel a bit off-putting, especially given the past flame wars over systemd
Agreed, I didn't mean to tone police, just acknowledging it might be off-putting to some people who got wrapped up in the battles back in the day. I wouldn't suggest the author make any changes
I was pro-systemd at the time of the controversial discussions. I still think it's a net positive relative to what was there before. But personally speaking, it's only the core of the software (service management) that improved things for me. The other things (timers, journald) I either ignore or don't like, but perhaps they're useful for distributions.
I've been quite happy with -boot, -networkd, -resolved. -logind too, what a powerful upshift in routing input.
I forget what but timesync is missing a lot of nice to haves. But ditching grub for something much much much more manageable & a joy to use. Networkd and resolved have both had a lot of nice clear straightforward options, and I love how systemd unit files snap together for these networking concerns.
Journald has been ok for me, and I use it a lot, but not intensively. Having adequate performance is definitely a huge concern.
systemd-resolved is a godsend.
I run, at home and at work, vpns and having a good way to configure split-horizon dns is incredibly useful.
At a previous company we were able to ship a binary that would automatically configure a bunch of company settings on developers' linux laptops and got VPN configurations working with split-horizon dns very easily.
prior to this, doing split-horizon dns on linux basically meant running dnsmasq (!!!) on every linux laptop
I think if it was just service management most people wouldn't mind it nearly as much.
Yes, systemd's weirdly bundled system services largely suck. The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs) and resolvd has or used to have basic bugs that other systems don't have.
> The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs)
That sounds like an unequal comparison. For the same amount of log data and the exact same filter operation, journald should be strictly faster or equal to text logs. Are you sure this isn't because in your journald test you're going through a lot more logs? Because journald makes it easier/feasible to collect more logs.
No, journald shouldn't be strictly faster. It signs/encrypts/whatever the logs and stores them in some kind of structured format that needs to be converted to plaintext before processing it as such. That is not faster than just reading the plaintext. Grep and less search etc are very fast.
And no, I don't want to use the structured features. For error diagnosis, I want every occurrence of the search string in every field over a long time for comparison purposes.
> stores them in some kind of structured format
No, it stores them with binary metadata, rather than ASCII metadata. "Plaintext" needs to be parsed, which is expensive. You don't really think binary RPC formats like protobuf are slower than JSON because it's "structured format" compared to "plaintext" I hope. Or SQL databases are slower than CSV files.
> And no, I don't want to use the structured features.
I don't see why you'd prefer having to figure out different time formats to parse from different software logs compared to a standard time format.
I'm searching for "service name" and I don't care if it comes from "service" or mentions "service". That's unstructured.
The rest of your comment doesn't even apply. Search is not parsing and not structured data transfer (which is what binary RPC formats are for), and unless you have a fulltext index, there is no faster way to do it than the usual algorithms, which work at 100+ MB/s. With log files, the text is right there ready to be searched. With journald, it needs to be assembled first, and it's pretty slow in practice.
I've worked as a sysadmin/devops for a double digit number of years.
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
And made a number of new ones appear.
E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations.
Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse.
Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons.
I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
The grand-parent post was about the general consensus: sysadmins generally like systemd. So I'm not sure answering individual concerns is useful, but I'll take the bait...
> now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
No, you don't have to, because a syslog daemon is not required any more. BTW, rsyslogd is not the only syslog service, I've had to work with alternatives like rsyslog-ng.
And the daily usage is now much simpler: `journalctl -b -1 -p err` or `journalctl -u mariadb --since "24 hour ago"` were painful queries with (rotated and compressed) syslog files.
> Boot and shutdown has previously been deterministic, now things just randomly hang.
That was not my experience. IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
> Network device naming is also a fun one.
The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel. I think it's still possible to configure the kernel, udev and systemd to work with the old naming, but if it's just because "shorter is better", I don't think it's worth it.
> No, you don't have to, because a syslog daemon is not required any more.
That is "not required" for the single-computer-sysadmin. If all you know is your laptop, then journald might be fine. If your computers are cattle and have numbers, you don't want to read your logs on each individual computer, if possible you don't want to read logs at all, you want some automation to do it for you, aggregate it, warn you about things, etc. journald just makes that harder, and going through some syslog protocol implementation (of which there are a few, you are right there) is the only relevant game in town.
> IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
Dependencies and insserv were a later addition. Basically you just defined a boot order by numbering your /etc/rc[0-6].d/S* links in ascending order, either manually or in a dependency-based fashion through insserv. This means that you can have a suboptimal ordering if those dependencies are wrong or you choose your ascending order poorly. But it also means that once you have a working boot order, it can be repeated, and it will be repeated. That is what I mean by determinism. Once you have a working boot, it will keep working. Solaris SMF did something more modern, you would define dependencies in service definitions, and it would compute a static deterministic ordering from that, without those /etc/rc[0-6].d/S* symlinks. Deterministic, but dependency-based. If it has worked once, it will work again. And they even had a rollback: if after a change, the next boot wouldn't work, you could go back to the previous boot order and fix things from there. And neither had race conditions, because you usually wouldn't do things in parallel, one service would wait for the previous one in the boot order (parallel was possible but rarely used, because of the inherent fragility and possible race conditions).
Systemd does it dynamically on every boot, no precomputation, just looking at all the files and starting everything that could possibly be started at the same time, in parallel. The reason for this is that this makes Lennarts laptop boot faster, which is true in the ideal case. But it is neither deterministic, nor debuggable. And other than the previous methods, it can lead to race conditions.
Note that there is one more race condition at boot, which is races against kernel devices and systemd-udevd. That neither got worse nor better with systemd, it just lead to the kernel people giving up and handling more things like firmware loading in kernel.
> The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel.
Utterly totally completely wrong. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
The Linux kernel does not have a net.ifnames parameter, and the kernel interface naming scheme is still eth[0-9]+. When you stick that parameter in the kernel commandline, the kernel ignores it and only systemd and systemd-udevd interpret it later on.
Please, if you want to sing the praises of software or deflect blame from broken crap, make sure that you know what you are talking about...
It's a little weird to reply to someone complaining about administrative problems caused by systemd with "generally sysadmins seem to like it."
That kind of response is exactly why everyone gets emotional in these discussions. Technical problems are one thing but when you gaslight people experiencing and reporting legitimate problems you should not be surprised when they get mad and rip your solution out.
GP is being far kinder to the systemd devs than they deserve, I gave up on the project years ago for exactly this reason.
> consistent-net.rules nailed it to a mac address on first boot
fails even harder for
> put in a live-CD to debug something and your network devices are suddenly weird
I don't know that there is really any good answer here. Just methods that break in ways you're used to, and methods that break in ways you aren't. I appreciate the attempt at static naming based on hardware connection, even if the realities of how these things are presented by the bios make the results an inscrutable mess.
Maybe pure enx would have been better? But that looks like crap for even the simple cases, and fails if you swap out a piece of hardware or start doing weird things with MAC addresses.
The only consistent naming scheme I've ever seen working was Solaris on Sun hardware. But that only worked because they controlled both the hardware and the software side of things.
And yes, consistent-net.rules doesn't solve LiveCDs and reinstallations. But as I've said, systemd doesn't solve those either, and in most cases actively makes them worse.
My experience has not been that the systemd device names are "actively worse". The biggest problem I've had is changing around VMs where subtle shifts in PCIe IDs (specifically subdevices) end up changing changing names. Meanwhile, consistent-net.rules seems like a hack, always in the back of your mind as an arbitrary stateful part of the system. Why should I think of my primary network device as eth2 if there is no reason for it to be eth2 other than merely being the order on the first install?
But if you prefer consistent-net.rules, then more power to you. I'm guessing most of your frustrations could have been addressed by systemd having some shame about changing functionality out from under users, and providing an up-front option to switch between the different methods.
I used to be fine with it, until I started writing my own software tools, then looking at how to actually do stuff with systemd and I'm like, nope nope nope nope. At some point when I have space for a new project I'm going to migrate from Arch to Obarun, which uses a modern update on the excellent daemontools to provide init. When systemd works it's mostly fine, when it breaks, path of least resistance is often to reinstall the OS. Which is fine if you're just working on it. Painful if it's your personal machine, and those are what's most likely to break systemd.
> when it breaks, path of least resistance is often to reinstall the OS
Can you elaborate how do you even break this?
Adding a systemd unit cannot really break much on systemd itself. Unless you’re messing with internal systemd stuff, I am not sure what is even there to break.
At the moment, on this machine I'm using, audio just doesn't work. Dunno why. Tried a few different things to fix it, no dice. Trying to troubleshoot is made enormously more difficult by the complexity of systemd. If I had a simpler system I could run it all the way back to the source, but really, I can't be bothered, I don't really need audio on this machine atm.
i have the same sentiment, and for me systemd was getting so slow and hard to tshoot (the tools to analyze boot graphs and sequence showed service sequence breakage, some times there is no problem, sometimes sequence changed and problem happened) and i just changed for another distro without systemd and boom, startup in seconds no problem at all.
Which distro did you land on?
This doesn't match my experience at all. For the life of me, I cannot imagine what you are talking about, what any of your issues are. Did you file any tickets, or report any issues?
What about "actually doing stuff with systems" was "nope nope nope nope"? What situations did systemd break in? What did you try before giving up to reinstall the OS?
Thise fud train is cast aspersions, no meat.
+1, systemd is very usable, and works. I would really not go back to what was before
You probably use a desktop environment like GNOME or KDE. Maybe do some light coding in an IDE. I do not. I run window manager only, i3, with a really stripped down toolset. I like the idea of knowing my system in and out. My current project is writing my own shell. It can switch between command mode and ruby REPL with ctrl-z.
You mention filing tickets / reporting issues, sorry, that means contributing to a project. I'm not wholly opposed to doing that, but I have to like the project first. And there's nothing to like about systemd.
You go do that if you like it so much. My guess is you simply won't, and will just keep using other people's software without a care in the world, thinking this kind of online warriorship is actually helping anything.
Bro, I've been running wmii since before i3 forked off from it. Don't come at me with this dripping disdainful attitude, pretending like you are some high and mighty elite system whisperer.
It's so unfortunate that there is such a widespread persistence of hater-dom. It's so toxic and awful, and so rarely does it have even the faintest shred of anything to stand up on. I struggle so much to see these attractors into the negative as so compelling, so ensnaring of the human souls as they have been lately.
Systemd is amazing, provides such a rich world versus piecemeal poorly managed systems of yore. Yes you could understand init scripts with very little effort but what a sad virtue, to dare to go no further than what an afternoon of exploration might get you.
I don't fucking hate systemd. I'm marginally glad it exists. I just don't want to live under it's despotic regime. You might like it because you think trains running on time is more important than freedom.
You're the one coming at me with fucking attitude, like a 16 year old who just discovered IntelliJ. I have done the exploration, then looked online for why everything is so damn complicated now. It's because systemd colonized the world.
It's not just an init system, it's everything below the userland. And a lot of those components are very poorly designed. And it's this big ball of mush where if you use one component, thy must use them all. It's so bad Arch does not support replacing systemd, like at all. You're totally on your own.
Go ahead, try replacing systemd components in an Arch system. I'll wait. And lose the fucking attitude.
> It's not just an init system, it's everything below the userland. And a lot of those components are very poorly designed. And it's this big ball of mush where if you use one component, thy must use them all. It's so bad Arch does not support replacing systemd, like at all. You're totally on your own.
It's true, it's not only an init system anymore, it's also a service manager, a network manager, a DNS cache and resolver, a proper logging system (as in, with metadata, all in one place) and more.
Despite being opposed to it in the beginning, probably because it was immature at the time, I grew to love it even on Gentoo - my desktop and server OS of choice - which gives you OpenRC as a fully supported alternative:
- on my desktop, laptop and work VMs, it just works, with varying adoption of its components and services started, lazily activated if unneeded
- on my server, it simply makes sense for service auto-restart: OpenRC still requires you to manually enable support for service auto-restart ( https://wiki.gentoo.org/wiki/OpenRC/supervise-daemon#General... ), a feature that comes built-in in systemd and without extra complexity or components, and with predictable outcomes given the unit file
- almost all modern DEs rely on systemd, because it generally improved their session handling, sandboxing, brought user (rootless) services and other features
And generally, I think the biggest win is creating a system service management platform that can be used by all distros in the same way, without having to know 5+ init systems, having different scripts for the same service for each distro. I can basically work on my own devices, on work devices, on cloud VMs, on IoT devices, and have the same behavior, across different distros and hardware.
It's an improvement, I'll give you that much. But it's moved in the opposite direction and seeks to reduce freedom, not give more of it. I should be able to run my own DNS cache, network manager, service manager if I want to. Systemd makes that all but impossible.
I semi-respectfully disagree that I've given you undue attitude.
One of the things I really enjoyed about the Flix programming language submission today was running across their principles section. Right off the top of the list:
> Simple is not easy
> We believe in Rich Hickey's creed: simple is not easy. We prefer a language that gets things right to one that makes things easy. Such a language might take longer to learn in the short run, but its simplicity pays off in the long run.
In my view, yes, there are warts to systemd, but it is an operable understandable system which has a hard fought simplicity to it. I can go to any systemd computer and get a good idea of what's running, understand how it's wired. I use common practices and configs to change or drop in config.
My feeling is that a lot of people want easy. That systemd doesn't pass the easy test.
The past? The past was enormously complicated. It was complicated because every shop has its own way of configuring things. Every shop has their own tools to assess system state, their own way to configure subsystems like logins, dns, networking, logging, time sync, booting. There were countless countless easy answers, a path littered with the bodies of dead easy answers to this or that concern. And you needed an easy answer for each concern; easy didn't go hard to integrate the different factors.
That past was a horrible mess of easy. Systemd feels complex, but compared to the past, there is a persistent understandable simplicity that is a raw joy to use. Things make sense. When you see one thing, you sort of learn how the next works. The back to the woods rejectionism that springs up today is a bizarre weird thing; folks have not seen how wildly different and bedazzling system management used to be, within and across orgs. You might not appreciate why but there's very very good reasons, and I'm fact, the world is inherently complex, there's aot to do, and systemd has created a wonderful pervasive cross-sectional (via it's many opt-in components) "easy" that harnesses and enables the complexity that is, in a sensible glorious way. ("This has made a lot of people very angry and been widely regarded as a bad move.")
Systemd is anything but simple. And the past definitely wasn't easy. You seem to have the terms backward. For the end user, yes, it's a polished experience that you mostly don't have to mess with, and the parts of it you're most likely to interact with are, well, fine. Not great, but fine.
> operable understandable system
Every system is understandable, it's just a question of how much work you have to put into understanding it. Good design reduces the amount of this effort needed. Many many systemd components simply aren't well-designed. But they all interop with each other so there's no way to slowly wean a system off of systemd. Trust me I tried.
I don't want to understand systemd. It feels like being at work. Good on you for putting in the effort. I should be able to use software I like and want to use. Maybe even hand-write some replacements, like I'm doing with my shell. Systemd doesn't get in my way there, thankfully, it's been awhile so I don't have specifics off the top of my head for the last time I tried removing a few systemd components only to find hard dependencies on each other.
This is not how Linux needs to be. I agree the old ways sucked. I don't want to go back to them either. But neither do I want to be locked into the big ugly mess that systemd turned into. I want clean, modern, well-designed components that are independent of each other.
Systemd makes me feel like I'm on a Mac.
I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext. Somehow journald manages to provide a database which is 40x slower to query than running grep on a compressed text file. I'm all in favour of storing logs in an indexed structured format but journald ain't it.
Even doing zcat | grep is faster than journald.
I now turn off journald and use rotated pain text log files. It's more efficient in all metrics.
Journald is an odd one. I don't think it being a binary log/database makes sense. If you have a tiny operation, with a single server, then the binary database doesn't really make sense, having plain text is just easier and faster. If you're a bigger operation, you'll have a central logging solution, in which case you need journald to store the longs as plain text as well, before you can do log shipping.
The only use case where the binary format might make sense is if you ship journald logs to another central journald instance. That's just very much an edge case.
afaik journald can just forward logs via rsyslog directly to a remote server
Why would it need to store plaintext locally?
Doesn't that still involve a conversion? I believe that rsyslog can read the journald database, but you're typically not querying syslog data directly, so there's a conversion between rsyslog and logstash, Splunk, Datalog, whatever.
Some folks like having some local storage to act as a buffer in case the remote syslog server is down. Not sure if journald can do that on its own.
My view is pragmatic: if you are managing the server and I am a user, then use systemd if you want to.
If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.
FreeBSD is many great things, but ... a pragmatic choice?
It works for many, including Netflix. The jails are better integrated as a lightweight container system than Docker is on Linux, IMHO.
Yes; it does work nicely in many circumstances.
And powers zillions of unsung, critical infrastructure tasks.
I don't know what to think about Systemd.
I think however that a lot of hate comes from:
- a very opaque structure: it's actually not that opaque, since configuration files are still plain text, and better yet with more structure, but it first appears as inscrutable, since the programs aren't shell scripts;
- a set of new tools to learn: Systemd doesn't make use of the existing Unix tool-set (sed, awk, etc), ie. the vocabulary most sysadmins are familiar with in this ecosystem.
It seems on the outset that Systemd is trying to get away from the traditional Unix "one program should do one and only one thing well" (which is colloquial phrasing for separation of concerns).
Still, one idea that occurred to me is that in the early days of Unix, system programs might have been quite simple, with very few options (that man pages were probably very short), but that's clearly not the case nowadays. So haven't we left the "one program should do one and only one thing well" paradigm a long time ago anyway?
Actually, I think that one crucial issue with systemd is its specialization: can it be used for anything else than service management? At first, this seems to clash with the Unix principle, but even awk and sed are meant to do one thing, yet their field of action is very general, not just confined to handle one style of files or perform one style of transformation.
I remember when I first upgraded a system to a version using it. This was probably a RHEL 6 -> 7 update back in 2014 or so.
It made so many common things I had to do so as a sysadmin super straightforward.
Same, that was my first experience as well. It was a breath of fresh air in many ways, and after my initial learning of the new syntax, I quickly became a convert.
That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
I think the biggest problem with systemd is that they named the main init service manager systemd and also named the organization systemd! It'd be like if GNU named their initrd "GNU". Their other tools just have systemd prepended, e.g. systemd-journald, much like GNU Make or GNU libc (glibc) or similar. The "scope creep" issue doesn't get complained about for GNU because it's clear they're an organization making a bunch of tools for different purposes. Systemd-the-organization is the same, but the naming makes it unclear that many of these tools are not depended on by systemd-the-init-system.
And some things got super hard to do, e.g. sequence. I guess it is sorted now, but either way, I wouldn't want to go back.
Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example).
You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
I use journald whenever I feel my blood pressure getting too low.
It's slow, truncates lines, and doesn't work well at all with less. It's almost like Pottering created it so that pulseaudio wouldn't be his worst program anymore.
One convenience of journald is that it exposes a single place to plug in log collection for observability tooling
opentelemetry-collector, promtail, and so on have native plugins for it, which makes aggregation easier to setup
Most tools have "tail this plaintext file" as well, but if it's all flowing to journald, setting up log collection ends up being that much simpler
That is what syslogd has been doing since forever. Journald actually made this harder due to not supporting the established syslog protocol.
And a lot of software don't use syslog because it's easier to print to stderr/stdout or some random log file. Journald makes it easier to capture everything no matter what the software does, including the established syslog protocol, so I don't even see your point.
If everything on your machine uses syslog, journald is a drop in replacement for the dozen of possible syslogd implementations.
journald isn't drop-in because it only saves logs locally. syslog also is a protocol to send your logs to a log-server.
And the usual syslog APIs are 2 lines: Initialize with openlog(stream, process_name, flags) and after that do syslog(urgency, message). That is on par with stderr/stdout, and far simpler than handling your own logfiles. Except if you use log4$yourlanguage or something, then everything is just the same, you just configure a different destination.
And if you can't change your code to not use stdout/stderr, you can easily do yourcode | logger -t yourcode -p daemon.info 2| logger -t yourcode -p daemon.err
Pipe it into lnav eg:
journalctl -b | lnav
or use -f instead of -b for follow instead of everything since boot. Now you have a colourised journal and the power of lnav.
For debugging my desktop:
For anything else: export to a syslog server, which basically any tool that matters will support in some fashion.journalctl: unrecognized option '--tail'
I architected and built our entire log-ingestion pipeline for intrusion detection on it, at Square.
I built a small Ruby wrapper around the C API. Then I used that to slurp all the logs, periodically writing the current log ID to disk. Those logs went out onto a pubsub queue, where they were ingested into both BigQuery for long-term storage / querying, and into our alerting pipeline for real-time detection.
Thanks to journald, all the logs were structured and we were able to keep a bunch of trusted metadata like timestamp, PID, UID, the binary responsible, etc. (basically anything with an underscore prefix) separate from the log message all the way to BigQuery. No parsing, and you get free isolation of the trusted bits of metadata never intermingling with user-controlled attributes.
Compared to trying to durably tail a bunch of syslog files, or having a few SPOF syslog servers that everyone forwarded to, or implementing syslog plugins, this was basically the Promised Land for us. I think we went from idea to execution in maybe a month or two (I say “we” but really this was “me”) and rolled it out as a local daemon to the entire fleet of thousands. It has received—I think—one patch release in its six+ year lifetime, and still sits there quietly collecting everything to be shipped off-host.
The only issue we’ve ever really ran into that I never figured out is a handful of times per year (across a fleet of thousands) the journald database corrupted and you couldn’t resume collecting from the saved message ID. But we were also on an absolutely ancient version of RHEL, and I suspect anything newer probably fixed that bug. We just caught the error and restarted from an earlier timestamp. We built the whole thing around at-least-once delivery so having duplicates enter the pipeline didn’t really matter.
Damn, honestly at this point I’m wishing I’d pushed to open source it.
Ironically, actually, I did write a syslog server that also forwarded into this pipeline since we had network hardware we couldn’t install custom services onto but you could point them at syslog. I also wrote this in Ruby, using the new (at the time) Fibers (“real” concurrency) feature. The main thread fired up four background threads for listening (UDP, UDP/DTLS, TCP, TCP/TLS), and each of those would hand off clients to a dedicated per-connection worker thread for message parsing. Once parsed they went onto one more background thread for collecting and sending to PubSub. Even in Ruby it could handle gazillions of messages without breaking a sweat. Fun times.
Since I’m rambling, we also made cool use of zstd’s pre-trained dictionary feature. Log messages are small and very uniform so they were perfect for it. By pre-sharing a dictionary optimized for our specific data with both ends of the pubsub queue, we got something like 90%–95% compression rates. Given the many terabytes of logs we were schlepping from our datacenters to GCP, this was a pretty nice bit of savings.
Well, everyone and noone uses journald. The usual way to log is journald -> (r)syslogd -> remote log destination(s). I've never actually seen any other way.
The preferred journald way of "fetch your logs periodically" wouldn't pass any audit and doesn't work with any kind of log processing software that people use.
I use it everyday?
Cloudwatch fucking sucks.
Plenty of log shippers can slurp journald. (Fluentd, filebeat, vector)
Even ChromeOS uses it, even on devices that still use Upstart.
> Even ChromeOS uses it, even on devices that still use Upstart.
I was curious about this, because I thought journald hard required systemd as pid 1, so I did a search, which promptly turned up https://www.chromium.org/chromium-os/developer-library/refer... -
> Jounald is deprecated and is about to be removed.
Hmm couldn't exactly tell when it was removed, but it looks like it lasted maybe 3-4 years. This is the commit that added it to the upstart config.
https://chromium.googlesource.com/chromiumos/platform2/+/870...
Fascinating. An initscript for journald is a special kind of cursed that I didn't expect to read today:)
You're being downvoted but you're absolutely right. The fact that Cloudwatch doesn't support journald is a major, major fail on AWS' part. It's not like this is new or obscure software.
I'm just surprised anyone wants to use cloudwatch when they don't need to. It is expensive, and far from the best observability platform.
It's always there by default. That's the only reason I've ever seen it used.
I wouldn’t call it a success it all. Almost managed to do was moving or forcing other software to get locked in. GDM, Podman, Pipewire…just to name a few.
As another comment says, it's a success in the same way Windows has been a success.
> The unix philosophy cries out: is this the end of Linux (or, as many are calling it, GNU plus Linux)?
I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence:
https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd
https://news.ycombinator.com/item?id=42918448
https://news.ycombinator.com/item?id=42889792
(I reposted my comment from previous submission)
I'm sure it's a success for someone 8-/ Mostly IBM.
It's certainly been a successful takeover of all of the userspace runtime by one giant integrated set of executables.
Pwned off as an init replacement, then feature creeping it's way into every nook of userspace.
For me, it's still a disaster for distro and system s/w independence and the "do one thing well" philosophy.
While many welcome our new system overlords, the s/w is still not liked by many many people...
Move into the void! The DJB way and runit await...
I've been a fan of systemd even if just for how much simpler creating services is. I just need to write a simple config file instead of a complex init script.
+1 - there was a misconception that initscripts were easy or simple - if you wanted _correct, bug free_ initscript you were probably looking at around 100 lines of shell script.
Or if the service didn't support pam_limits because it was legacy trash, you had to hack something into the initscript like `ulimit -n XYZ` and restart it. Now things like this are trivial and easy to solve. Using systemd makes large scale Linux systems administration much easier.
Now it has gone a bit overboard. Some of the stuff like the dns resolver or the nspawn capability seem a bit over the top, but overall, it has massively improved all Linux distributions it is used in.
Never again will I worry about trash buggy init scripts not actually stopping a service due to a stale pid file. Now it puts the service into a control group and can kill all things in the control group even if the service is bad code.
https://www.cvedetails.com/vendor/15978/Systemd-Project.html
https://www.theregister.com/2017/07/28/black_hat_pwnie_award...
journald is such a disaster. How is it possible to be so much slower than even gzipped plain text.
These were pretty much my feelings back when I envied launchd and was frustrated with some upstart behaviour. Systemd turned out better than both.
I am not a sysadmin -- so I keep forgetting the command to see logs cause I only need it once every 3 months.
So, systemd change is a pain for me.
use a shell function/alia?
(Not that I think journalctl -u even needs an alias tbh but if you want it you can have it)Keep a file with a list of commands used semi-regularly. Or have a script dump your command history into a file.
Better to spend a minute searching for a command in a file than to waste an hour recreating it.
AI is so helpful for doing those kinds of tasks in the terminal that you only do a couple times a year and can never remember the exact incantation
You only need to look those things up once, put it in a snippets.txt file, and then search in the file when you need it.
I keep a per project file like that with common commands and related things.
systemd has been a success for 2 groups, Windows admins and Fortune 500 Companies.
For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though.
For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.
DRM is almost there. Try using Netflix on a *BSD, you can't. The DRM Netflix uses is not supported on BSDs.
No, it's terrible. I think what they mean by "success" here is that it functions, and is easier to use and more sturdy than init scripts. But any number of things would have been better than those. Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now.
Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.
> Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.
OpenRC is great, there's just no flashy marketing behind it or team of consultants pushing it.
EDIT: Heh. At least I can debug a shell script, I can't debug a config file.
I think this is a fundamental disagreement many people have in what's ultimately desirable when designing software. I personally can't stand non-trivial config files and both at work and at home have wasted way too much time pulling apart software consuming them.
The problem with systemd is that, because of all the new userspace APIs it adds, we can't just agree to disagree. You either target a Systemd OS and the users get to suffer non-trivial, often broken, configs or you target a normal GNU/Linux OS and the systemd users suffer all kinds of surprises when systemd rug pulls them.
#!/sbin/openrc-run
depend() { need net }
# Export some ENVs for it export SERVER_ADDRESS="0.0.0.0:8088"
name="my-program" description="My program" command="/usr/local/bin/my-program" # If it does not know how to background iself command_background=true pidfile="/run/${RC_SVCNAME}.pid" # To have logs output_log="/var/log/${RC_SVCNAME}.log" error_log="/var/log/${RC_SVCNAME}.err"
It seems that it still uses Turing-complete shell scripts as configuration. Hard pass.
>Who exactly was going to write the systemd alternative?
And that quote proves 100% what is wrong with Linux these days.
In the 90s, if someone posted in USENET they want init to do X, at least 2 different solutions would appear with in a week, probably many more.
That happened to me when I went from Coherent OS to Slackware in the early 90s. I asked if there was a way to have virtual consoles on my VGA monitor and my B&W Hercules monitor (386sx), or to have at least 1 VC on the Hercules. Coherent 486 supported that, Linux did not. A post appeared in a couple of days with a patch.
Now, between the Linux Foundation and the Companies that owns it, that would never happen. People who have gotten into Linux in the last 20 years cannot imagine how responsive the developers were to user requests back then.
Seems these days we are dealing with some corporate behemoth drowning in red tape.
Those alternatives exist, actually. OpenRC, dinit, runit, and s6 are all good options and all used by some distros.
Hi, author here. The title is intentionally a little attention-grabbing hyperbole but I've appreciated the discussion and feedback about the post. Thanks for reading!
This is a nit-pick, but if you picked the title that way to BE hyperbolic, I am not sure if that is the best thing to do for a technical discussion. It sets an internal opinion before even clicking on it. It subconsciously makes people go into the article with an already preconceived notion.
I like systemd, but do agree that it's clunky in a lot of places. I'd be interested in hearing from anyone who has used Obarun or its 66suite.
You can have S66 with AntiX too. Just use the 'Init Diversity Edition' (Respin), and pick it from several other options. AFAIK the author of S66 directly collaborated with Antix, to make it perfect.
I personally love the init/service/unit-file portion of systemd and have few complaints. It has a lot of powerful features for security, cron, etc that are very simple to use. I also really like journald.
The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.
And those ancillary managers are usually half-baked or outright broken and dangerous, in that they only implement half of the stuff they are trying to replace. E.g. https://github.com/systemd/systemd/issues/25676
Hard disagree. Name resolution feels like a solved problem, so why fix it if it isn't broken?
But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better.
I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved.
Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
The older init systems were incredibly frail and often did not work. The reason is because Bash, as much as we all love it, is actually super mega ass. It's very bug-prone and actually one of the least portable languages ever invented, in practice. Because you can't, or shouldn't, call to any other binary from within Bash, otherwise you risk your portability. Now your script won't work the same on other platforms, or even the same platform with slightly different configuration.
But um... yeah... calling to other binaries is the entire ethos of Bash. It's a shell, that's supposed to be all it does. Even getting a "true" value should be a call to another binary.
Unit files, despite being a bespoke format for one singular program, are actually more portable. Like, WAY more portable.
I think even for new people the systemd configs are much more complex (and the APIs are certainly much more complex) than the libc ones which worked well (often better.)
The problem is that almost nothing in Systemd shares theory with the rest of the OS. It's entirely its own island of new ideas, commands, and configuration language that you have to learn and memorize.
> i used to think that systemd was made the default and adopted by most distros because of its ease of use and the fact it supplied a whole bunch of things in one suite and i see where the appeal is in that but after switching to artix openrc, im just lost on why they decided to use systemd when openrc is objectively better when it comes to being an init system and for managing services, and all the other components of systemd suite can just be replaced, like why would they do this?
I was following where the author was coming from until they quoted this, and rather than addressing it they just slammed it, slammed the author, and moved on.
It's weird to say that ini files are not a domain specific language. And this kind of hand waving away complexity is the crux of systemd putting a bad taste in my mouth. The whole thing is a vast kingdom of nouns (cf Yegge). Forgoing expressing semantics with the config file syntax means that expressing those semantics requires creating even more nouns. Splaying the config across a bunch of tiny files forgoes juxaposition and makes for even more nouns. Owning the aspect of whether a given config file is enabled or not creates even more nouns. Needing tools to analyze the distilled config (eg dependency graph) necessitates even more nouns.
... and this was all dumped on a community that very much focuses on a smaller number of verbs - ie command line utilities. The main problem with SysV scripts is that they are bare scripts doing anything they want with those longstanding verbs. So yes, some abstraction was necessary, and that was inevitably going to seem a bit unnecessary-complexity. But systemd just took that dynamic and went overboard with it.
Sure, I can read "type=forking" and know what that means, but no, I cannot change it without looking up the documentation. Any interaction with systemd beyond start/stop/journalctl involves looking up the documentation, re-figuring-out its implicit logical model, copying the right magical incantation, and then after I've solved the problem promptly forgetting everything I figured out because the next time I touch systemd I'm guaranteed to need to know something different.
I don't dislike it enough to desperately want to move to a different init system (although if there were an easy alternative on NixOS I'd try), and I'm certainly not going to defend SysV or BSD init scripts. But it certainly feels like there is plenty of room to implement systemd's comprehensive functionality in a more user-friendly discoverable way, especially for the main type of advanced-but-casual user that interacts with init systems.
(Also as a fellow NixOS user I think it's easy to have a blindspot about systemd because its config files align with our config format. A more user-friendly concise config format would make NixOS have to do more work to splay it out for its own composability)
Agree wholeheartedly.
Loonix users: We hate windows
Loonix developers: Now Loonix is windows
Loonix users: booo
… several hours later
Loonix users: yaaaay
/s
I remember running Linux servers for the first time and couldn’t believe there wasn’t a straight forward way to get something to run on startup. Ask glue and duct tape and non portable.
Don’t get me started on the Python situation.
I assume it’s much better now.
With the clown services U hadn’t had to run bare Linux in a long time.
There are numerous distributions not using systemd. So adding “complete, utter, unmitigated “ in front of success seems far-fetched.
The whole post does have a somewhat hyperbolic tone to it. Personally, I agree with the content and analysis, but I could see the language could feel a bit off-putting, especially given the past flame wars over systemd
It's a lighthearted post, so probably not worth tone policing.
Agreed, I didn't mean to tone police, just acknowledging it might be off-putting to some people who got wrapped up in the battles back in the day. I wouldn't suggest the author make any changes
It has nothing to do with the tone, it is just a fact. So there is nothing to police about.