This looks cool, I like this... but if I read the screenshot correctly, it took them 12 seconds from initrd to fully loaded. Maybe this would make sense there, but my laptop boots so fast, those 12 seconds are like two seconds. It would take longer to render the screen than it would just spraying its usual text to the VT.
(Honestly, it spends more time in uefi preboot than it does to actually boot the system.)
This is an ongoing pet peeve of mine, how the UEFI of a completely solderd laptop board can take longer than the OS to initialize a fully-known set of components.
It's not really UEFI issue, but what modules were loaded, how it was configured/wired up - and sometimes it ends up serializing on slow boot of some components (like DDR5 memory), and weird crappy add-ons from manufacturer.
DDR5 training is a huge (UX) issues wrt. boot times (mainly on desktop systems).
sure only "once" if your system is correctly setup, but I have seen too many cases of buggy motherboards not properly handling this
and the biggest issue is the transparency, i.e. you don't see that it does DDR5 "training"/"parameter self adaption". It looks like it's "just stuck" in early EFI.
I was unprepared for how long training would take on my Framework laptop kit with 64 GB of DDR5, and actually thought there was something wrong with it.
Nope, training just took minutes, with nothing but a black screen.
I returned a motherboard because I thought it was DOA because it wouldn't boot but in the end it was just training for 96GB. It took around 10 minutes of black screen before I saw anything.
Similar with my new NUC which also used DDR5. I genuinely thought there was something wrong, powered it off and reseated RAM and such. Kept debugging it for about 30 minutes before I searched and found out it might be training. Left it alone and sure enough after about 15 minutes the display started showing stuff.
I wasn't aware of DD5 training being expected to take that long often. That would explain why the box that is my current home-lab taking so long to boot when I first put it together (and on one other occasion) but behaving normally every other time. Every day is a school day!
The memory training happens before UEFI is initialized proper, though it provides a minimal framework for handling such parts - it's mainly so that every bit of very early code can properly pass information later on.
As for reusing the training, that's what normally happens, but there are various things that can trigger retraining beyond swapping anything in memory modules, and it's perfectly possible the vendor somehow managed to write an add-on that triggers that on every boot (just like intel managed to wreck NVRAM on every netboot, for example)
The training reuse is already happening on machines with regular user-replaceable DIMM modules. It's why there is such a scary long wait on the first boot of a newly-assembled system, but not subsequent ones.
So even if it can change, starting out with a simple "Are the same modules still installed?" check saves a lot of time.
except this should only happen _once_ (until you remove/replace components, reset the BIOS etc.). Which is why it normally doesn't happen for laptops, it was run once in the factory and DDR being soldered one mean they can just "fixed" disable the retraining avoiding bugs in the "retraining needed" detection
but yes, there are pretty bad motherboards out there, like the one I have turned out to be a complete shit show with multiple sever design issues I unluckily "lucked out of not running into" in the first 2 years of usage making a return unlikely and a replacement is useless as this is are fundamental issue affecting all models AFIK.
I usually don't like TUI's and just want outputs into emacs text buffers, but this interface is beautiful, A nice thing to glance at that tells you everything you need to know about your systemd units. Well done.
For profiling the latest boot, use `systemd-analyze critical-chain` (Unicode tree of blocking units) or `systemd-analyze plot > boot.svg` (SVG profiler view of all units).
When you do that the stuff at the top (at last for me) doesn't change much.
I see you can add parameters after the "status". Any suggestions for interesting things to add there?
but also no reason to not render the tree with proper graphics, TUI is an UI just like GUI is and living dags are particularly well suited to be animated.
This is outstanding user interface design. Whoever came up with this is a real talent.
I would love to understand the process of coming up with the UX design is, what the principles are, and where the understanding of what "good" looks like, comes from.
Heh, I’m so old that I remember when systemd promised that the new init system would be easier to understand than those "vague startup scripts."
Meanwhile, on systems where I don’t use systemd, the entire initialization process can be displayed in about 30 lines on the screen - and I don’t have to wade through a chaotic web of dependencies, some of which are even auto-generated. As far as I can tell, I currently have about 350 of them active.
Boot times were supposed to be fast, too - yet, curiously enough, systemd often ends up saying, "waiting 1m30s for..."
I wonder if its because systemd is more rigorous about prerequisites. With systemd its normal to wait until networking is up before starting, say, a webserver. But in the days before you just started it without checking.
If it’s only happening under systemd, then I’d argue it’s systemd’s fault.
On one side, you have shell scripts that are generally pretty simple (FreeBSD’s init is something like 1000-2000 LOC for the implementation, and a simple service script is tens of lines).
On the other side, you have a monolith that hit 1.3 million lines of code a few years ago, and has gotten more and more complicated since then.
Assuming the waits are not a bug, then they are a misconfiguration. When I gave up on using it, the manual was long, incomprehensible, and the small documented fraction of systemd’s behavior didn’t match the manual.
> On the other side, you have a monolith that hit 1.3 million lines of code a few years ago, and has gotten more and more complicated since then.
Just a nit, that 1.3 million lines was reported incorrectly. You can see some talk at https://news.ycombinator.com/item?id=21935186 and I remember a Poettering tweet saying that was including things like documentation. At the time systemd was under 500k lines; still big but not quite so huge.
I'm also not sure if that count was for the full repository (which contains multiple modular projects like systemd-resolved that are not required for systemd) or just the actual systemd bin.
Imdeed. Ubuntu tried its own upstart for a while, and there were a few others too. Gentoo has (had?) some alternatives to sysV.. Devuan felt like boycotting systemd. Didn't seem to help stop the spread but at least you could use something else if you wanted. Systemd is indeed a bit of a mini OS under the OS.
We came a long way but I also feel that the improvements didn't necessarily result in faster boots.
Yep. I remember upstart, as well as some really neat soft-lockups, when a system gone "bricked", even without console, in result of bad-written network address :)
To be honest, from very beginning I viewed both (upstart and systemd) as dead-ends. Dependencies? Meh, rcorder in NetBSD has local dependencies years ago and, for larger installations, local-only dependency system has no value - because my app needs mysql... on another server. And modern init and supervision suite (systemd) without concept of inter-network dependecy? Meh. Like... let me think... a project of modern desktop without concept of remote work in eve of remote work explosion and covid times too - or something like that.
Supervising? I already had daemontools, monit, runit and whatever else (not mentioning s6) - even nowadays I really don't see cases, when systemd is used to restart services automatically.
This also tickles some part of my brain that likes watching spinners, and it gives you an intuitive idea of what’s happening on your system. But for those who find it scary you could just default to a splash.
Come to think of it I don't think I even have parallel boot enabled on my OpenRC systems since the Linux boot time is so much shorter than whatever the BIOS does before it hands control over (or before it even displays anything on some machines).
I dont have parallel boot enabled as well as it is disabled by default. Looking at this Systemd post triggered this thought to experiment with it one weekend. I have this laptop where I run Gentoo like a control freak. There is nothing in it other than the core system running as openrc services. So, I might play around adding readiness gates to those and see if anything at all can be parallelized.
Who knows, but any desktop and laptop I had access to in the last decade spent more time in POST or whatever than it does in OpenRC. Might be more of a case of the Linux bootup with a minimal set of services I actually need already being pretty fast even without paralellism rather than those BIOS being particularly slow. Windows for example is a lot slower there from my experience - and the difference is even more ridiculous for shutdown.
For me, it is pushing Linux in the opposite direction of the main reason why I like Linux.
I was attracted to Linux because unlike Windows or macOS, I actually have choice with Linux. I can choose my filesystem, my libc, my init system, my bootloader, my system encryption, my window manager, etc.
Sure, it's not perfect: many programs assume that you have glibc, or dbus, or X11, or wayland, etc. Actually many programs assume that you run Ubuntu. And it is frustrating when you would want to use a program and can't because of such an assumption. But again on Linux, there is (or used to be?) a big open source culture. This program is assuming glibc? I can patch it to work with musl. Etc.
Systemd, to me, sounds like it says: "why would I care about interoperability? I am the best, you should use me. And if you want something else, you should make that something else interoperable with me, because I make the rules". It's not like systemd is trying to standardise stuff to allow alternative components to exist, right? It doesn't actively try to prevent alternatives, but that's pretty much it.
And normally I wouldn't care (again, Linux is about freedom of choice for me). But systemd being as successful as it is dramatically reduces my freedom. And systemd it seemingly trying to take over more and more components. And the more adoption it gets, the more power it has. You didn't want systemd to replace that part of your system? Too bad, it will. What are you gonna do anyway, systemd is too big to care about you at this point.
A lot of people prefer to stick to the old sysV-like initscripts setup that most distros used before systemd (and is still an option on some, though with extra steps like adding orphan-sysvinit-scripts in Debian and manually removing the relevant systemd parts as IIRC there isn't a standard install-with-sysvinit option without installing a much older release and upgrading via a couple of dist-upgrade cycles (which keeps your old setup as much as is practical)). It works for them, has less bloat that they might not need, can be easier to debug, and they know it well, …, though isn't as well-supported any more with some packages that provide services just defining systemd jobs and you have to work out an alternative yourself. Debian even includes other init systems (like the one from Gentoo) - there is more than systemd out there if you want to try something different. Some people would still prefer upstart, which Ubuntu used for some time before moving to systemd.
I'd prefer to keep the older sysvinit init scripts, with all logs just going to text files rather than the systemd journal, but not so much that I make the effort to “fight” to keep things running that way on new installations. A lot of stuff still works the old way anyway (journal logs being mirrored to files in /var/log, init-like scripts in /etc/init.d calling systemd parts to mimic what the script on its own would have done previously so you can still `/etc/init.d/apache reload` or such).
Not OP, I run artix linux with runit. You can choose from 3 different alternatives in that distro, each one has its pros and cons (smaller size, easiness to setup and manage services, etc..)
The main idea behind inits other than systemd is that stuff in linux should aim to be small and modular and systemd is a giant that wants to do everything.
- Too many parts.
- Doing too many things, sometimes poorly.
- Fights with other services, no graceful fallback & logging (i.e. aggressive).
- As a result, unnecessarily complex, brittle and fragile.
- Harder to secure (at code level) than simpler tools.
- Project is too aggressive and want to replace any and every service.
- Sometimes break in unexpected ways.
- Not faster than Parallel Sys-V.
We can discuss these. I'm not the one who wants flame wars.
You can run the command with time, get the real field, subtract from 30, sleep that amount and run the command again in a script, and call that every minute from cron.
If the command is really instant you can do "command & sleep 30; command".
* * * * * user dothing
* * * * * user sleep 30 && dothing
them you don't need to muck about noting start time if `dothing` could take a significantly variable number of seconds to complete. Many moons ago I did something similar for a task that I wanted running every 10 seconds (it was a quick hack taking a few readings and dumping the data for me to later graph).
If the task might sometimes take 30 seconds to run you have extra faf debouncing it to avoid race conditions, with this solution or yours, which IIRC systemd timers will deal with (by being configurable to not run the next instance if the previous has not yet completed).
It has the exact right amount of parts. A modern Operating system just has a lot more to cover than 1970's Unix...
I can remember having to deal with SysVinit and thank god for Poettering and all the other systemd maintainers for bringing the Linux ecosystem to the 21st century
I think the reasonable argument is not old thing is better ( like some people make, but it has its own problems that are usually because of it trying to do everything.
My main issues stem from it's prevalence in server Distro's especially for micro service images.
D-bus is one of the great offenders for me and I don't like having it on even my desktop system but systemd wants it for many things and some things will just break without it. On a small micro service image, it would be nice to be able to strip lots of it down but systemd fights you on this because of the amount of moving parts and interconnected subs-sytems it introduces.
I concur. My main pain point is the size and aggressiveness of the whole "subsystem". I compare it to Sys-V init, because it's the best example I have used until systemd.
I have been finding myself more and more reaching for alpine linux and have liked openrc.
I am a nixos user so have to use systemd on nixos and that makes it very painful, but I can see myself moving away nixos as my host machines os for many reasons but systemd and the painful way nixos uses it is a big one.
I'm using Debian for more than I like to admit, and I'm entwined with the project somewhat, considering what I do as a work, and how much I like Debian as a whole, distro-hopping looks a bit distant, and I'm not very keen on leaving Debian as an ecosystem considering what's possible at the distro level.
Could you elaborate on the reasons for considering moving away from NixOS? :) I like it in principle, but my main issue is that if you filter out the package activity from the repository, the actual core contribution is low. At least in comparison, and if you filter out the ones with a few one off commits, the actual primary contributors are just a handful so to speak. I worry that, since there are so few who truly understands the internals, if something happens to them, things may go more awry than expected ... or something like that. It's been a few years since I last checked the project. There's also that, at the time, pull requests were growing much faster than being closed or merged, not sure if that's still the case? So very curious to hear. :)
> Could you elaborate on the reasons for considering moving away from NixOS?
Sure I also don't want to take away from what nix and even NixOS does do well, it is a great project and has many great ideas.
Also not sure you are aware (I think you probably are but I will state it any way) Nix != NixOS, and to complicate things further Nix it's self has many parts, it's a build system, a language, a package manager, enviornment profile manager (think like what python venv's provide) and kind of what ever you can make it do.
I will say Nix as a tool is a fantastic thing to have around and now with AI the wrangling and learning can become less painful but it does allow you to do a lot of cool things. Building Deterministically Deterministically building environments, integration tests/ setting up environments for such tests, more easily managing multiple possible builds for a project (well easier then wrangling make files or what ever directly and does provide nicer abstractions (not pain free) for building code). If nothing else give nix a try for building your next project and see what you think, very nice for setting up your dev env as well.
Personally I use something called flakes which is a way of managing a project and pinning it's deps. And this is one of the points things will break down and you will understand some of the issues. flakes are supported and used by many but also many reject them and the tooling is just kind of their and breaks some bits of nix. This aspect of different competing community standards is very very common in the nix world and is one of the major pain points.
> I worry that, since there are so few who truly understands the internals.
I have not filter it in that way but, I am not sure the issue lies with the lack of knowledge of the internals but nix seems to complicate things... a lot.
You have the nix daemon and nix store. Until you have good working knowledge of how nix works (not necessarily knowing the code I have only glanced at the internals of daemon and such and the more important aspect is actually the code written in the nix language generally). Don't touch store directly including trying to copy files manually between stores and such the daemon keeps a sql database of the entries and other things about the nix store means you should not directly touch it until you understand this and it's not intuitive or made to be understood easily.
>NixOS?
Finally NixOS it's self is great especially at the start and once you get it but the whole idea is declarative configs and well I know have spent lots of time fighting this and while it does have escape hatches they can be obscure and confusing and error prone. An example of this might be something like llama-swap module, you configure your modules in nix and llama-swap will module will configure a systemd unit for you once enable it which sounds great but in truth you probably want to do things outside the config and make ad-hoc changes and you can set it up so all this is possible but you end up finding your self avoiding a lot of the declarative features and having to edit your config then build it and push it every time for things that are small can be very annoying.
The thing is though you can generally just tell it to use a path to your config or give it a config or even not cofigure a systemd unit at all, but many of these things stack up and you find your self fighting it a lot of the time. The default will have things like a firewall unit that pushes some 'useful' firewall rules that block listening and such and works with declarative configs to enable particular ports which is good, but if you don't know that you might spend a bit of time wondering what the hell is going on.
I would say if you still interested look first at nix tooling and nix and if you want to dip into the operating system you will understand a lot more about why things are the way they are and if you are someone who is finding they want to make a lot of quick ad-hoc changes to the system in question nix can get very annoying. The language is not great and difficult to learn, the main issue is that it's not typed and a lot of the features and right way to do things is arcane and occult, AI will help learning all this a lot though and bare in mind it's a lazy functional language.
Alpine for me is looking more attractive to build my host system on and still use nixos for some dev vms and other vms I use and configure Alpine in a well defined way but still lets me quickly hack a bash script into something. Again you can use nix it's self on any distro and even mac and windows (I think).
My problem is not just Systemd but some issue around workflow. (though I love lisp and Guix looks Fantastic and shepherd seems quite nice.) I will probably play around with it as VM, I am moving towards having a very simple hypervisor base operating system (not using proxmox though).
It doesn't have to replace the cron, resolver, time synchronization, system logs, bootloader, home folder management, all of the disk encryption, TPM, signing, users, and a whole lot to manage services.
I can understand the frustrations with Sys-V init scripts, but on the other hand, I never had a problem with that both as a sysadmin and a distribution developer.
To be honest, loginctl allowed us to pull some hacks, but to be fair, that hacks were to handle bad behavior from a very bad codebase. Any self-respecting development team wouldn't have written anything which needed OS/kernel level reaping to clean from system correctly.
We have been chatting recently, and someone who's more experienced than me said that they have never seen Nixie's cron to die spontaneously in the last 30 years. systemd's timers not only did that, the systemd didn't bother to restart it. Ironic, considering that the biggest feature of systemd is service supervision and unattended recovery.
But isn't that the point of it having many parts? You don't have to use all of them.
I'm not that much of a sysadmin, but as a Linux user and home lab owner, I definitely prefer the integrated systemd solution. At least I can be certain that the interactions between the components are known and tested. If you instead throw together loads of KISS tools in the old UNIX style, all your tools might be individually correct but their interaction is just bad.
The problem is, these parts sometimes are forced upon you, creating rugpulls. You update your system, and tada! something has changed without you even notice, and that might be something you have relied on for a decade or more.
Learning new things is not a problem, but now I have to adapt into that thing and it might not even perform the same, and a fallback might not be possible. Opening a great can of worms you didn't want in the most unpleasant time.
One of the problems with integrated solutions is how much glue code they contain to handle these interactions. KISS tools talk over sockets, via very simple protocols. i.e. they shout each other over mountains using well defined horns and the voices they use are distinct and simple. Less glue code, simpler codebases, more robust tools.
KISS, Everything is a File, Send conservatively but receive generously are three pillars of what makes *NIXen robust systems. If that interconnectedness and meshing of things were robust, we wouldn't have made Windows a laughing stock for its crashes and security problems.
Tell me, how can a WMF or cursor file can infiltrate a whole OS? How can not you print without installing half of the desktop or literally can't have a terminal only Windows Server system? Because everything is connected and they can't be separated anymore.
These "tools thrown together in a bucket in haphazard manner" have kept systems running for literal years between reboots because they talked over these simple sockets. Heck, many good Linux servers die not because of software, but hardware failure.
And I respect Linux because this is still true despite systemd today.
> At least I can be certain that the interactions between the components are known and tested.
Oh good _lord_ you absolutely cannot. It has been my professional experience that Systemd Project is fractally complex, most of that complexity is accidental, and approximately no one on the project is interested in investigating and fixing the intermittent-but-showstopping problems that arise from that accidental fractal complexity. [0]
Poettering and crew really built the project a fantastic escape hatch with their whole "Your usecase is legacy and we're not supporting it." attitude.
[0] The mess of tools in the Systemd Project work fine until they very suddenly do not, and then the best you can do is bodge a workaround because you're very unlikely to ever get a fix for your problem. The folks that staff the project simply don't have the attitude required to work on and maintain a ever-more-sprawling project that aims to be _the_ userspace thing atop which every Linux distribution is built. "That's hard to fix/do. I don't care that you've had a working system for 10+ years, I'm not fixing the thing that's broken for you." is an absolutely unacceptable attitude for folks involved with such a project to have. [1]
[1] If the project was a little take-it-or-leave-it side project, that attitude would be entirely acceptable... encouraged, even! But because of the position in which the project willingly and aggressively put itself into, it's not a little side project.
> It doesn't have to replace the cron, resolver, time synchronization, system logs, bootloader, home folder management, all of the disk encryption, TPM, signing, users, and a whole lot to manage services.
All of those components you mentioned have improved my linux experience on both servers, desktops and mobile clients.
There is no one thing a modern operating system should be. Requirements differ and for many systems you do in fact not need to cover more than in 1970's Unix.
I'm using systemd for ages, never had a single issue at all with it.
But do you run systemd and complain or why do you even hate it?
its linux right, if a linux distribution decides to use it, its not systemds fault to be used and if you use it and don't take the time and effort to have it your way, you shouldn't complain either?
The definition of my and your ages might differ a little. I have used many things which have came before systemd and did a lot of work on them. As a result, I have seen and experienced a lot of fail-modes of both systemd and others which came before.
One of the most important feature of prior tools were transparency. Since they were single-purpose tools written with UNIX philosophy in mind, they were simpler, more robust and has less fail-modes. Systemd is pretty opaque and non-deterministic in that regard. This is not a "old man yells at cloud" stuff. You can't trust to a piece of code you can't trace and debug. Systemd makes it almost impossible with all the intertwined gears and IPC layers it has.
On the "I didn't have any problems" with it side. It might be working on your system, for your use case. It might not show its bugs on a system with a 90-day average uptime. I don't live on that world. I manage way more systems with way more different, niche, interesting hardware and software than that, and this tends to surface edge cases and where things break.
On the "do you run systemd?" part, yes I run it. On many, many machines 7/24/365.
It's easy to say "Why do you even run it, it's Linux, just distro-hop", when you have a single system. I don't. Projects, software has requirements, compatibility targets, etc. Also, considering the hardware we run, it's not that simple either.
Lastly, I run Debian personally since version 3. We use other distributions alongside Debian in our datacenter. I'm very involved and rooted in these systems, and Debian feels like home after using, contributing, and developing derivatives on top of that.
...and how systemd has been forced into systems is another story. I have been there, watched that, so, errm, it's not all voluntary, peaceful and friendly, to put it kindly.
Been there too. The thing I disliked most, that it sometimes get stuck in some strange pose, without any reasonable way of remediation. And because it is so annoying to debug, reboot become the only tool do deal with it.
The sheer complexity of systemd created nasty feedback loop: it is too complex to be debugged; people prefer to reboot it; that keeps problems keep accumulating inside.
I hate this 'unix philosophy in mind' 'principle' as an argument.
We are no longer 50 years ago were a linux system or distro was fitting on a floppy disk.
The handful of tools like grep, cp, etc. have a complexity of a banana and everything else successful is so big, that it doesn't fit at all.
Blender, container, ffmpeg, vlc, vim, firefox and co. Huge code bases. And if someone woudl argue "yeah but firefox is doing one thing, its browsing" thats an argument you can bring for systemd too.
Systemd is just not a 'start services in the right order' tool. Its more holistic
> And if someone woudl argue "yeah but firefox is doing one thing, its browsing"
Actually, I would argue that browsers are doing too much :-). Webapps shouldn't be a thing, browsers should be for loading websites. Apps should run outside the browser in the OS.
> I hate this 'unix philosophy in mind' 'principle' as an argument.
You did not deal with the most important complains that is the easy of debugging, modifying and changing stuff. It lacks a lot of this and one can question how much of an improvement ini like files are that are then interpreted in opaque ways that to provide desired features such as containerization or jailing. I personally want to usually run containers with podman, do systemd containers give me anything or is it just more cruft, same with isolation, do I really gain much from the complex namespace abstraction rather then using unshare.
> "yeah but firefox is doing one thing, its browsing"
Browsers as they exist today where a mistake that is clear. those other tools you mentioned might be large but they don't violate do just one thing well.
I am not even that much of a unix guy (I use emacs which the unix people like to shit on of course and does not fit with the do one thing well). But you are not providing arguments against the issues we have with systemd and have not demonstrated its value over other init 1 systems and even the harder case you seem to argue for it's applicability everwhere.
> For me systemd is similar to blender, firefox and co.
I explicitly said firefox is not like this (and browsers are bad) and I don't really see your point with blender (though don't use the program so I can defer to you knowledge), I even said I am a heavy emacs user and so don't believe in dogmatically following 'do only one thing well' principle.
All you have to demonstrate is what is the value.
> But the unix philosophy thing came not from a time of blender, firefox and systemd but a time of ls, cat, grep, xargs and co
are you trying to say ls, cat and grep have been superseded? this is absurd these tools are still useful and not replaced what is your point here what are you trying to establish?
My question would be where is systemd a better experience or easy to work with then openrc, have you ever had to disable any of the systemd defaults services such as resolving or networking and had issues with the way a distro set things up. I don't know how you can claim you have never an into an issue with it given that any piece of software you will run into issues with.
You can hate it all you want, but UNIX philosophy isn't related at codebase size at all.
All the examples you give fit into UNIX philosophy rather well. Firefox doesn't do text editing and ffmpeg doesn't render web pages, and vim doesn't transcode videos. They do one thing and do it well. They are as simple as they need to be not more.
Also, if GNU coreutils have a complexity of a banana, why uutils still can't replicate them as-is? Why they had ~140 CVEs in one release despite being written in Rust? If grep is simple, why we have only two performant implementations, grep and ripgrep?
You're simplifying too much, I believe. Try writing a cp clone, why not?
systemd started as a start services in the right order tool, then bloated from there. Being holistic is not the problem, being holistic in the wrong way is.
Now, they decided that DBus is not nice enough and created their own IPC mechanism. What was that? Varlink? They don't stop there even. They say "Hey, why not try Varlink?". So to translate: "We want to kill DBus so, we will force Varlink on you".
The features systemd has and encapuslates fits very well of what a service manager should do.
Just starting a service at boottime doesn't help much and ignores a lot of stuff around.
What do i start when a user logs in? how many resources is the service allowed to use? How do i control that service and read its logs?
If i already start a service, i can also start a service based on other events like bus events (which are needed potentially anyway for services to communicate) or by time.
And all of this needs to shutdown properly when finished.
Wait till you do have an issue or even just want to disable one of the 'helpful closely coupled' services something works to better fit what ever you doing or for security/reliability reasons (D-Bus is a great example for me).
It's difficult to avoid systemd at this point and it's not because it's the best technical solution for all use cases in the same way that modern windows is a complete shit show but still everywhere.
My question would be what does systemd do that is not possible outside of openrc and launching what ever daemons you want for the tasks you want or combining it with what ever programs you want to do things like sand-boxing or containerization? Not saying openrc is a panacea but systemd is definitely not either and for many systems is just plain annoying, I can consed for people who want a normal desktop (plug and play distro experiences) experience it can provide that.
The thing with the whole systemd hate thing is everyone now knows why people might hate systemd (I am not a fan). I think a lot of features around isolation and systemd containers where useful but now probably using your containerization system that best suits your needs.
For me when I get really mad at it is when I want to build small nixos Images and want to minimize the footprint of systemd and run few services (some of that is on nixos as well).
I think it's better suited for desktop systems then server systems and it's quite annoying how prevalent it is. D-bus is one thing that always annoys me and how everything in systemd wants to use it and you can run into issues disabling it (Also don't like D-bus even on a desktop system).
Overall I think and hope we will see a rise not necessarily in openrc but in simpler init 1 systems that try to do less and provide good strong primitives that allow for easily using what ever daemon configured how you want, rather then trying to provide coupled implementations for everything.
Again I know these criticism are something of a meme, but they still are true.
> I think it's better suited for desktop systems then server systems and it's quite annoying how prevalent it is. D-bus is one thing that always annoys me and how everything in systemd wants to use it and you can run into issues disabling it (Also don't like D-bus even on a desktop system).
systemd (or rather Poettering) also isn't really a fan of dbus. That is why most new IPC APIs are using Varlink (and socket activation) instead of dbus. They've also been slowly migrating some of the internal IPC calls to Varlink instead.
Systemd probably is never getting rid of the dependency of dbus due to providing public APIs via it, but maybe it can be downgraded at some point to an optional dependency where APIs are registered if it is installed and just skipped if not.
> systemd (or rather Poettering) also isn't really a fan of dbus.
True and he is right, but it's still a dependency on systemd, Varlink is some ways better, (giving much more control over listening and control over what receives messages) but the point still stands I don't want my init system to depend on this kind of userspace IPC mechanism or have to provide this, especially on systems I want to lock down.
Only Rube-Goldberg complexity on the scale of systemd made the XZ backdoor possible. Systemd was made by despicable individuals who went on to work for despicable companies (as a big fuck you to open-source lovers) and then later on to work on even more despicable surveillance project (as a big fuck you to humanity).
systemd has its tentacles meddling in every single part of the Linux ecosystem.
But thankfully there's light at the end of the tunnel: thanks to VMs and containers and the need for speed, we've seen many minimal distros (I'm thinking stuff like Talos for Kubernetes) and those of course don't use systemd.
As soon as I get the time to switch away from my current hypervisor (Proxmox, so Debian, so systemd) to an hypervisor not running under Linux, like maybe bhyve on FreeBSD, I'll have all my VMs and containers and hypervisor totally systemd free (my VMs and containers are already systemd free: I did in anticipation of, at long last, giving systemd the middle finger).
But of course none of this matter much because according to HN we're already at a point in time where we can write: "Plz rewrite an init system".
The real question is: why don't people hate a rube-goldberg init system made by someone who went to work for Microsoft, while using crypto made up by his brother (certainly secure we're sure), and who then went on to create a full-on surveillance company working on locking down hardware?
(Honestly, it spends more time in uefi preboot than it does to actually boot the system.)
DDR5 training is a huge (UX) issues wrt. boot times (mainly on desktop systems).
sure only "once" if your system is correctly setup, but I have seen too many cases of buggy motherboards not properly handling this
and the biggest issue is the transparency, i.e. you don't see that it does DDR5 "training"/"parameter self adaption". It looks like it's "just stuck" in early EFI.
Nope, training just took minutes, with nothing but a black screen.
Also the soldered ram couldn't have changed from last boot, so reuse the last training? That's exactly what I mean.
As for reusing the training, that's what normally happens, but there are various things that can trigger retraining beyond swapping anything in memory modules, and it's perfectly possible the vendor somehow managed to write an add-on that triggers that on every boot (just like intel managed to wreck NVRAM on every netboot, for example)
So even if it can change, starting out with a simple "Are the same modules still installed?" check saves a lot of time.
except this should only happen _once_ (until you remove/replace components, reset the BIOS etc.). Which is why it normally doesn't happen for laptops, it was run once in the factory and DDR being soldered one mean they can just "fixed" disable the retraining avoiding bugs in the "retraining needed" detection
but yes, there are pretty bad motherboards out there, like the one I have turned out to be a complete shit show with multiple sever design issues I unluckily "lucked out of not running into" in the first 2 years of usage making a return unlikely and a replacement is useless as this is are fundamental issue affecting all models AFIK.
Though if it suddenly starts taking a lot longer, perhaps this sort of display could help in finding a clue as to why?
What i don't get is why booting still takes seconds. When power is on, the hardware is on.
You can initalize most hardware in parallel, reading the amount of operation system into ram is fast.
but also no reason to not render the tree with proper graphics, TUI is an UI just like GUI is and living dags are particularly well suited to be animated.
It's perfectly fine for the system's essentials to be non-fancy, to ensure they will also work under adverse conditions.
I would love to understand the process of coming up with the UX design is, what the principles are, and where the understanding of what "good" looks like, comes from.
Meanwhile, on systems where I don’t use systemd, the entire initialization process can be displayed in about 30 lines on the screen - and I don’t have to wade through a chaotic web of dependencies, some of which are even auto-generated. As far as I can tell, I currently have about 350 of them active.
Boot times were supposed to be fast, too - yet, curiously enough, systemd often ends up saying, "waiting 1m30s for..."
wouldn't that be the fault of the service it's waiting for and not systemd itself?
On one side, you have shell scripts that are generally pretty simple (FreeBSD’s init is something like 1000-2000 LOC for the implementation, and a simple service script is tens of lines).
On the other side, you have a monolith that hit 1.3 million lines of code a few years ago, and has gotten more and more complicated since then.
Assuming the waits are not a bug, then they are a misconfiguration. When I gave up on using it, the manual was long, incomprehensible, and the small documented fraction of systemd’s behavior didn’t match the manual.
Just a nit, that 1.3 million lines was reported incorrectly. You can see some talk at https://news.ycombinator.com/item?id=21935186 and I remember a Poettering tweet saying that was including things like documentation. At the time systemd was under 500k lines; still big but not quite so huge.
I'm also not sure if that count was for the full repository (which contains multiple modular projects like systemd-resolved that are not required for systemd) or just the actual systemd bin.
To be honest, from very beginning I viewed both (upstart and systemd) as dead-ends. Dependencies? Meh, rcorder in NetBSD has local dependencies years ago and, for larger installations, local-only dependency system has no value - because my app needs mysql... on another server. And modern init and supervision suite (systemd) without concept of inter-network dependecy? Meh. Like... let me think... a project of modern desktop without concept of remote work in eve of remote work explosion and covid times too - or something like that.
Supervising? I already had daemontools, monit, runit and whatever else (not mentioning s6) - even nowadays I really don't see cases, when systemd is used to restart services automatically.
It may terrify windows and osx refugees, but that's a sacrifice I am happy to make ;)
On which system? Is your machine, for whatever reason, re-doing "memory training" everytime it boots? This is one can be really slow.
Why?
I was attracted to Linux because unlike Windows or macOS, I actually have choice with Linux. I can choose my filesystem, my libc, my init system, my bootloader, my system encryption, my window manager, etc.
Sure, it's not perfect: many programs assume that you have glibc, or dbus, or X11, or wayland, etc. Actually many programs assume that you run Ubuntu. And it is frustrating when you would want to use a program and can't because of such an assumption. But again on Linux, there is (or used to be?) a big open source culture. This program is assuming glibc? I can patch it to work with musl. Etc.
Systemd, to me, sounds like it says: "why would I care about interoperability? I am the best, you should use me. And if you want something else, you should make that something else interoperable with me, because I make the rules". It's not like systemd is trying to standardise stuff to allow alternative components to exist, right? It doesn't actively try to prevent alternatives, but that's pretty much it.
And normally I wouldn't care (again, Linux is about freedom of choice for me). But systemd being as successful as it is dramatically reduces my freedom. And systemd it seemingly trying to take over more and more components. And the more adoption it gets, the more power it has. You didn't want systemd to replace that part of your system? Too bad, it will. What are you gonna do anyway, systemd is too big to care about you at this point.
I'd prefer to keep the older sysvinit init scripts, with all logs just going to text files rather than the systemd journal, but not so much that I make the effort to “fight” to keep things running that way on new installations. A lot of stuff still works the old way anyway (journal logs being mirrored to files in /var/log, init-like scripts in /etc/init.d calling systemd parts to mimic what the script on its own would have done previously so you can still `/etc/init.d/apache reload` or such).
The main idea behind inits other than systemd is that stuff in linux should aim to be small and modular and systemd is a giant that wants to do everything.
If the command is really instant you can do "command & sleep 30; command".
heh.
If the task might sometimes take 30 seconds to run you have extra faf debouncing it to avoid race conditions, with this solution or yours, which IIRC systemd timers will deal with (by being configurable to not run the next instance if the previous has not yet completed).
I can remember having to deal with SysVinit and thank god for Poettering and all the other systemd maintainers for bringing the Linux ecosystem to the 21st century
My main issues stem from it's prevalence in server Distro's especially for micro service images.
D-bus is one of the great offenders for me and I don't like having it on even my desktop system but systemd wants it for many things and some things will just break without it. On a small micro service image, it would be nice to be able to strip lots of it down but systemd fights you on this because of the amount of moving parts and interconnected subs-sytems it introduces.
I am a nixos user so have to use systemd on nixos and that makes it very painful, but I can see myself moving away nixos as my host machines os for many reasons but systemd and the painful way nixos uses it is a big one.
Sure I also don't want to take away from what nix and even NixOS does do well, it is a great project and has many great ideas.
Also not sure you are aware (I think you probably are but I will state it any way) Nix != NixOS, and to complicate things further Nix it's self has many parts, it's a build system, a language, a package manager, enviornment profile manager (think like what python venv's provide) and kind of what ever you can make it do.
I will say Nix as a tool is a fantastic thing to have around and now with AI the wrangling and learning can become less painful but it does allow you to do a lot of cool things. Building Deterministically Deterministically building environments, integration tests/ setting up environments for such tests, more easily managing multiple possible builds for a project (well easier then wrangling make files or what ever directly and does provide nicer abstractions (not pain free) for building code). If nothing else give nix a try for building your next project and see what you think, very nice for setting up your dev env as well.
Personally I use something called flakes which is a way of managing a project and pinning it's deps. And this is one of the points things will break down and you will understand some of the issues. flakes are supported and used by many but also many reject them and the tooling is just kind of their and breaks some bits of nix. This aspect of different competing community standards is very very common in the nix world and is one of the major pain points.
> I worry that, since there are so few who truly understands the internals.
I have not filter it in that way but, I am not sure the issue lies with the lack of knowledge of the internals but nix seems to complicate things... a lot.
You have the nix daemon and nix store. Until you have good working knowledge of how nix works (not necessarily knowing the code I have only glanced at the internals of daemon and such and the more important aspect is actually the code written in the nix language generally). Don't touch store directly including trying to copy files manually between stores and such the daemon keeps a sql database of the entries and other things about the nix store means you should not directly touch it until you understand this and it's not intuitive or made to be understood easily.
>NixOS? Finally NixOS it's self is great especially at the start and once you get it but the whole idea is declarative configs and well I know have spent lots of time fighting this and while it does have escape hatches they can be obscure and confusing and error prone. An example of this might be something like llama-swap module, you configure your modules in nix and llama-swap will module will configure a systemd unit for you once enable it which sounds great but in truth you probably want to do things outside the config and make ad-hoc changes and you can set it up so all this is possible but you end up finding your self avoiding a lot of the declarative features and having to edit your config then build it and push it every time for things that are small can be very annoying.
The thing is though you can generally just tell it to use a path to your config or give it a config or even not cofigure a systemd unit at all, but many of these things stack up and you find your self fighting it a lot of the time. The default will have things like a firewall unit that pushes some 'useful' firewall rules that block listening and such and works with declarative configs to enable particular ports which is good, but if you don't know that you might spend a bit of time wondering what the hell is going on.
I would say if you still interested look first at nix tooling and nix and if you want to dip into the operating system you will understand a lot more about why things are the way they are and if you are someone who is finding they want to make a lot of quick ad-hoc changes to the system in question nix can get very annoying. The language is not great and difficult to learn, the main issue is that it's not typed and a lot of the features and right way to do things is arcane and occult, AI will help learning all this a lot though and bare in mind it's a lazy functional language.
Alpine for me is looking more attractive to build my host system on and still use nixos for some dev vms and other vms I use and configure Alpine in a well defined way but still lets me quickly hack a bash script into something. Again you can use nix it's self on any distro and even mac and windows (I think).
I can understand the frustrations with Sys-V init scripts, but on the other hand, I never had a problem with that both as a sysadmin and a distribution developer.
To be honest, loginctl allowed us to pull some hacks, but to be fair, that hacks were to handle bad behavior from a very bad codebase. Any self-respecting development team wouldn't have written anything which needed OS/kernel level reaping to clean from system correctly.
We have been chatting recently, and someone who's more experienced than me said that they have never seen Nixie's cron to die spontaneously in the last 30 years. systemd's timers not only did that, the systemd didn't bother to restart it. Ironic, considering that the biggest feature of systemd is service supervision and unattended recovery.
I'm not that much of a sysadmin, but as a Linux user and home lab owner, I definitely prefer the integrated systemd solution. At least I can be certain that the interactions between the components are known and tested. If you instead throw together loads of KISS tools in the old UNIX style, all your tools might be individually correct but their interaction is just bad.
Learning new things is not a problem, but now I have to adapt into that thing and it might not even perform the same, and a fallback might not be possible. Opening a great can of worms you didn't want in the most unpleasant time.
One of the problems with integrated solutions is how much glue code they contain to handle these interactions. KISS tools talk over sockets, via very simple protocols. i.e. they shout each other over mountains using well defined horns and the voices they use are distinct and simple. Less glue code, simpler codebases, more robust tools.
KISS, Everything is a File, Send conservatively but receive generously are three pillars of what makes *NIXen robust systems. If that interconnectedness and meshing of things were robust, we wouldn't have made Windows a laughing stock for its crashes and security problems.
Tell me, how can a WMF or cursor file can infiltrate a whole OS? How can not you print without installing half of the desktop or literally can't have a terminal only Windows Server system? Because everything is connected and they can't be separated anymore.
These "tools thrown together in a bucket in haphazard manner" have kept systems running for literal years between reboots because they talked over these simple sockets. Heck, many good Linux servers die not because of software, but hardware failure.
And I respect Linux because this is still true despite systemd today.
Oh good _lord_ you absolutely cannot. It has been my professional experience that Systemd Project is fractally complex, most of that complexity is accidental, and approximately no one on the project is interested in investigating and fixing the intermittent-but-showstopping problems that arise from that accidental fractal complexity. [0]
Poettering and crew really built the project a fantastic escape hatch with their whole "Your usecase is legacy and we're not supporting it." attitude.
[0] The mess of tools in the Systemd Project work fine until they very suddenly do not, and then the best you can do is bodge a workaround because you're very unlikely to ever get a fix for your problem. The folks that staff the project simply don't have the attitude required to work on and maintain a ever-more-sprawling project that aims to be _the_ userspace thing atop which every Linux distribution is built. "That's hard to fix/do. I don't care that you've had a working system for 10+ years, I'm not fixing the thing that's broken for you." is an absolutely unacceptable attitude for folks involved with such a project to have. [1]
[1] If the project was a little take-it-or-leave-it side project, that attitude would be entirely acceptable... encouraged, even! But because of the position in which the project willingly and aggressively put itself into, it's not a little side project.
All of those components you mentioned have improved my linux experience on both servers, desktops and mobile clients.
So, experiences differ, apparently.
Just wanted to take a jab. Feel free to downvote. =]
But do you run systemd and complain or why do you even hate it?
its linux right, if a linux distribution decides to use it, its not systemds fault to be used and if you use it and don't take the time and effort to have it your way, you shouldn't complain either?
One of the most important feature of prior tools were transparency. Since they were single-purpose tools written with UNIX philosophy in mind, they were simpler, more robust and has less fail-modes. Systemd is pretty opaque and non-deterministic in that regard. This is not a "old man yells at cloud" stuff. You can't trust to a piece of code you can't trace and debug. Systemd makes it almost impossible with all the intertwined gears and IPC layers it has.
On the "I didn't have any problems" with it side. It might be working on your system, for your use case. It might not show its bugs on a system with a 90-day average uptime. I don't live on that world. I manage way more systems with way more different, niche, interesting hardware and software than that, and this tends to surface edge cases and where things break.
On the "do you run systemd?" part, yes I run it. On many, many machines 7/24/365.
It's easy to say "Why do you even run it, it's Linux, just distro-hop", when you have a single system. I don't. Projects, software has requirements, compatibility targets, etc. Also, considering the hardware we run, it's not that simple either.
Lastly, I run Debian personally since version 3. We use other distributions alongside Debian in our datacenter. I'm very involved and rooted in these systems, and Debian feels like home after using, contributing, and developing derivatives on top of that.
...and how systemd has been forced into systems is another story. I have been there, watched that, so, errm, it's not all voluntary, peaceful and friendly, to put it kindly.
Been there too. The thing I disliked most, that it sometimes get stuck in some strange pose, without any reasonable way of remediation. And because it is so annoying to debug, reboot become the only tool do deal with it.
The sheer complexity of systemd created nasty feedback loop: it is too complex to be debugged; people prefer to reboot it; that keeps problems keep accumulating inside.
We are no longer 50 years ago were a linux system or distro was fitting on a floppy disk.
The handful of tools like grep, cp, etc. have a complexity of a banana and everything else successful is so big, that it doesn't fit at all.
Blender, container, ffmpeg, vlc, vim, firefox and co. Huge code bases. And if someone woudl argue "yeah but firefox is doing one thing, its browsing" thats an argument you can bring for systemd too.
Systemd is just not a 'start services in the right order' tool. Its more holistic
Actually, I would argue that browsers are doing too much :-). Webapps shouldn't be a thing, browsers should be for loading websites. Apps should run outside the browser in the OS.
You did not deal with the most important complains that is the easy of debugging, modifying and changing stuff. It lacks a lot of this and one can question how much of an improvement ini like files are that are then interpreted in opaque ways that to provide desired features such as containerization or jailing. I personally want to usually run containers with podman, do systemd containers give me anything or is it just more cruft, same with isolation, do I really gain much from the complex namespace abstraction rather then using unshare.
> "yeah but firefox is doing one thing, its browsing"
Browsers as they exist today where a mistake that is clear. those other tools you mentioned might be large but they don't violate do just one thing well.
I am not even that much of a unix guy (I use emacs which the unix people like to shit on of course and does not fit with the do one thing well). But you are not providing arguments against the issues we have with systemd and have not demonstrated its value over other init 1 systems and even the harder case you seem to argue for it's applicability everwhere.
For me systemd is similar to blender, firefox and co.
But the unix philosophy thing came not from a time of blender, firefox and systemd but a time of ls, cat, grep, xargs and co.
I explicitly said firefox is not like this (and browsers are bad) and I don't really see your point with blender (though don't use the program so I can defer to you knowledge), I even said I am a heavy emacs user and so don't believe in dogmatically following 'do only one thing well' principle.
All you have to demonstrate is what is the value.
> But the unix philosophy thing came not from a time of blender, firefox and systemd but a time of ls, cat, grep, xargs and co
are you trying to say ls, cat and grep have been superseded? this is absurd these tools are still useful and not replaced what is your point here what are you trying to establish?
My question would be where is systemd a better experience or easy to work with then openrc, have you ever had to disable any of the systemd defaults services such as resolving or networking and had issues with the way a distro set things up. I don't know how you can claim you have never an into an issue with it given that any piece of software you will run into issues with.
All the examples you give fit into UNIX philosophy rather well. Firefox doesn't do text editing and ffmpeg doesn't render web pages, and vim doesn't transcode videos. They do one thing and do it well. They are as simple as they need to be not more.
Also, if GNU coreutils have a complexity of a banana, why uutils still can't replicate them as-is? Why they had ~140 CVEs in one release despite being written in Rust? If grep is simple, why we have only two performant implementations, grep and ripgrep?
Even FreeBSD guys discussed this: https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug...
You're simplifying too much, I believe. Try writing a cp clone, why not?
systemd started as a start services in the right order tool, then bloated from there. Being holistic is not the problem, being holistic in the wrong way is.
Now, they decided that DBus is not nice enough and created their own IPC mechanism. What was that? Varlink? They don't stop there even. They say "Hey, why not try Varlink?". So to translate: "We want to kill DBus so, we will force Varlink on you".
Just starting a service at boottime doesn't help much and ignores a lot of stuff around.
What do i start when a user logs in? how many resources is the service allowed to use? How do i control that service and read its logs?
If i already start a service, i can also start a service based on other events like bus events (which are needed potentially anyway for services to communicate) or by time.
And all of this needs to shutdown properly when finished.
It's difficult to avoid systemd at this point and it's not because it's the best technical solution for all use cases in the same way that modern windows is a complete shit show but still everywhere.
My question would be what does systemd do that is not possible outside of openrc and launching what ever daemons you want for the tasks you want or combining it with what ever programs you want to do things like sand-boxing or containerization? Not saying openrc is a panacea but systemd is definitely not either and for many systems is just plain annoying, I can consed for people who want a normal desktop (plug and play distro experiences) experience it can provide that.
As I said, i'm using it for probably a decade by now.
The thing with the whole systemd hate thing is everyone now knows why people might hate systemd (I am not a fan). I think a lot of features around isolation and systemd containers where useful but now probably using your containerization system that best suits your needs.
For me when I get really mad at it is when I want to build small nixos Images and want to minimize the footprint of systemd and run few services (some of that is on nixos as well).
I think it's better suited for desktop systems then server systems and it's quite annoying how prevalent it is. D-bus is one thing that always annoys me and how everything in systemd wants to use it and you can run into issues disabling it (Also don't like D-bus even on a desktop system).
Overall I think and hope we will see a rise not necessarily in openrc but in simpler init 1 systems that try to do less and provide good strong primitives that allow for easily using what ever daemon configured how you want, rather then trying to provide coupled implementations for everything.
Again I know these criticism are something of a meme, but they still are true.
systemd (or rather Poettering) also isn't really a fan of dbus. That is why most new IPC APIs are using Varlink (and socket activation) instead of dbus. They've also been slowly migrating some of the internal IPC calls to Varlink instead.
Systemd probably is never getting rid of the dependency of dbus due to providing public APIs via it, but maybe it can be downgraded at some point to an optional dependency where APIs are registered if it is installed and just skipped if not.
True and he is right, but it's still a dependency on systemd, Varlink is some ways better, (giving much more control over listening and control over what receives messages) but the point still stands I don't want my init system to depend on this kind of userspace IPC mechanism or have to provide this, especially on systems I want to lock down.
Only Rube-Goldberg complexity on the scale of systemd made the XZ backdoor possible. Systemd was made by despicable individuals who went on to work for despicable companies (as a big fuck you to open-source lovers) and then later on to work on even more despicable surveillance project (as a big fuck you to humanity).
systemd has its tentacles meddling in every single part of the Linux ecosystem.
But thankfully there's light at the end of the tunnel: thanks to VMs and containers and the need for speed, we've seen many minimal distros (I'm thinking stuff like Talos for Kubernetes) and those of course don't use systemd.
As soon as I get the time to switch away from my current hypervisor (Proxmox, so Debian, so systemd) to an hypervisor not running under Linux, like maybe bhyve on FreeBSD, I'll have all my VMs and containers and hypervisor totally systemd free (my VMs and containers are already systemd free: I did in anticipation of, at long last, giving systemd the middle finger).
But of course none of this matter much because according to HN we're already at a point in time where we can write: "Plz rewrite an init system".
The real question is: why don't people hate a rube-goldberg init system made by someone who went to work for Microsoft, while using crypto made up by his brother (certainly secure we're sure), and who then went on to create a full-on surveillance company working on locking down hardware?
Regain control of PID 1 people.
edit: Having written that, this tool looks very cool and useful when you need to deal with a systemd distribution.