Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup.
Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem.
If anything, it helped keep my stuff alive during all the hardware issues a lot longer.
I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running?
All the enterprisey stuff makes it feel a lot more complex than it really is.
When you are only used to the Kubernetes at work I can understand how people dislike it. If you set it up yourself and start with minimal feature and not lots of annotations the config files become very simple and not more complicated than a docker compose file.
It quickly realized that after just using the managed Kubernetes from Digital Ocean and deploying a side project there.
I got introduced to UNIX with Xenix, have used plenty of flavours including containers in HP-UX and Solaris before they became a thing in Linux, I have zero need to use Kubernetes at home.
In fact, I also have zero needs for it at work directly, as I have become an advocate for serverless and managed runtimes, unless there is really a business need to control the whole infrastructure, including the Kubernetes cluster directly.
> Kubernetes is Too Hard. I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble.
Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn.
(My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)
Hey, author here. This is a piece about moving away from kubernetes and toward something that I can actually maintain as a solo person who has a life outside of k9s. It's not really intended to be "anti-kubernetes", more like "kubernetes really is too hard for my purposes".
IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress.
I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".
Meanwhile I’m busy moving the other direction. More K8S.
Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate…
I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s
IMO, kubernetes is overkill for a small non-homogeneous home cluster.
What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately.
I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.
As someone who one week ago switched from a Debian to NixOS setup, using docker compose, I'd be very interested in hearing more if you have any resources or tips to share.
I was hoping to move over to running rootless containers, but so far my HA setup has proven to be a pita to get working.
I still don't know really what Kubernetes is for or why so many people outside specific environments are using it, but it's cool that you're using Ruby.
Kubernetes is good for two things. Zero downtime deployments and self-healing (where the looping state mechanism comes in). There are people who want k8s to handle every single operation that can run on a server, do not listen to those kinds of people they will lead you astray.
Kubernetes makes complex things (e.g blue/green deployment, auto-scaling, failover) possible irrespective of the underlying cloud/hardware with a good and standardized API.
It's absolutely overkill for small teams and homelabs (I run a cluster myself) but an absolute godsend if you do need the advanced functionality.
Somehow we were doing that with deployment scripts and VM management tooling before Kubernetes became a thing, and without having to deal with YAML spaghetti.
Yes, and Kubernetes came around as another player in that ecosystem and became popular for a reason, largely so we didn't have to manage clusters with imperative non-idempotent scripts with no runtime introspection or self-healing. I've done light devops (lab scale, not enterprise) off and on since cfengine was a thing, and while I'm no fan of the explosion of YAML (there's a special place in hell for helm in particular for using text/template to generate yaml), I'll take the controller loop design any day over most of the alternatives. Just having a sane API alone is a godsend: you ever try scripting vSphere?
Kubernetes is built around a JSON API: those yaml files are just an encoding of the objects that it manipulates with fairly vanilla REST commands. You could actually use JSON in your manifests instead, it's a proper subset and all, though it's not going to help the verbosity. For that you need a better abstraction like Pulumi, cdk8s, Yoke, or even just good old Terraform/OpenTofu. Or just write your own, there's an openapi spec and clients in every language out there.
I'm hardly trying to pretend like I'm a hyperscaler: I usually run k3s on a single node, and most of the time I admin it "clickops" style with k9s, something I find much easier than most other management tools.
Yes but Kubernetes takes these battle-tested scripts, and allow everyone to use them with a few lines of YAML ;)
I understand the dislike of YAML but a Kubernetes deployment is ~50 lines, if I had to build my own scripts with a similar feature set I don't think I would be able to get it down much more than that.
I did the same move as you away from k8s to plain proxmox containers and VMs. Professionally i do work with k8s, and see the benefits of it (not always, but i see the use cases), but in my homelab it was consuming a lot of energy. Just dropping the whole k8s, made me save 1 kw energy when idle... I guess mainly because of the active API, shifting workload from different workloads and the whole machinery that happens behind the scenes..
On low power, mostly idle systems that's definitely a downside. K8s is always busy, probably largely because it's whole architecture is centered around reconciliation loops which contentiously compare the actual state to the desired state. It's not much but enough to prevent the CPU from entering its lowest power states.
Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem.
If anything, it helped keep my stuff alive during all the hardware issues a lot longer.
I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running?
All the enterprisey stuff makes it feel a lot more complex than it really is.
I believe this is the key difference, because it doesn't make sense to learn such a complicated tool for a simple use case.
If you already know it sure, it may make sense. But you are removing a huge initial effort required otherwise.
It quickly realized that after just using the managed Kubernetes from Digital Ocean and deploying a side project there.
In fact, I also have zero needs for it at work directly, as I have become an advocate for serverless and managed runtimes, unless there is really a business need to control the whole infrastructure, including the Kubernetes cluster directly.
Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn.
(My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)
IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress.
I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".
Before that though, I had a single computer (a NUC if memory serves) with systemd running docker containers. Dead simple.
Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate…
I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s
What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately.
I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.
I was hoping to move over to running rootless containers, but so far my HA setup has proven to be a pita to get working.
Even better, systemd+podman (=quadlets).
Quadlets are great, you basically can declare a compose file as a systemd unit with all the good and bad that comes with that.
So duh.
It's absolutely overkill for small teams and homelabs (I run a cluster myself) but an absolute godsend if you do need the advanced functionality.
Because that is the thing, not only it is a YAML spaghetti, everything changes depending on the puzzle pieces.
I had to follow CNCF related podcasts only to be aware of what cool projects were changing Kubernetes all the time.
Thankfully nowadays I only care about managed containers, regardless how the hyperscalers do it, it is no longer my headache.
Never needed to deal with vSphere directly.
I'm hardly trying to pretend like I'm a hyperscaler: I usually run k3s on a single node, and most of the time I admin it "clickops" style with k9s, something I find much easier than most other management tools.
I understand the dislike of YAML but a Kubernetes deployment is ~50 lines, if I had to build my own scripts with a similar feature set I don't think I would be able to get it down much more than that.
How?