Upgrade Kubernetes from 1.23 to 1.24

Posted on 22 May 2023, tagged KubernetescontainerDockerLinux

In the last blog post, I introduced using Kubernetes to setup high available infrastructure. I had that setup a long time ago. I did the long overdue upgrade for Kubernetes from 1.23 to 1.24 recently. Since GlusterFS is deprecated(though not removed) in 1.25, I have no plans to continue the upgrade without exploring alternative storage options.

There is a big change from 1.23 to 1.24 as well, namely, Docker Engine support has been removed. I migrated the container engine to containerd. But the process is not without pain. I need to search different sources to fix the issues. So I list my upgrade steps so that if anyone has the same issue, this may help.

My Kubernetes cluster is set up locally with kubeadm. There is an official upgrade guide for kubeadm to upgrade from 1.23 to 1.24, but it doesn’t mention any steps to remove Docker and setup containerd. So here are the steps I took:

  1. Add --container-runtime-endpoint option to kubelet. The way I did it is adding KUBELET_ARGS="--container-runtime-endpoint=/run/containerd/containerd.sock" to /etc/kubernetes/kublet.env. Without this, Kubelet will fail to start.
  2. Remove --network-plugin=cni from /var/lib/kubelet/kubeadm-flags.env.
  3. Add the following configuration in /etc/crictl.yaml, otherwise kubeadm will not be able to pull needed images: 
    1runtime-endpoint: unix:///run/containerd/containerd.sock
2image-endpoint: unix:///run/containerd/containerd.sock
3timeout: 10
4debug: false
  4. Configure SystemdCgroup permission for containerd. Otherwise kube-apiserver will always be restarted because of “sandbox environment changes” (see more in Github issue): 
    1sudo mkdir -p /etc/containerd/
2containerd config default | sudo tee /etc/containerd/config.toml
3sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
4sudo systemctl restart containerd
  5. Follow the official upgrade guide.
  6. After the upgrade, remember to restart Docker so that the old containers started by Docker will be stopped.
comments powered by Disqus