Kured: Restart your Azure Kubernetes Service Nodes

by Carlos Mendible on 28 Jul 2019 » Azure, kubernetes, DevOps

Two weeks ago I got an email message from Microsoft Azure explaining that Azure Kubernetes Services had been patched but that I had to restart my nodes (reboot the clusters) to complete the operation.

The Microsoft Azure email explaining that I had to reboot my clusters

The first thing you need to know is that, when things like this happens, the Azure platform creates a file called /var/run/reboot-required in each of the nodes of your cluster.

The second thing is that a Kubernetes Reboot Daemon named Kured exists and if installed in your cluster will run on each pod watching for the existence of the /var/run/reboot-required file. Kured then takes care of the reboots for you so only one node is restarted at a time.

So how do you install it and check what’s going on with your nodes? Let’s see:

1. Check the nodes


Damn I was running the affected kernel version: 4.15.0-1049-azure.

kubectl get nodes -o wide
NAME                       STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-agentpool-14502547-0   Ready     agent     3d        v1.12.8   <none>        Ubuntu 16.04.6 LTS   4.15.0-1049-azure   docker://3.0.4
aks-agentpool-14502547-1   Ready     agent     3d        v1.12.8   <none>        Ubuntu 16.04.6 LTS   4.15.0-1049-azure   docker://3.0.4
aks-agentpool-14502547-2   Ready     agent     3d        v1.12.8   <none>        Ubuntu 16.04.6 LTS   4.15.0-1049-azure   docker://3.0.4

2. Install Kured


kubectl apply -f https://github.com/weaveworks/kured/releases/download/1.2.0/kured-1.2.0-dockerhub.yaml

3. Check the nodes again


kubectl get nodes -o wide

You can se that the first two nodes already restarted and the second has scheduling disabled cause it was still in the process.

aks-agentpool-14502547-0   Ready                      agent     3d        v1.12.8   <none>        Ubuntu 16.04.6 LTS   4.15.0-1049-azure   docker://3.0.4
aks-agentpool-14502547-1   Ready                      agent     3d        v1.12.8   <none>        Ubuntu 16.04.6 LTS   4.15.0-1050-azure   docker://3.0.4
aks-agentpool-14502547-2   Ready,SchedulingDisabled   agent     3d        v1.12.8   <none>        Ubuntu 16.04.6 LTS   4.15.0-1050-azure   docker://3.0.4

To learn more about Kured and the features it offers, please check here.

Hope it helps!