AnsibleByExample

How to Upgrade Kubelet in Kubernetes: A Complete Guide

BeginnerKubernetesDevOpsKubeletcluster management
2024-01-015By Luca Berton

Lesson 1 of 15 in Foundations

Ansible Foundations to Production — full curriculum
  1. 1. Introduction to Ansible and Automation
  2. 2. Installing and Configuring Ansible
  3. 3. Running Ad-Hoc Commands
  4. 4. Writing Your First Playbook
  5. 5. Understanding and Using Modules
  6. 6. Variables and Facts
  7. 7. Conditionals and Loops
  8. 8. Handlers and Idempotency
  9. 9. Structuring Projects with Roles
  10. 10. Inventory Management
  11. 11. Using Ansible Vault for Secrets
  12. 12. Debugging and Troubleshooting
  13. 13. Real-World Automation Examples
  14. 14. Final Project: Deploy a Web Server
  15. 15. Bonus - What's Next with Ansible?
View path overview →

Introduction

The Kubelet is the primary node agent in Kubernetes — it runs on every node and ensures containers match their PodSpecs. Keeping Kubelet up-to-date is critical for security patches, bug fixes, and compatibility with the control plane. This guide covers the complete upgrade workflow.

Prerequisites

RequirementNotes
kubectl accessCluster admin permissions
kubeadmFor managed upgrades
SSH access to nodesFor manual kubelet restart
Maintenance windowPods will be evicted during drain

Version Skew Policy

Kubernetes enforces strict version compatibility:

ComponentAllowed Skew from API Server
kubeletUp to 2 minor versions behind
kube-proxySame minor version as kubelet
kubectl±1 minor version
kubeadmSame minor version as target

Example: If API server is v1.29, kubelet can be v1.27, v1.28, or v1.29.

# Check current versions
kubectl get nodes -o wide
# NAME    STATUS   VERSION   OS-IMAGE             KERNEL-VERSION
# node1   Ready    v1.28.4   Ubuntu 22.04.3 LTS   5.15.0-91

# Check API server version
kubectl version --short

Upgrade Workflow

Step 1: Upgrade kubeadm

# RHEL/CentOS
sudo dnf install -y kubeadm-1.29.0-0 --disableexcludes=kubernetes

# Ubuntu/Debian
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.29.0-1.1
sudo apt-mark hold kubeadm

# Verify
kubeadm version

Step 2: Drain the Node

# Evict pods and mark node unschedulable
kubectl drain node1 --ignore-daemonsets --delete-emptydir-data

# Verify no workload pods running
kubectl get pods --all-namespaces --field-selector spec.nodeName=node1

Step 3: Upgrade Kubelet Configuration

# On the node being upgraded
sudo kubeadm upgrade node

For control plane nodes (first one):

# Check available upgrades
sudo kubeadm upgrade plan

# Apply upgrade
sudo kubeadm upgrade apply v1.29.0

Step 4: Upgrade Kubelet and kubectl

# RHEL/CentOS
sudo dnf install -y kubelet-1.29.0-0 kubectl-1.29.0-0 --disableexcludes=kubernetes

# Ubuntu/Debian
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.29.0-1.1 kubectl=1.29.0-1.1
sudo apt-mark hold kubelet kubectl

Step 5: Restart Kubelet

sudo systemctl daemon-reload
sudo systemctl restart kubelet

# Verify it's running
sudo systemctl status kubelet
journalctl -u kubelet -f --no-pager | tail -20

Step 6: Uncordon the Node

# Allow scheduling again
kubectl uncordon node1

# Verify node is Ready
kubectl get nodes
# NAME    STATUS   ROLES           AGE   VERSION
# node1   Ready    control-plane   90d   v1.29.0

Upgrade All Nodes (Rolling)

#!/bin/bash
TARGET_VERSION="1.29.0"

for NODE in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
    echo "=== Upgrading $NODE ==="

    # Drain
    kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=120s

    # SSH and upgrade (adjust for your distro)
    ssh "$NODE" "
        sudo dnf install -y kubeadm-${TARGET_VERSION}-0 kubelet-${TARGET_VERSION}-0 kubectl-${TARGET_VERSION}-0 --disableexcludes=kubernetes
        sudo kubeadm upgrade node
        sudo systemctl daemon-reload
        sudo systemctl restart kubelet
    "

    # Uncordon
    kubectl uncordon "$NODE"

    # Wait for node to be Ready
    kubectl wait --for=condition=Ready "node/$NODE" --timeout=120s
    echo "=== $NODE upgraded ==="
done

Automate with Ansible

---
- name: Upgrade Kubelet on worker nodes
  hosts: k8s_workers
  serial: 1  # One node at a time
  become: true
  vars:
    k8s_version: "1.29.0"

  tasks:
    - name: Drain node
      ansible.builtin.command: >
        kubectl drain {{ inventory_hostname }}
        --ignore-daemonsets --delete-emptydir-data --timeout=120s
      delegate_to: "{{ groups['k8s_control_plane'][0] }}"

    - name: Upgrade kubeadm
      ansible.builtin.dnf:
        name: "kubeadm-{{ k8s_version }}-0"
        state: present
        disable_excludes: kubernetes

    - name: Run kubeadm upgrade
      ansible.builtin.command: kubeadm upgrade node

    - name: Upgrade kubelet and kubectl
      ansible.builtin.dnf:
        name:
          - "kubelet-{{ k8s_version }}-0"
          - "kubectl-{{ k8s_version }}-0"
        state: present
        disable_excludes: kubernetes

    - name: Restart kubelet
      ansible.builtin.systemd:
        name: kubelet
        state: restarted
        daemon_reload: true

    - name: Wait for kubelet to be ready
      ansible.builtin.command: kubectl get node {{ inventory_hostname }}
      register: node_status
      until: "'Ready' in node_status.stdout"
      retries: 30
      delay: 10
      delegate_to: "{{ groups['k8s_control_plane'][0] }}"

    - name: Uncordon node
      ansible.builtin.command: kubectl uncordon {{ inventory_hostname }}
      delegate_to: "{{ groups['k8s_control_plane'][0] }}"

Troubleshooting

Kubelet Won't Start After Upgrade

# Check logs
journalctl -u kubelet -e --no-pager | tail -50

# Common fix: reload systemd
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Node Stuck in NotReady

# Check kubelet status
sudo systemctl status kubelet

# Check certificate issues
sudo ls -la /var/lib/kubelet/pki/

# Rotate certificates if expired
sudo kubeadm certs renew all

Pods Not Rescheduling After Uncordon

# Check node conditions
kubectl describe node node1 | grep -A 5 Conditions

# Check taints
kubectl describe node node1 | grep Taints

Pre-Upgrade Checklist

  • Back up etcd: etcdctl snapshot save backup.db
  • Check version skew policy compatibility
  • Review release notes for breaking changes
  • Test upgrade on a non-production cluster first
  • Ensure PodDisruptionBudgets allow eviction
  • Verify sufficient cluster capacity during drain
  • Schedule maintenance window

Conclusion

Upgrade Kubelet following the drain → upgrade → restart → uncordon workflow. Always upgrade kubeadm first, then kubelet and kubectl. Respect the version skew policy (kubelet can be up to 2 minor versions behind the API server). Use serial: 1 in Ansible for rolling upgrades, and always back up etcd before upgrading control plane nodes.

Related reading: Ansible-driven Kubernetes operations covers this in real-world detail.

All Tutorials