Inside Kubernetes [Part 5]: Kubelet - The Node’s Brain

What's Kubelet ?!

Kubelet is a crucial component in the K8s that runs on each node in the cluster but the role of Kubelet running on the Data Plane and Control Plane differs

Data Plane (Worker Nodes): It receives the Pod Spec from the Kube API Server and runs the application Workload.

Control Plane (Master Nodes): It runs the Critical Components as the API Server, ETCD, Scheduler, Controller Manager & Cloud Controller Manager, etc, and maintains them as Static Pods.

Also, it helps manage the logs, monitoring, and metrics of the control plane.

Kubelet acts as the DaemonSet, which means it is kept running on all the Nodes, including the Control Plane and Data Plane.

Kubelet Interaction with API Server, Container Runtime

Kubelet and API Server Interaction

• Registers the node in the cluster.

• Pulls Pod specs assigned to the node.

• Reports node & Pod status back to the API server.

Kubelet & Container Runtime Interaction

• Requests container creation, deletion, and health checks.

• Monitors running containers & restarts if needed.

• Uses CRI (Container Runtime Interface) for communication

Responsibilities of Kublelet?

It is responsible for the creation of the Pods under the Node; The Scheduling responsibility lies with the scheduler and the creation of Pods lies with the Kubelet.

Container Garbage Collection

Kubelet removes the stopped or dead container to free up the disk space and checks the Max Container Age and Inactive Containers.

How does it determine max container age & inactive images?

Kubelet does not impose a strict "max-age" for containers, but it does limit the number of dead containers per pod.

Default setting is

--maximum-dead-containers-per-container=1  # Default: Keep 1 dead container per pod

So Kubelet will retain only one stopped container per pod, removing older ones automatically.

Kubelet determines which images to delete based on disk usage thresholds and the image's last used timestamp, and queries the CRI to check the images currently in use and when the image was last pulled.

It starts removing old images when disk usage exceeds a set limit:

--image-gc-high-threshold=85  # Start GC when disk usage > 85%
--image-gc-low-threshold=80   # Stop GC when usage < 80%

Also, it prioritizes the deletion based on the LRU(Least Recently Used) principle.

Pod Lifecycle management

Kubelet ensures that the Pod goes through the transition from the Creation to the Termination of the Pod Lifecycle ensuring that transitions happen correctly by monitoring, restarting, and cleaning up resources.

POD Phases

Node and Pod Status monitoring

Node and Pod status monitoring is vital to ensure the health, availability, and performance of a Kubernetes cluster. Kubelet, Controller Manager, and Scheduler continually monitor and update Node and Pod status.

Kubelet also keeps Node Conditions up to date, which decide node health based on conditions such as MemoryPressure (low memory), DiskPressure (running low on disk space), PIDPressure (excessive processes running on node), Ready (Node is ready to accept new Pods), and NetworkUnavailable (Node with network problems).

Pod Security & Authentication

Kubelet enforces Security Contexts and defines how a Pod or a container should run securely.

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsUser: 1000    # Run as non-root user
    runAsGroup: 3000
    fsGroup: 2000      # Group ownership of files
  containers:
    - name: app
      image: nginx
      securityContext:
        allowPrivilegeEscalation: false   # Prevent privilege escalation
        capabilities:
          drop:
            - ALL   # Remove all Linux capabilities

Kubernetes 1.25+ enforces Pod Security Standards (PSS) at the namespace level using Pod Security Admission (PSA).

Eviction and Resource Pressure Handling in Kubelet

When resources like CPU, memory, disk, or inodes become critically low, Kubelet evicts Pods to free up space and prevent node failures.

What is Eviction in Kubernetes?

Eviction is the action where the Kubelet terminates Pods forcefully when resource utilization surpasses specified thresholds.

It helps in the prevention of OOM(Out of Memory) Crashes, Maintain Cluster Stability

It's divided into two types Soft Eviction (with Graceful Shutdown) and Hard Eviction (Immediate Termination).

Soft Evictions

They are invoked when the resource consumption crosses a defined threshold for a specific period

# Eviction starts if available memory drops below 500Mi for 1 minute and 30 seconds.
eviction-soft:
  memory.available: "500Mi"
eviction-soft-grace-period:
  memory.available: "1m30s"

Hard Evictions

They are triggered when the resource availability drops below critical levels.

# If available memory drops below 100Mi, Kubelet immediately evicts pods.
eviction-hard:
  memory.available: "100Mi"
  nodefs.available: "10%"

While evicting the pods there are certain priorities that the Kubelet follows based on the Quality of Service(QoS) and Priority Class

• BestEffort pods (no resource requests/limits) are evicted first.

• Burstable pods (some resource requests but no strict limits) are evicted next.

• Guaranteed pods (requests == limits for CPU & memory) are evicted last.

• Priority Classes (if defined) can override QoS-based eviction.

Is Eviction different than preemption?

• Eviction (managed by Kubelet) releases node resources reactively.

• Preemption (managed by Scheduler) creates space for higher-priority pods by killing lower-priority ones.

Congratulations you made it to the last !! Stay ahead; Subscribe to the EzyInfra Knowledge Base for more DevOps wisdom.

Conclusion

Kubelet is the engine that keeps Kubernetes running. It pulls Pod specs, manages containers, monitors node health, and enforces security—all while ensuring smooth communication between the API server, container runtime, and CNI. Without Kubelet, Kubernetes wouldn’t function. Mastering it means mastering Kubernetes itself!

EzyInfra.dev – Expert DevOps & Infrastructure consulting! We help you set up, optimize, and manage cloud (AWS, GCP) and Kubernetes infrastructure—efficiently and cost-effectively. Need a strategy? Get a free consultation now!