One of the core features of Kubernetes (or of any other container orchestration solution) is the ability to perform health checks against running containers to determine the health of the container instance. Container instances can experience performance-related issues, connectivity errors, or application bugs that can cause the instance to be unhealthy. Having health checks in place can help restore unhealthy containers quickly in an automated fashion. Kubernetes has a built-in mechanism within the kubelet process that ensures containers are running and healthy based on the Liveness and Readiness probes defined within the pod manifest. Unique Liveness and Readiness probes can be configured for each specific container within a Pod, which is discussed in greater detail below.

Readiness Probe

A readiness probe is used by the kubelet process to identify whether a container instance is ready to accept traffic or not. When all of the containers within a Pod pass the health checks defined within the readiness probe, then the Pod is marked as ready. Until this happens, traffic will not be routed to the Kubernetes Services running on that pod. For pods that don’t have any readiness probe defined, the default state of the Pod is “Success,” meaning that workloads can be routed to the container instances within that pod with no validation whether they will be able to handle that workload or not. Readiness probes are particularly critical for applications that take time to initialize and warm up before they can start servicing requests.

Liveness Probe

A Liveness Probe is similar to a readiness probe, but rather than running during pod startup, it periodically checks if the container is still able to handle workloads by performing the health checks defined during pod operation. In the event of health check failures, the kubelet will kill and restart the container based on the restart policy defined. This probe is particularly helpful when an application is unresponsive but the pod itself has not terminated the health checks can help detect unhealthy containers and restart them proactively as needed.

Configuring Probes

Probes have various attributes that can be used to perform and control the health checks on the containers.

Kubernetes - Container Health Checks - configuring probes

Types of Checks used by Probes

Probes can perform different types of checks, including HTTP, TCP, and Command execution checks in order to verify the health of the container.

HTTP Checks

The Kubelet process uses webhooks to perform HTTP GET calls against the containers to verify health; for this type of check, any HTTP response code between 200 and 399 is deemed a successful response. The HTTP Check is helpful when the application can return an HTTP status code upon successful initialization, or when it is functionally able to service requests.

Example: HTTP Check

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 10

TCP Checks

The Kubelet process attempts to establish a TCP connection with the container and marks it healthy upon successfully connecting.

Example: TCP Check

          livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 10

Command Execution Checks

          livenessProbe:
            exec:
              command: 
              - cat 
              - /tmp/ping
            initialDelaySeconds: 15
            periodSeconds: 5
            timeoutSeconds: 1
            failureThreshold: 10

The Kubelet process executes a command inside the container instance; if the command succeeds with a return code 0 the container is considered healthy.

Ex: Command Execution Check

Sample Probe Deployment

Here is a sample Kubernetes deployment manifest with readiness and liveness probes defined leveraging HTTP checks for an nginx container instance.

sample-probe.yml

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: probe-sample
  labels:
    app: nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 10
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 10

Create Sample Deployment

1. Use kubectl to create the nginx deployment using sample-probe.yml file.

Kubernetes - Container Health Checks - Create Sample Deployment 1

2. Monitor the status of the nginx pods created.

Kubernetes - Container Health Checks - Create Sample Deployment 2

In the above output “STATUS” column displays “Running” status, which indicates the container is Running. The “Ready” column showing 0/1 as the Liveness and Readiness checks are not triggered yet due to the initialDelay set to 30 seconds.

A Pod is healthy when its status is “Running” and marked “Ready” after successful Liveness/Readiness checks. “Ready” column with 1/1 value means the Pod has only one container and its Ready.

Kubernetes - Container Health Checks - Create Sample Deployment 3

If the container is unhealthy and fails the health check, Kubernetes tries to restart the pod. In case of any restarts, the restart count will be displayed under the “RESTARTS” column.

3. Describe the pod to view the probe settings and events. In case of any probe failures, they will be listed under the “Events” section.

Kubernetes - Container Health Checks.- Create Sample Deployment 5

Conclusion

One of the key features of the Kubernetes orchestration platform is its ability to execute health checks to provide better reliability and uptime for containerized applications. Having readiness and liveness probes in place allows Kubernetes to detect and mitigate issues with individual container instances automatically with no external intervention, providing self-healing capabilities to the container orchestration environment.

If you have questions on how you can best leverage Kubernetes, or are looking for help with your Kubernetes-based implementation, please engage with us via comments on this blog post, or reach out to us here.

Additional Reading

You can also continue to explore Kubernetes by checking out Kubernetes, OpenShift, and the Cloud Native Enterprise blog post, or Demystifying Docker and Kubernetes. You can reach out to us to plan for your Kubernetes implementation or AWS related posts such as DNS Forwarding in Route 53.