Readiness/Liveliness Probe on CPU % Utilisation of a java process in POD in K8s

For the case when the java application is not quite stable and the java process gets blocked due to deadlock or blocked threads and continuously keeps on consuming High CPU, which can cause new API calls to fail.

This post is only for those requirements where sometimes the only solution we have is to restart the container/stop any new traffic if CPU utilisation of the java process is above a limit for a specific period of time.

The easiest way to implement this is the liveliness and readiness probe.

In the simplest terms, Liveliness Probe’s function is to restart the container if the condition in probe fails.

And Readiness Probe’s function is to stop any new incoming traffic to the container if the condition in probe fails.

Implementation

Below is a deployment.yaml example for deploying xyz container with Liveliness Probe enables on the CPU utilisation of the main java process.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xyz-deployment
  labels:
    app: xyz
spec:
  replicas: 3
  selector:
    matchLabels:
      app: xyz
  template:
    metadata:
      labels:
        app: xyz
    spec:
      containers:
      - name: xyz
        image: xyz:1.14.2
        ports:
        - containerPort: 80
        env:
        - name: CPU_LIMIT
          valueFrom:
            resourceFieldRef:
              containerName: xyz
              divisor: 1m
              resource: limits.cpu
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - cpu=$(echo "$(top -bn1 -w512 | grep "^[[:blank:]]*$(pgrep java)"  | awk '{print $9}')*1024/$CPU_LIMIT" |bc); cpu=$(echo $cpu| sed 's/ //g') && if (( $(echo "$cpu < 85" |bc  ) )); then exit 0; else exit 1; fi
          failureThreshold: 18
          initialDelaySeconds: 300
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10              

in env of a container, CPU_LIMIT gets the limit CPU resource for the xyz container

        - name: CPU_LIMIT
          valueFrom:
            resourceFieldRef:
              containerName: xyz
              divisor: 1m
              resource: limits.cpu

The command uses top and bc command to get CPU utilisation of java process and do arithmatic operations, do ensure that the container already has bc and top installed.

cpu=$(echo "$(top -bn1 -w512 | grep "^[[:blank:]]*$(pgrep java)"  |
              awk '{print $9}')*1024/$CPU_LIMIT" |bc); cpu=$(echo $cpu| sed 's/ //g') && if (( $(echo "$cpu < 85" |bc  ) )); then exit 0; else exit 1; fi