Kubernetes Readiness Probe Debugging

Debug Kubernetes readiness probes in 15 minutes: read events, test endpoints, trace Service endpoints, and fix timing, path, port, and dependency errors.

A readiness failure can take every new Pod out of Service traffic while the containers remain Running. Kubernetes documents that a failed readiness probe marks a Pod unready and removes its IP from matching Service EndpointSlices; that is why a rollout can look healthy at the process level while clients receive errors.

What does a readiness probe tell Kubernetes?

A readiness probe answers one narrow question: can this container receive traffic now? It is not a general application test and it is not a replacement for monitoring. The Kubernetes Pod lifecycle documentation defines the Pod Ready condition as the signal that a Pod can serve requests and should be added to the load-balancing pools of matching Services. Readiness checks continue for the container’s whole lifecycle, so a temporary overload or lost dependency can remove a running Pod from traffic and let it return later.

A liveness probe has a different consequence. Liveness tells the kubelet that a container is stuck and should be restarted; readiness tells traffic management to stop sending requests without killing the process. Kubernetes recommends using both when you need to keep traffic away from a broken container and restart a deadlocked one. The official probe configuration guide describes these as independent checks.

The most useful timing fields are easy to confuse:

The Kubernetes Pod API reference documents defaults of 10 seconds for periodSeconds, 1 second for timeoutSeconds, 3 for failureThreshold, and 1 for successThreshold. successThreshold must be 1 for liveness and startup probes, while readiness probes can use a higher value. Treat those defaults as part of the incident: a health endpoint that regularly takes 1.2 seconds will fail a default probe even if it eventually returns a valid response.

Read the Pod event before changing the manifest

The first command should expose both the current condition and the kubelet’s explanation:

kubectl get pod -n production -l app=orders
kubectl describe pod -n production <pod-name>

The official running-Pod debugging guide notes that kubectl describe pod includes the container’s readiness, restart count, conditions, and recent events. Look at the Events section and the container’s Ready line together. Running only describes the process state; Ready: False is the traffic signal.

A typical event gives you the failure class:

Warning  Unhealthy  42s (x8 over 2m)  kubelet
Readiness probe failed: HTTP probe failed with statuscode: 503

Other messages point to different fixes:

Readiness probe failed: Get "http://10.42.3.17:8080/ready": dial tcp 10.42.3.17:8080: connect: connection refused
Readiness probe failed: command timed out
Readiness probe failed: HTTP probe failed with statuscode: 404

connection refused usually means the process is not listening on that address and port, or it has not finished binding. A timeout means the endpoint did not answer within the probe budget. A 404 means the server answered but the configured path does not exist. A 503 can be intentional when a dependency, migration, or warm-up step is incomplete.

List recent events separately when a busy namespace makes the Pod description hard to scan:

kubectl get events -n production \
  --field-selector involvedObject.name=<pod-name> \
  --sort-by=.lastTimestamp

The Kubernetes debugging guide also documents kubectl get events, kubectl logs, kubectl logs --previous, and kubectl exec as standard inspection steps. Use the previous log when a liveness failure is restarting the container; use the current log when readiness is the only failing condition.

Test the exact endpoint from the Pod

A probe can fail even when the endpoint works from your laptop. The kubelet checks the container through the Pod’s network context, so test from inside the Pod or with a temporary diagnostic container. First inspect the live spec rather than trusting the file in your editor:

kubectl get pod -n production <pod-name> \
  -o jsonpath='{.spec.containers[?(@.name=="orders")].readinessProbe}'
echo
kubectl get pod -n production <pod-name> -o yaml

For an HTTP probe, verify all five values: scheme, host, port, path, and headers. The Pod API reference says an HTTP probe defaults to the Pod IP for host and HTTP for scheme; the port can be a number or a named container port. A custom Host value belongs in httpHeaders, not in the host field when you are setting an HTTP host header.

A small, explicit readiness configuration is easier to diagnose than a probe that calls a deep business endpoint:

readinessProbe:
  httpGet:
    path: /ready
    port: http
    scheme: HTTP
    httpHeaders:
      - name: X-Health-Check
        value: kubernetes
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
  successThreshold: 1

Kubernetes considers an HTTP response successful when its status code is at least 200 and less than 400; every other status is a failed probe. That makes the 503 reference page useful when an application intentionally returns “Service Unavailable.” A readiness endpoint should report whether this replica can serve its normal traffic, not whether every optional feature in the system is perfect.

Run the same request from the container when the image has a client:

kubectl exec -n production <pod-name> -c orders -- \
  wget -S -O- --timeout=3 http://127.0.0.1:8080/ready

If the image is distroless or has no wget, use an ephemeral debug container or a short-lived diagnostic Pod. The Kubernetes debug guide documents kubectl debug when kubectl exec is insufficient because the image has no shell or diagnostic utilities.

Three readiness failures that look alike

1. The path or port is wrong

A common mismatch is a process listening on :8080 while the probe checks a named port that resolves to :8000, or an application exposing /healthz while the manifest checks /ready. Compare the container’s declared ports with the probe and with the server’s startup log:

ports:
  - name: http
    containerPort: 8080
readinessProbe:
  httpGet:
    path: /healthz
    port: http

A named port reduces accidental drift, but it does not make the name correct. Check the rendered Pod, not just the Deployment source, because a Helm value or mutating webhook may have changed the object that the kubelet runs. The Kubernetes API reference defines the HTTP probe port as either a number from 1 through 65535 or an IANA service name.

2. The check depends on something that is not required for serving

If /ready calls a database, cache, feature-flag service, and third-party API before returning 200, one optional dependency can remove every replica from traffic. That may be the right policy for a write API, but it is a poor default for an endpoint that only needs to confirm the process can accept a request. Split the signals: keep readiness focused on dependencies required for this replica’s normal request path, and expose deeper dependency health as metrics or a separate diagnostic endpoint.

This is especially visible during a rollout. The Kubernetes Deployment documentation lists readiness probe failures among the causes of a stalled rollout, alongside image-pull errors, insufficient permissions, quota, and application misconfiguration. The Deployment controller waits for new replicas to become available before it scales the old ReplicaSet down, so a false negative can leave a rollout paused while the old version still serves.

3. Startup takes longer than the probe budget

Do not solve slow startup by making liveness forgiving forever. Add a startupProbe when the application needs a bounded warm-up period, then let readiness decide when it can receive traffic and liveness detect later deadlocks. Kubernetes documents that a startup probe prevents liveness checks from killing a slow-starting container and that liveness takes over after the startup probe succeeds.

For an application that needs up to five minutes to initialize, this gives the budget explicitly:

startupProbe:
  httpGet:
    path: /startup
    port: http
  periodSeconds: 10
  failureThreshold: 30

readinessProbe:
  httpGet:
    path: /ready
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /healthz
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

The startup budget is failureThreshold × periodSeconds, or 300 seconds in this example. Once startup succeeds, readiness and liveness begin their normal work. If a restart is not warranted for a temporary dependency problem, do not point liveness at the same dependency-heavy endpoint as readiness.

Trace the failure through the Service

When Pods show Ready: False, prove whether the Service has any usable endpoints:

kubectl get svc -n production orders -o yaml
kubectl get endpointslices -n production \
  -l kubernetes.io/service-name=orders
kubectl get pods -n production -l app=orders \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,IP:.status.podIP

A Service normally selects Pods by label, and Kubernetes continuously updates its EndpointSlices from that selector. The Service documentation describes EndpointSlices as the backing network endpoints for a Service. The Service debugging guide recommends checking the selector, the selected Pods, the Service port, the targetPort, and the EndpointSlices in that order.

If ENDPOINTS is <none>, compare the Service selector with the Pod labels. If the EndpointSlice contains addresses but clients still fail, compare the Service port with targetPort, then test a selected Pod directly from a temporary Pod. A successful direct request but failed Service request moves the investigation toward DNS, NetworkPolicy, kube-proxy, or the cluster’s alternative Service implementation. Do not keep editing the readiness probe after the endpoint list proves the probe is passing.

For a Deployment rollout, watch the controller’s view as you test:

kubectl rollout status deployment/orders -n production --timeout=90s
kubectl get deployment orders -n production
kubectl get rs -n production

The kubectl rollout status reference says the command watches the latest rollout until it completes by default and returns a non-zero result if the wait fails. Pin --revision when another rollout could start while you investigate. If the new revision is clearly bad, record the event and use kubectl rollout undo deployment/orders rather than widening probe thresholds until the error disappears.

A 15-minute readiness probe checklist

1. Run kubectl get pods and identify whether the problem is Ready: False, restarts, Pending scheduling, or an image pull.
2. Run kubectl describe pod and copy the newest Unhealthy event, including path, port, status code, and timeout text.
3. Print the live readinessProbe and container ports from the Pod YAML.
4. Test the exact URL or command from inside the Pod. Match the scheme, host header, port, path, and credentials.
5. Read the application log at the event timestamp. Look for startup completion, dependency errors, bind failures, and authentication failures.
6. Check the Service selector and EndpointSlices. Confirm that the expected Pod IPs are present only when the Pods are ready.
7. If startup is slow, add a bounded startupProbe; if recovery is temporary, tune readiness with measured latency instead of copying a large timeout.
8. Watch kubectl rollout status and keep the change small enough to roll back.

For manifest review before the next deployment, run the Kubernetes YAML linter against the Deployment and Service together. Pair the probe change with the kubectl cheatsheet for the inspection commands, and use the SLO primer to decide whether probe failures should page someone or only reduce traffic.

Read next

• Run the Kubernetes YAML linter before applying a probe change.
• Use the kubectl cheatsheet for rollout and Pod inspection commands.
• Read the HTTP 503 reference when a health endpoint reports Service Unavailable.

Tags: #kubernetes, #readiness-probes, #debugging, #deployments, #sre