Kubernetes HPA Scale-to-Zero Safely

Kubernetes 1.37 makes HPA scale-to-zero practical for queue workers. Configure external metrics, preserve wake-up signals, and roll out the beta safely.

Kubernetes 1.37 turns a long-requested autoscaling pattern into a built-in option: an HPA can now take a queue worker or batch processor down to zero Pods, then bring it back when work appears. The Kubernetes 1.37 release notes list HPA scale-to-zero as beta and enabled by default. The practical change is not the number in one field. It is that the metric pipeline must keep producing a wake-up signal after the last worker disappears.

That makes this a good fit for work that can wait in a durable queue. It is a poor fit for a request-driven HTTP service whose callers expect an immediate response: a Service does not buffer requests while no Pods are ready. The official scale-to-zero announcement spells out the trade-off between idle capacity and cold-start latency. Use the feature where that trade-off is explicit, measurable, and acceptable.

What Kubernetes 1.37 changes

An HPA normally compares a metric with its target and adjusts the replica count between minReplicas and maxReplicas. With minReplicas: 0, the lower bound can now be zero when the HPA has an object or external metric. The Horizontal Pod Autoscaler documentation excludes resource metrics such as CPU and memory from this mode because those values come from running Pods. At zero, there are no containers producing those measurements.

An external metric describes demand outside the target workload: queue depth, pending jobs, or another signal exposed through the External Metrics API. An object metric describes a Kubernetes object that remains present while the workload is idle. Both give the controller a signal that survives the worker count reaching zero. A queue metric is usually the clearest starting point because it maps directly to work waiting to be processed.

The beta feature is enabled by default in 1.37, but beta does not mean “skip the operational checks.” The release’s HPA scale-to-zero guidance recommends starting the target above zero and letting the HPA perform the first downscale. A Deployment that an operator manually sets to zero is treated as paused; the controller does not assume that it should wake a workload someone intentionally stopped.

The distinction matters during incident response. A zero replica count can mean “no work,” “the HPA scaled this down,” or “someone paused it.” Kubernetes 1.37 records a ScaledToZero condition so you can tell those states apart with kubectl describe hpa. If the metric adapter is unavailable, the HPA reports an inactive scaling state such as FailedGetExternalMetric rather than inventing a wake-up value.

Make the wake-up metric reliable first

Do not begin with the HPA manifest. Begin by proving that the signal remains queryable with zero workers. The Kubernetes resource metrics pipeline documentation separates the resource metrics API from the custom and external metrics paths used by an HPA. Metrics Server is useful for CPU and memory dashboards, but it is not the queue adapter that scale-to-zero requires.

For a queue worker, expose a metric with stable identity and a clear unit. This example treats queue_consumer_lag as the number of pending tasks for one named worker:

queue_consumer_lag{namespace="default",name="worker_tasks"} 120

The metric should continue to exist when the Deployment has no Pods. That usually means the queue exporter or monitoring system owns the series, not the worker process that consumes the queue. Define what happens when the queue is empty: a value of zero should be a real sample, not a missing time series. If the series disappears, the adapter cannot distinguish “zero work” from “the monitoring path is broken.”

The Prometheus Adapter configuration guide describes the four pieces you need to review: discovering Prometheus series, associating labels with Kubernetes resources, naming the metric exposed by the API, and building the query. A minimal external rule can select the queue series and group it by worker name:

externalRules:
  - seriesQuery: '{__name__="queue_consumer_lag",name!=""}'
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (name)
    resources:
      overrides:
        namespace:
          resource: namespace

The exact adapter release and label mapping will vary by cluster. Keep the namespace association explicit. A metric that looks correct in Prometheus but cannot be selected through the External Metrics API is not ready for an HPA.

Verify the API path before you create or change the HPA:

kubectl get --raw \
  '/apis/external.metrics.k8s.io/v1beta1/namespaces/default/queue_consumer_lag?labelSelector=name%3Dworker_tasks'

The response should contain the current value for worker_tasks. Run the same check while the worker is at one replica and again after a controlled scale-down. If it fails at zero, stop there. Fix the exporter, adapter discovery, RBAC, or label selector instead of adding a second autoscaler to hide the missing signal.

Configure an HPA that can wake the worker

Once the metric path is proven, create the HPA with a bounded range and a target that matches the unit exposed by the adapter. The autoscaling/v2 API reference defines minReplicas as the lower bound and maxReplicas as the upper bound. It also notes that the HPA uses the maximum desired replica count when multiple metrics are configured.

This example requests one worker per 30 pending tasks and caps the Deployment at ten Pods:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-worker
  namespace: default
  annotations:
    kubernetes.io/description: "Scales queue-worker from the durable task queue"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: queue_consumer_lag
          selector:
            matchLabels:
              name: worker_tasks
        target:
          type: Value
          value: "30"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

The target is a queue policy, not a universal formula. Thirty small tasks may be a reasonable load for one Pod and a terrible load for another. Measure processing time, memory, downstream limits, and cold-start time before choosing it. maxReplicas is the protection against a bad queue reading or an unexpectedly large backlog turning into an unbounded rollout.

The five-minute scaleDown stabilization window is a useful starting point. The HPA API reference documents stabilizationWindowSeconds as the period in which earlier recommendations are considered; the default for scale down is 300 seconds. A queue that briefly drains should not immediately remove every worker if another burst is likely. Increase the window when cold starts are expensive, or decrease it only after you have measured the cost of idle capacity and flapping.

Keep the worker’s readiness contract honest. The HPA can decide to create a Pod, but it cannot make a slow image pull, a locked migration, or an unavailable dependency become ready. A readiness-probe configuration and a queue-side visibility timeout should agree on how long a newly started worker may take before it is trusted.

Roll it out without losing the wake-up path

Use a staged rollout rather than applying minReplicas: 0 to every worker at once.

1. Prove the metric at one replica. Record the raw External Metrics API response, the HPA status, queue depth, and the time from a new task to a Ready worker. This gives you a baseline for a later zero-to-one test.
2. Apply the HPA with minReplicas: 1. Confirm the target and selector match the intended queue. Check kubectl describe hpa queue-worker for a healthy ScalingActive condition and watch the Deployment during a small test backlog.
3. Change the lower bound to zero. Let the HPA drain the queue and perform the downscale. Do not manually run kubectl scale deployment/queue-worker --replicas=0; that creates the paused state described in the Kubernetes documentation.
4. Wake it with real work. Add a small, reversible test batch. Watch the external metric, HPA conditions, Deployment events, Pod scheduling, and queue age together. Test the full cold-start path, not only the final replica count.
5. Set an alert on the failure boundary. Alert when the queue has work but the HPA is not ScalingActive, when the metric API returns no value, when ScaledToZero=True persists while queue age rises, or when the zero-to-one time exceeds the workload’s budget.

During a control-plane upgrade, check version skew before creating new zero-enabled HPAs. The scale-to-zero announcement says that both the kube-apiserver and kube-controller-manager must support and enable HPAScaleToZero before these HPAs are created during a skewed upgrade. If you must disable the feature or roll back to a version without the condition-based behavior, first change affected HPAs to minReplicas: 1 and raise any target currently at zero.

The KEP-2021 enhancement record provides the longer history: scale-to-zero began as an alpha feature and is now in beta after work on the condition model and end-to-end coverage. That history is a reminder to keep a rollback path even though the setting is now enabled by default.

A fifteen-minute operator checklist

When evaluating one candidate workload, answer these questions in order:

1. Can the workload wait in a durable queue, or would a caller lose a request while it has zero Pods? If the latter, keep a nonzero floor or add a buffering layer first.
2. Which object or external metric remains available with no workers? Write down its name, unit, namespace, labels, owner, and empty-queue behavior.
3. Does the External Metrics API return a current value through the exact selector the HPA will use? Test it with kubectl get --raw before touching the manifest.
4. What is the cold-start budget from a new queue item to a Ready worker? Include scheduling, image pull, initialization, and dependency checks.
5. What stops runaway scale-up? Set maxReplicas, check downstream quotas, and define an alert for queue growth without replica growth.
6. What proves that zero was HPA-managed? Record the ScaledToZero condition and preserve the kubectl describe hpa output in the runbook.
7. What is the rollback command? Keep minReplicas: 1 as a reviewed change, and know how to restore a worker before disabling the feature or downgrading the control plane.

Scale-to-zero is a capacity decision, not a replacement for queue durability or observability. If the wake-up signal is independent of the Pods it starts, the new beta behavior can remove meaningful idle cost without hiding work. If that signal depends on the last Pod staying alive, keep a floor of one and fix the metric architecture before trying to save the final replica.

Read next

• Try the Kubernetes YAML linter before applying an autoscaler manifest.
• Read the readiness probe debugging guide for the worker’s startup boundary.
• Review the SLO burn-rate alerts guide for alert windows around queue age and recovery.
• See the Kubernetes 1.37 upgrade readiness checklist before a control-plane rollout.

Tags: #kubernetes, #hpa, #autoscaling, #external-metrics, #prometheus