SLO Burn-Rate Alerts with PromQL

Build SLO burn-rate alerts with PromQL: calculate error ratios, choose multiwindow thresholds, prevent flapping, and test the policy before paging.

An SLO alert should answer one operational question: is the service spending its reliability budget quickly enough that a human needs to act now? A raw error-rate threshold cannot answer that on its own. A 0.2% error rate is catastrophic for a 99.99% target and unremarkable for a 99% target.

The useful unit is burn rate: the speed at which the service consumes its error budget compared with the speed that would exhaust the budget exactly at the end of the SLO window. Google’s SRE Workbook recommends multiwindow, multi-burn-rate alerts because they combine a long window that proves meaningful budget consumption with a short window that proves the problem is still active.

This guide builds the pattern for a request-based service in Prometheus. The examples use HTTP availability, but the same shape works for latency, freshness, and other SLIs that can be expressed as good events divided by valid events. The goal is a small set of alerts that an on-call engineer can understand, test, and attach to a written error-budget policy.

Start with an SLI and a budget you can defend

An SLI is the measured level of service; an SLO is the target for that measurement over a defined period. For request availability, define the SLI as successful requests divided by all valid requests. Google’s SLO implementation guidance recommends this ratio because it stays between 0% and 100% and maps directly to an error budget.

Suppose the checkout API has this objective:

SLO: 99.9% of valid requests succeed
Window: rolling 30 days
Error budget: 0.1% of valid requests

The budget is not 0.1% of time unless the SLO itself measures uptime. For a request-based SLO, it is 0.1% of valid requests. If the service receives 3,000,000 valid requests in the window, the budget is 3,000 failed requests. The SRE Workbook’s example SLO document shows why the denominator and the failure classification belong in the SLO definition, not in an undocumented dashboard query.

Write down what is valid before writing PromQL:

1. Count user requests that reached the service or load balancer.
2. Exclude synthetic checks only if the SLO explicitly excludes them.
3. Decide whether 4xx responses are user errors or service failures.
4. Include dependency failures when the user experiences them as a failed request.
5. Keep the same classification in the numerator, denominator, dashboard, and alert.

That last rule prevents a common failure mode: an alert says the service is healthy because the application counted only responses that made it through a proxy, while users were receiving gateway errors before the application ran. If the request boundary is outside the application, instrument or collect the SLI there. For new HTTP instrumentation, the OpenTelemetry HTTP metric conventions define http.server.request.duration as a recommended histogram in seconds and include http.response.status_code as a conditional attribute.

Calculate the error ratio in PromQL

For a Prometheus counter named http_requests_total, calculate the error ratio by taking the rate of each counter first, then aggregating:

sum(
  rate(http_requests_total{
    service="checkout",
    status=~"5.."
  }[5m])
)
/
sum(
  rate(http_requests_total{
    service="checkout"
  }[5m])
)

This expression returns a fraction: 0.001 means 0.1% of valid requests were classified as errors during the five-minute lookback. Multiply by 100 only for a human-facing percentage; burn-rate comparisons are clearer when both sides remain fractions.

The order matters. Prometheus’s query guidance says “rate then sum, never sum then rate” for counters because a restart can reset one series independently of the others. The PromQL rate() documentation also explains that rate() adjusts for counter resets and estimates a per-second average over the selected range.

For production rules, put this calculation in recording rules instead of repeating a large query in every alert. Recording rules also give dashboards and incident responders a stable name:

groups:
- name: checkout.slo
  interval: 30s
  rules:
  - record: service:slo_errors_per_request:ratio_rate5m
    expr: |
      sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="checkout"}[5m]))

Keep the service identity in labels rather than hard-coding it into a fleet of nearly identical rule files. If you aggregate by service, preserve the labels that identify the owning team and environment. Prometheus’s recording-rule guidance recommends naming rules by aggregation level, metric, and operations, and warns against averaging ratios: aggregate numerators and denominators separately, then divide.

For latency, use a threshold ratio rather than averaging percentiles. If “good” means a request finishes within 500 ms, the numerator can be the le="0.5" bucket and the denominator can be the histogram count. The Prometheus histogram documentation explains why a ratio of counts composes better than an average of per-instance quantiles.

Turn the error ratio into burn rate

Burn rate is the observed error ratio divided by the error-budget fraction. For a 99.9% SLO:

error budget fraction = 1 - 0.999 = 0.001
burn rate              = observed error ratio / 0.001

If the service has a 0.6% error ratio, its burn rate is 0.006 / 0.001 = 6. At a constant 6x burn rate, the service would consume a 30-day budget in about five days. A burn rate of 1 consumes the whole budget at exactly the expected pace; it does not mean the service is error-free.

Google’s starting thresholds for a 99.9% objective are useful defaults, not laws:

The short window is one-twelfth of the long window. Requiring both windows to exceed the threshold makes a page prove two things: the incident has consumed enough budget to matter, and the error rate is still elevated now. The short window also lets an alert resolve soon after recovery instead of remaining active for the entire long window. These are the parameters in Google’s multiwindow alerting guidance; tune them after observing traffic volume and on-call load.

Create a recording rule for the budget and for each burn window:

groups:
- name: checkout.slo
  interval: 30s
  rules:
  - record: service:slo_error_budget:ratio
    expr: 1 - 0.999

  - record: service:slo_burn_rate:ratio_rate1h
    expr: |
      service:slo_errors_per_request:ratio_rate1h{service="checkout"}
      /
      service:slo_error_budget:ratio

  - record: service:slo_burn_rate:ratio_rate5m
    expr: |
      service:slo_errors_per_request:ratio_rate5m{service="checkout"}
      /
      service:slo_error_budget:ratio

In a real ruleset, define service:slo_errors_per_request:ratio_rate1h and service:slo_errors_per_request:ratio_rate5m from the same numerator and denominator pattern used for the five-minute example. Keep the metric names explicit so an incident responder can inspect the raw ratio, budget fraction, and burn rate separately.

Require both windows before paging

The core page rule is an AND between the long and short windows:

- alert: CheckoutSLOBurnRatePageFast
  expr: |
    (
      service:slo_burn_rate:ratio_rate1h{service="checkout"} > 14.4
    )
    and
    (
      service:slo_burn_rate:ratio_rate5m{service="checkout"} > 14.4
    )
  for: 2m
  labels:
    severity: page
    service: checkout
  annotations:
    summary: "Checkout is burning its SLO budget quickly"
    description: "The checkout API has exceeded the 14.4x burn-rate threshold over 1h and 5m."
    runbook: "/posts/slo-burn-rate-alerts/"

The and operator is doing practical work here. A single five-minute spike can exceed 14.4x without meaning that a significant fraction of the monthly budget has been consumed. A long-window breach with a recovered short window describes a past incident, not necessarily a page-worthy active incident.

Prometheus’s alerting-rule documentation says an alert with for: 2m remains pending until its expression stays active through the configured duration. That is different from the SLO short window: for protects against evaluation noise and missing scrapes, while the short window confirms recent user impact. Do not use for: 1h as a substitute for the 1-hour burn window; it delays detection without checking the intended budget math.

Add the slower page and ticket rules separately:

- alert: CheckoutSLOBurnRatePageSlow
  expr: |
    (
      service:slo_burn_rate:ratio_rate6h{service="checkout"} > 6
    )
    and
    (
      service:slo_burn_rate:ratio_rate30m{service="checkout"} > 6
    )
  for: 5m
  labels:
    severity: page
    service: checkout

- alert: CheckoutSLOBurnRateTicket
  expr: |
    (
      service:slo_burn_rate:ratio_rate3d{service="checkout"} > 1
    )
    and
    (
      service:slo_burn_rate:ratio_rate6h{service="checkout"} > 1
    )
  for: 15m
  labels:
    severity: ticket
    service: checkout

Route the page to the on-call path and the ticket to a queue with a useful deadline. Prometheus evaluates the rules; Alertmanager handles grouping, inhibition, silencing, and notification delivery. If a fast outage satisfies all three rules, use routing or inhibition so one incident does not produce three unrelated pages.

If you prefer a generator, Sloth’s architecture is a useful reference: it turns one SLO specification into SLI, metadata, and multiwindow alerting rules, while Prometheus evaluates those rules and Alertmanager sends notifications. A generator can standardize rule shape, but it cannot decide whether your denominator represents the user.

Account for low traffic and alert behavior

Burn rate is sensitive to small denominators. If a service receives 10 requests in an hour and one fails, the hourly error ratio is 10%. For a 99.9% objective, that is a 100x burn rate even though the incident may not justify waking an engineer. Google’s low-traffic alerting guidance recommends considering artificial traffic, grouping services carefully, changing the unit of failure, or using a longer window when the signal is too sparse.

Do not solve low traffic by hiding real failures. Synthetic traffic can miss user-specific paths, and grouping services can make one completely broken service disappear inside a large aggregate. Record the choice in the SLO document and review it against real incidents.

When a queue worker scales down during quiet periods, pair the alert with a wake-up signal that survives zero replicas. The Kubernetes HPA scale-to-zero guide shows how to verify an external metric before relying on it for recovery.

Use keep_firing_for when your scrape or remote-write path can briefly lose data. Prometheus documents this option as a way to prevent flapping and false resolutions after an alert is firing. It is not a replacement for fixing missing telemetry: a long retention period can make a recovered service look unhealthy to the person receiving the page.

Give every page an owner, a runbook, and one immediate action. Prometheus’s alerting practice separates rule evaluation from notification management; the runbook should bridge the alert to the dashboards and logs that identify the cause. A burn-rate page is a symptom alert. Keep cause alerts, such as a single pod restart or a disk threshold, out of the paging path unless they independently require immediate action.

Test the math before you wake anyone

Run these checks before enabling a page:

1. Steady state: feed a ratio below the budget fraction and confirm every burn-rate series stays below 1.
2. Fast outage: inject enough errors to exceed both the 1-hour and 5-minute 14.4x thresholds; confirm the page becomes firing after the for duration.
3. Short spike: exceed the five-minute threshold only; confirm the long-window AND prevents a page.
4. Recovered incident: stop the errors and verify the short window clears promptly.
5. Slow burn: hold the ratio above the 3-day ticket threshold; confirm the ticket rule fires without a page.
6. Counter reset: restart an instrumented process and verify rate() does not create an impossible negative or giant ratio.
7. Missing data: stop the exporter and confirm your no-data policy is explicit rather than silently treated as success.

Prometheus provides promtool test rules for rule tests. Test both the expression result and the alert state at the times that matter. A green YAML parse proves only that the file is syntactically valid; it does not prove that your denominator, labels, windows, and severity match the policy.

Finally, write the policy beside the rules. Google’s error-budget policy example treats a 99.9% SLO as a 0.1% budget and calls for halting nonessential changes when the preceding rolling window has exhausted that budget, with explicit exceptions for urgent work. Your team may choose different actions, but the owner, trigger, exception path, and review date should be written before the first incident.

Read next

• Review the SLO, SLI, and error-budget primer.
• Inspect response classes with the HTTP status code reference.
• Check the HTTP 503 reference when an upstream outage is part of the error ratio.
• Use the HTTP header inspector when the edge and application disagree about response behavior.

Tags: #sre, #slo, #prometheus, #promql, #alerting