Practical Anomaly Detection with Prometheus and PromQL

On-call engineers know the drill: pager at 3 AM, CPU is at 95%, and you wonder — “Is this normal for the time of day, or is something really wrong?” Hard thresholds often miss context. Instead, use PromQL’s native functions to establish statistical baselines and flag deviations. In Prometheus you can compute a moving average and standard deviation, then treat anything beyond “mean ± N·σ” as an anomaly. In plain terms, we compute a Z-score: (current — μ) / σ . For example, this PromQL calculates a Z-score on HTTP requests per minute vs. the last 15 minutes of data:

(
  rate(http_requests_total[1m])
  - avg_over_time(rate(http_requests_total[1m])[15m])
)
/
stddev_over_time(rate(http_requests_total[1m])[15m])

Values beyond roughly ±3 indicate an outlier . You would then alert when that expression exceeds a threshold (e.g. >3) for a sustained period (e.g. for:5m) to avoid flapping .

CPU Usage Spikes

A common scenario is detecting abnormal CPU usage. For example, use the node-exporter idle-CPU metric: if idle time suddenly drops far below its normal range, CPU is spiking. A rule might compare the 5-minute idle average to its historical baseline (say 1 hour or 1 day) and divide by the long-term stddev. For instance:

# Alert if idle CPU is significantly below its 1h baseline (i.e. CPU busy)
(
  avg_over_time(node_cpu_seconds_total{mode="idle"}[5m])
  - avg_over_time(node_cpu_seconds_total{mode="idle"}[1h])
)
/
stddev_over_time(node_cpu_seconds_total{mode="idle"}[1h])
> 3

This fires when the 5m idle-average is 3σ below the 1h mean. (Low idle => high usage.) Such “3-sigma” thresholds work well: they adapt as the machine warms up or loads vary. The query above uses avg_over_time and stddev_over_time exactly as recommended for baseline alerting . In practice, you’d replace 1h with a window that fits your environment (4h, 24h, etc.) and tune the multiplier (2–3σ is common ).

Latency Regression Detection

Likewise, you can catch service latency regressions by comparing current percentiles to a baseline. For example, take the 95th-percentile HTTP latency over the last 5 minutes and compare it to its 1h moving average:

# Alert if P95 latency is far above its 1h baseline
(
  histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
  - avg_over_time(histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))[1h])
)
/
stddev_over_time(histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))[1h])
> 4

This triggers when current P95 latency exceeds the recent mean by 4σ (adjust σ-threshold as needed). (Feel free to use quantile(0.95, sum(rate(…))) instead if you prefer that syntax.) It’s often helpful to smooth percentile-based signals first or use offset baselines for known patterns: for daily or weekly traffic cycles, compare against the same time yesterday or last week . For example, you might compute “expected” latency = last week’s value plus any growth trend , then alert if (now — expected) / stddev is high.

Error-Rate Floods

Error floods (e.g. sudden surge in HTTP 5xx) can be caught similarly. Suppose you have a counter of errors per service. You could alert if the short-term error rate jumps far above its baseline. For instance:

# Alert if error rate is anomalously high
(
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
  - avg_over_time(sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)[1h])
)
/
stddev_over_time(sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)[1h])
> 3

This raises an alert when 5xx errors are 3σ above their hour-average norm. (GitLab’s team used a similar approach for request rates .) Keep in mind that error counts can be bursty. If your error metric isn’t roughly Gaussian, you might prefer a threshold or percentile approach instead (error and latency distributions are often heavy-tailed ). But in many cases a Z-score rule will catch true floods while suppressing normal jitter.

Refinements: Smoothing, Margins, Offsets

Basic Z-score alerts can be noisy if used “raw.” A few battle-tested refinements help:

• Smoothing STD. A single spike can inflate σ and widen the alert band too much. To smooth this, compute the short-window σ and then average it over a longer range. For example, record stddev_5m = stddev_over_time(metric[5m]), then std_smoothed = avg_over_time(stddev_5m[24h]). This low-pass filters σ . Grafana’s SREs did exactly this: take 1h σ windows and average them over ~26h , which stabilized their bands.
• Minimum margin. If a metric’s normal variation is tiny, even a small blip can exceed 2σ. A “margin band” or minimum threshold ensures alerts aren’t fired for trivial changes. For example, require that the anomaly exceed either N·σ or some percentage of the mean. One trick is to add a fixed multiplier of the mean as a floor: e.g. set upper = avg + max(σ*2, avg*0.1). This enforces at least a 10% swing before alerting . (In practice you may tune this on a per-metric basis.)
• Offset for seasonality. Many services have strong daily/weekly cycles. PromQL’s offset lets you compare “now” to “24h ago”. For example:

current = avg_over_time(request_rate[5m])
past = avg_over_time(request_rate[5m] offset 1d)
anomaly_z = (current - past) / stddev_over_time(request_rate[1d])

• This catches if the metric right now deviates from yesterday’s same-minute value by too much. In more complex setups (like GitLab’s) teams combine several weeks of history via multiple offsets and averages to predict an expected value . But even a simple offset vs. sliding-window compare can greatly reduce false alarms on periodic metrics.
• “For” and bucketing. Always use a for: clause (e.g. 2–5m) so transient spikes don’t immediately alert. Also, choose aggregation judiciously: group by the service (job) or host level, not by tiny dimensions. As GitLab notes, the “right” aggregation is usually the service level (keep job and environment, drop high-cardinality labels) . Too coarse and you might miss localized anomalies; too fine and you’ll alert on noise .

Each alert rule should be tested with real traffic patterns. Simulate an error spike or latency regression and verify the Z-score swings as expected. When well-tuned, these rules give on-call responders context (“Yes, 95% latency is 3σ above normal” ) instead of blind panic.

Checklist: Integrating Anomaly Rules

• Choose good aggregation: Group metrics at a stable level (service or host). Avoid breaking out tiny subsets unless necessary .
• Define baseline windows: Select short and long windows (e.g. 5m vs 1h, or 1h vs 24h) that capture your system’s normal cycle.
• Compute mean and σ: In PromQL, use avg_over_time(…[long]) and stddev_over_time(…[long]) (or recording rules for efficiency). For efficiency, pre-record baselines with recording rules, if your Prometheus setup allows.
• Set Z-score threshold: A common starting point is 3σ, meaning ~99.7% of normal data falls inside. You can lower to 2σ for more sensitivity, but expect more noise. Also tune the for: duration to require sustained deviation.
• Add buffers: Implement smoothing (e.g. averaging stddev) and/or minimum margin to prevent small jitters from alerting . Include an offset- or seasonality-based check for known patterns.
• Test and iterate: Run these PromQL queries in Grafana or Prometheus console first. Inject synthetic load or examine historical data to ensure they fire on real incidents (and stay quiet during normal peaks).
• Document context: Label the alert with context (expected value, baseline, etc.) in the annotation. When the alert fires, your runbook should note “this uses a moving-average baseline over the past 1h and ±3σ”.

By following these steps, your team can move beyond brittle static thresholds. PromQL’s native functions let you codify “this is normal behavior” and have alerts only when something truly unusual happens. The payoff is fewer false alarms in the night and faster detection of the critical anomalies that actually demand action .