Prometheus and PromQL

• Prometheus for metrics
• Loki or Elasticsearch for logs • Tempo or Jaeger for traces • Grafana as a unified interface

Prometheus is a monitoring system built around time-series data. It periodically scrapes metrics from applications, stores the samples locally, and provides PromQL (Prometheus Query Language) for querying them. Grafana can use Prometheus as a data source and turn the same queries into dashboards.

This article explains the Prometheus data model and the PromQL patterns needed for day-to-day monitoring.

How Prometheus Works #

A typical Prometheus setup contains:

1. Applications or exporters that expose metrics over HTTP, usually at /metrics.
2. A Prometheus server that discovers targets and scrapes them at a fixed interval.
3. Rules that precompute frequently used queries or evaluate alert conditions.
4. Alertmanager, which groups and routes alerts to notification systems.
5. Grafana, which queries Prometheus and visualizes the results.

Prometheus uses a pull model: the server asks each target for its current metrics. This makes target health easy to inspect. If Prometheus cannot scrape a target, the automatically generated up metric becomes 0 for that target.

up == 0

The scrape model also means Prometheus is not a general-purpose event store. Metrics should describe numeric state that can be sampled repeatedly. Logs and individual request records belong in log or tracing systems.

The Prometheus Data Model #

A time series is identified by a metric name and a set of labels:

http_requests_total{
  method="GET",
  route="/api/users",
  status="200",
  instance="10.0.0.8:8080",
  job="api"
}

Each unique combination of labels is a separate time series. Prometheus stores timestamped numeric samples for that series.

Labels make metrics flexible, but every additional label value increases the number of series. Labels such as user_id, request IDs, email addresses, and unbounded URLs create excessive cardinality and should not be used. Prefer bounded dimensions such as HTTP method, normalized route, status code, region, and service name.

Prometheus metrics usually follow one of four types:

• Counter: A cumulative value that normally only increases, such as http_requests_total. Use rate() or increase() rather than graphing the raw value.
• Gauge: A value that can increase or decrease, such as memory usage, queue length, or temperature.
• Histogram: Observations divided into buckets, plus a count and sum. It is suitable for request duration and response size.
• Summary: Client-side calculated quantiles, plus a count and sum. Its quantiles generally cannot be aggregated across instances.

Metric names should include units and use base units where practical, for example http_request_duration_seconds rather than http_request_duration_milliseconds.

PromQL Result Types #

PromQL expressions operate on four result types:

• Instant vector: One current sample from each matching time series.
• Range vector: A time window of samples from each matching time series.
• Scalar: A single numeric value.
• String: A string value, which is rarely used.

An instant vector selector returns the latest sample for every matching series:

node_memory_MemAvailable_bytes

A range selector returns samples from a time window:

http_requests_total[5m]

A range vector cannot be graphed directly. It is normally passed to a function such as rate(), increase(), avg_over_time(), or max_over_time().

Selecting and Filtering Series #

Use label matchers inside braces to narrow a selector:

http_requests_total{job="api", method="GET"}

PromQL supports four label matchers:

http_requests_total{status="200"}       # Equal
http_requests_total{status!="200"}      # Not equal
http_requests_total{status=~"2..|3.."}  # Regular expression match
http_requests_total{status!~"4..|5.."}  # Negative regular expression match

Regular expressions are fully anchored. The matcher route=~"/api/.*" matches the complete label value, not an arbitrary substring.

An offset queries data from an earlier time:

rate(http_requests_total[5m] offset 1d)

It is useful for comparing the current rate with the same period yesterday:

sum(rate(http_requests_total[5m]))
/
sum(rate(http_requests_total[5m] offset 1d))

Working with Counters #

Never use a raw counter to represent activity during a period. The raw value depends on process age and resets when the process restarts.

rate() calculates the average per-second increase over a range and adjusts for counter resets:

rate(http_requests_total[5m])

increase() calculates the estimated total increase over the range:

increase(http_requests_total[1h])

The two expressions describe the same behavior with different units:

increase(counter[1h]) ≈ rate(counter[1h]) * 3600

Use rate() for dashboards and alerts that compare ongoing rates. Use increase() when the question is “how many events occurred during this period?”

For a counter that changes infrequently, choose a range long enough to contain multiple scrape intervals. A five-minute range is a common starting point for a 15- or 30-second scrape interval.

Aggregating Series #

Aggregation operators combine multiple series:

sum(rate(http_requests_total[5m]))

Preserve only the labels needed in the result:

sum by (job, status) (
  rate(http_requests_total[5m])
)

Alternatively, remove specified labels and keep the rest:

sum without (instance) (
  rate(http_requests_total[5m])
)

Other useful aggregation operators include avg, min, max, count, topk, and bottomk:

topk(5, sum by (route) (rate(http_requests_total[5m])))

Apply rate() before aggregation. This lets Prometheus detect counter resets independently for each source series:

# Correct
sum by (job) (rate(http_requests_total[5m]))

# Avoid
rate(sum by (job) (http_requests_total)[5m:])

Calculating Ratios #

Suppose http_requests_total has a status label. The fraction of requests that return a 5xx response is:

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

To calculate the ratio per job, retain the same labels on both sides:

sum by (job) (
  rate(http_requests_total{status=~"5.."}[5m])
)
/
sum by (job) (
  rate(http_requests_total[5m])
)

Binary operations between vectors match series by their labels. If the two sides have different label sets, use vector matching explicitly. For example, divide each instance’s error rate by the total error rate for its job:

sum by (job, instance) (
  rate(http_requests_total{status=~"5.."}[5m])
)
/
on (job) group_left
sum by (job) (
  rate(http_requests_total{status=~"5.."}[5m])
)

on (job) chooses the labels used for matching. group_left allows multiple instance series on the left to match one job series on the right. Use these modifiers carefully: unexpected missing results often indicate a label-matching problem.

Querying Gauges #

Gauges can usually be queried directly:

process_resident_memory_bytes

Range functions summarize their recent values:

avg_over_time(process_resident_memory_bytes[15m])
max_over_time(process_resident_memory_bytes[1h])

To calculate memory utilization from Node Exporter metrics:

1
-
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes

Multiply by 100 only when a percentage from 0 to 100 is required. Keeping ratios between 0 and 1 often makes queries easier to compose.

Querying Histograms #

A classic Prometheus histogram named http_request_duration_seconds exports:

http_request_duration_seconds_bucket{le="0.1"}
http_request_duration_seconds_bucket{le="0.5"}
http_request_duration_seconds_bucket{le="1"}
http_request_duration_seconds_bucket{le="+Inf"}
http_request_duration_seconds_sum
http_request_duration_seconds_count

Each _bucket series is a counter. Buckets are cumulative: the le="0.5" bucket includes every observation less than or equal to 0.5 seconds.

The average request duration is the rate of the sum divided by the rate of the count:

sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Use histogram_quantile() to estimate a percentile. The following query calculates the 95th percentile by route:

histogram_quantile(
  0.95,
  sum by (route, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

The le label must remain in the aggregation because histogram_quantile() needs the bucket boundaries. The result is an estimate whose accuracy depends on the chosen buckets.

To calculate the fraction of requests completed within 500 milliseconds:

sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Detecting Missing or Stale Data #

Prometheus marks a series stale when its target stops exposing it. Use absent() when the absence itself is meaningful:

absent(up{job="api"})

Use absent_over_time() to require absence throughout a window:

absent_over_time(up{job="api"}[10m])

For scrape failures, querying up == 0 is usually better because it preserves the target labels:

max_over_time((up{job="api"} == 0)[5m:])

This expression reports targets that had at least one failed scrape during the last five minutes.

Recording and Alerting Rules #

Long or expensive queries can be saved as recording rules. Prometheus evaluates the expression periodically and stores the result as a new time series:

groups:
  - name: api
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

Dashboards can then query the recorded metric:

job:http_requests:rate5m

Alerting rules turn PromQL conditions into alerts:

groups:
  - name: api
    rules:
      - alert: HighHTTPErrorRate
        expr: |
          sum by (job) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (job) (
            rate(http_requests_total[5m])
          )
          > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High HTTP error rate for {{ $labels.job }}"

The for clause prevents a short spike from immediately firing the alert. Prometheus evaluates the condition; Alertmanager handles grouping, inhibition, silencing, and notification delivery.

Using PromQL in Grafana #

After adding Prometheus as a Grafana data source, panel queries are ordinary PromQL expressions. Grafana supplies useful variables:

• $__rate_interval selects a range suitable for rate() based on the panel resolution and scrape interval.
• $__interval represents the panel’s calculated grouping interval.
• $__range represents the dashboard’s selected time range.

For example:

sum by (status) (
  rate(http_requests_total{job="$job"}[$__rate_interval])
)

Use a rate interval rather than Grafana’s raw step for counter queries. The range passed to rate() must be long enough to contain enough samples even when a user zooms in.

Set panel units to match the metric: bytes, seconds, requests per second, or a percentage. PromQL should produce a meaningful number; Grafana should format and visualize it.

Common PromQL Mistakes #

1. Graphing raw counters. Use rate() or increase().
2. Using a range shorter than the scrape interval. The query may not have enough samples to calculate a rate.
3. Aggregating counters before applying rate(). Counter resets can become invisible.
4. Dropping le before histogram_quantile(). Classic histogram quantiles require bucket boundaries.
5. Dividing vectors with incompatible labels. Aggregate both sides to the same label set or use on, ignoring, and group modifiers deliberately.
6. Using unbounded labels. High-cardinality labels increase memory, disk, and query costs.
7. Treating missing data as zero. Missing data may mean a failed scrape, stale series, or a selector that matched nothing.
8. Building every dashboard from raw metrics. Use recording rules for queries that are expensive and frequently reused.

A Practical Query Workflow #

Build complex PromQL incrementally:

1. Start with a metric name and inspect its labels.
2. Add label filters until the selector contains the intended series.
3. Apply a range function such as rate() when the metric is a counter.
4. Aggregate while preserving only the dimensions needed in the result.
5. Add arithmetic or vector matching.
6. Test the query over several time ranges and inspect individual series when the result is surprising.

PromQL becomes much easier once every query is viewed as a transformation of labeled time series. Select the right series, apply a time operation, aggregate their labels, and only then combine the results.