LEARN. PRACTICE. GROW.Concept and mechanism
A counter accumulates events and can reset when its process starts; a gauge represents a value that can rise or fall. In PromQL, rate estimates a counter's per-second rate over a window and handles resets. Calculate the rate per series before aggregation to preserve reset detection. Do not treat a queue-depth gauge as an accumulated total. A missing series also differs from an observed zero. absent_over_time can identify a window without samples, but the cause still requires diagnosis. Check discovery, scraping, labels, retention, and collection-infrastructure health before concluding that traffic is absent. An extrapolated rate is an estimate rather than an exact request-by-request event ledger.
Guided application
For aggregate latency, do not average each instance's precomputed p95. Histograms allow observations to be aggregated before estimating the percentile. With classic histograms, preserve the le label when aggregating cumulative buckets; native histograms use a different representation and query. Current documentation recommends native histograms where supported, but installed libraries and versions constrain that choice. Resolution and window affect the estimate. For an explicit threshold, the proportion of observations within an appropriate bucket may be more direct than a percentile. Also control cardinality: every unique label combination creates a series. Use parameterized routes and bounded categories instead of customer IDs. Hashing each ID preserves the number of distinct values.
One series per customer remains one series per customer after hashing the ID.
Common pitfalls
Rate after summing; gauge as counter; absence as zero; averaging percentiles.
Related topics: Signals and service outcomes · Traces and context · Instrumentation and protection
Respect each metric's type, population, and aggregation.
Reference: PromQL rate aggregation and absent-series functions · Observability 2026-09; selected OpenTelemetry, Prometheus and Dynatrace Classic concepts