prometheus

SKILL.md

Metrics with Prometheus and Grafana

Value Proposition

Prometheus is an open-source monitoring and alerting toolkit for cloud-native environments. Combined with
Grafana Cloud Metrics (powered by Grafana Mimir), it provides a fully managed Prometheus-compatible service
with long-term storage, global query performance, and enterprise scalability.

Key Differentiators: Pull-based model, dimensional data model with labels, PromQL, automatic service
discovery, scales to billions of active series.

PromQL Quick Reference

Instant Vector Selectors

# By metric name
http_requests_total

Label filter

http_requests_total{job="api-server"}

Multiple labels (AND)

http_requests_total{job="api-server", method="GET"}

Regex

http_requests_total{job="api.*", status="5.."}

Negative

http_requests_total{status!="200"}

### Range Vectors & Rates

```promql
# Per-second rate over 5 minutes
rate(http_requests_total[5m])

# Increase over interval
increase(http_requests_total[1h])

# Instant rate (last two samples)
irate(http_requests_total[5m])

# Offset (5 minutes ago)
rate(http_requests_total[5m] offset 5m)

Aggregations

# Sum by label
sum by (job) (rate(http_requests_total[5m]))

# Average
avg by (instance) (node_cpu_seconds_total)

# Top-K
topk(5, rate(http_requests_total[5m]))

# Histogram quantiles
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# Count distinct
count(up{job="api"})

Common Patterns

# Error rate percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

# Saturation (CPU usage %)
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

# Predict disk full (linear extrapolation)
predict_linear(node_filesystem_free_bytes[6h], 24*3600) < 0

Metrics Drilldown

Queryless Prometheus metrics exploration (preinstalled in Grafana 12+):

Browse metrics without writing PromQL

Smart segmentation and anomaly detection

Auto-visualization with optimal chart types

Metric relationship discovery

Telemetry pivoting (metrics to logs)

Alerting

Prometheus Alertmanager

Route, group, silence, and deduplicate alerts. Multi-destination routing (PagerDuty, Slack, Email, webhooks).

Grafana Alerting

Unified alerting across all data sources. Supports multi-dimensional alerts, notification policies,
and contact points.

Recording Rules

Pre-compute expensive PromQL queries for dashboard performance:

groups:
  - name: api_rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

Architecture

Pull-based scraping: Prometheus scrapes HTTP endpoints at configured intervals

Service discovery: Automatic target discovery for K8s, EC2, Consul

Push gateway: For short-lived jobs that can't be scraped

Remote write/read: Send metrics to Grafana Cloud, Thanos, Mimir

Local storage: Efficient on-disk time-series database

Resources

Prometheus Documentation

PromQL Reference

Grafana Cloud Metrics

Metrics Drilldown App

Grafana Alerting

Grafana Mimir