mimir

SKILL.md

Grafana Mimir

Docs: https://grafana.com/docs/mimir/latest/

Horizontally scalable, multi-tenant, long-term storage for Prometheus + OpenTelemetry metrics.

Prerequisites

Docker (for quick start) or a Kubernetes cluster (for Helm)

An object-storage bucket for production (S3/GCS/Azure) — filesystem only for dev

Prometheus or Alloy able to remote_write to the Mimir push endpoint

Common Workflows

1. Stand up monolithic Mimir locally
# demo.yaml
target: all
multitenancy_enabled: false

blocks_storage:
  backend: filesystem
  bucket_store:
    sync_dir: /tmp/mimir/tsdb-sync
  filesystem:
    dir: /tmp/mimir/data/tsdb
  tsdb:
    dir: /tmp/mimir/tsdb

compactor:
  data_dir: /tmp/mimir/compactor
  sharding_ring:
    kvstore: { store: memberlist }

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore: { store: memberlist }

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore: { store: memberlist }
    replication_factor: 1

server:
  http_listen_port: 9009
  grpc_listen_port: 9095
  log_level: error

# 1. Run it
docker run --rm -p 9009:9009 -v $(pwd)/demo.yaml:/etc/mimir/demo.yaml \
  grafana/mimir:latest --config.file=/etc/mimir/demo.yaml

# 2. Verify readiness (expect HTTP 200, body: "ready")
curl -sf http://localhost:9009/ready

# 3. Verify it's serving the API
curl -s http://localhost:9009/api/v1/labels | jq '.status'   # → "success"

# 4. Verify self-metrics scrape
curl -s http://localhost:9009/metrics | grep -E '^mimir_(distributor|ingester)_' | head

2. Send metrics — Prometheus remote_write

remote_write:
  - url: http://localhost:9009/api/v1/push
    headers:
      X-Scope-OrgID: tenant1   # required when multitenancy_enabled: true

# Verify samples are landing
curl -s -H 'X-Scope-OrgID: tenant1' \
  'http://localhost:9009/api/v1/query?query=up' | jq '.data.result | length'
# Expect > 0 within ~30s of Prometheus scraping

3. Send metrics — Grafana Alloy

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
    headers = { "X-Scope-OrgID" = "tenant1" }
  }
}

4. Deploy on Kubernetes (Helm, microservices)

# 1. Install
helm repo add grafana https://grafana.github.io/helm-charts
helm install mimir grafana/mimir-distributed --version 6.0.6 -f values.yaml

# 2. Verify every pod is Running / Ready
kubectl get pods -n mimir
#   distributor, ingester, querier, query-frontend, store-gateway, compactor, ruler

# 3. Verify the gateway is ready
kubectl port-forward -n mimir svc/mimir-nginx 9009:80 &
curl -sf http://localhost:9009/ready

Multi-tenancy

multitenancy_enabled: true
# Every request must include header:  X-Scope-OrgID: <tenant-id>

For storage backends (S3 / GCS / Azure / filesystem) see references/storage.md. For component roles, ring options, limits, and API endpoint dumps see references/architecture.md.

Troubleshooting

/ready returns 503 → ingester still joining ring; check mimir_ring_members and ingester logs

429 Too Many Requests on push → bump limits.ingestion_rate / ingestion_burst_size

Samples written but query returns empty → confirm X-Scope-OrgID matches between write and read

Query for old data returns nothing → check compactor logs and that store-gateway has synced blocks

Resources

Mimir docs

Helm chart mimir-distributed

Architecture reference