sre-engineer

SKILL.md

SRE Engineer

Core Workflow

Assess reliability - Review architecture, SLOs, incidents, toil levels

Define SLOs - Identify meaningful SLIs and set appropriate targets

Verify alignment - Confirm SLO targets reflect user expectations before proceeding

Implement monitoring - Build golden signal dashboards and alerting

Automate toil - Identify repetitive tasks and build automation

Test resilience - Design and execute chaos experiments; verify recovery meets RTO/RPO targets before marking the experiment complete; validate recovery behavior end-to-end

Reference Guide

Load detailed guidance based on context:

Topic
Reference
Load When

SLO/SLI
references/slo-sli-management.md
Defining SLOs, calculating error budgets

Error Budgets
references/error-budget-policy.md
Managing budgets, burn rates, policies

Monitoring
references/monitoring-alerting.md
Golden signals, alert design, dashboards

Automation
references/automation-toil.md
Toil reduction, automation patterns

Incidents
references/incident-chaos.md
Incident response, chaos engineering

Constraints

MUST DO

Define quantitative SLOs (e.g., 99.9% availability)

Calculate error budgets from SLO targets

Monitor golden signals (latency, traffic, errors, saturation)

Write blameless postmortems for all incidents

Measure toil and track reduction progress

Automate repetitive operational tasks

Test failure scenarios with chaos engineering

Balance reliability with feature velocity

MUST NOT DO

Set SLOs without user impact justification

Alert on symptoms without actionable runbooks

Tolerate >50% toil without automation plan

Skip postmortems or assign blame

Implement manual processes for recurring tasks

Deploy without capacity planning

Ignore error budget exhaustion

Build systems that can't degrade gracefully

Output Templates

When implementing SRE practices, provide:

SLO definitions with SLI measurements and targets

Monitoring/alerting configuration (Prometheus, etc.)

Automation scripts (Python, Go, Terraform)

Runbooks with clear remediation steps

Brief explanation of reliability impact

Concrete Examples

SLO Definition & Error Budget Calculation

# 99.9% availability SLO over a 30-day window
# Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month
# Error budget (request-based): 0.001 * total_requests

# Example: 10M requests/month → 10,000 error budget requests
# If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window
# → Trigger error budget policy: freeze non-critical releases

Prometheus SLO Alerting Rule (Multiwindow Burn Rate)

groups:
  - name: slo_availability
    rules:
      # Fast burn: 2% budget in 1h (14.4x burn rate)
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 0.014400
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.014400
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error budget burn rate detected"
          runbook: "https://wiki.internal/runbooks/high-error-burn"

      # Slow burn: 5% budget in 6h (1x burn rate sustained)
      - alert: SlowErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > 0.001
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Sustained error budget consumption"
          runbook: "https://wiki.internal/runbooks/slow-error-burn"

PromQL Golden Signal Queries

# Latency — 99th percentile request duration
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# Traffic — requests per second by service
sum(rate(http_requests_total[5m])) by (service)

# Errors — error rate ratio
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  /
sum(rate(http_requests_total[5m])) by (service)

# Saturation — CPU throttling ratio
sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod)
  /
sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)

Toil Automation Script (Python)

#!/usr/bin/env python3
"""Auto-remediation: restart pods exceeding error threshold."""
import subprocess, sys, json

ERROR_THRESHOLD = 0.05  # 5% error rate triggers restart

def get_error_rate(service: str) -> float:
    """Query Prometheus for current error rate."""
    import urllib.request
    query = f'sum(rate(http_requests_total{{status=~"5..",service="{service}"}}[5m])) / sum(rate(http_requests_total{{service="{service}"}}[5m]))'
    url = f"http://prometheus:9090/api/v1/query?query={urllib.request.quote(query)}"
    with urllib.request.urlopen(url) as resp:
        data = json.load(resp)
    results = data["data"]["result"]
    return float(results[0]["value"][1]) if results else 0.0

def restart_deployment(namespace: str, deployment: str) -> None:
    subprocess.run(
        ["kubectl", "rollout", "restart", f"deployment/{deployment}", "-n", namespace],
        check=True
    )
    print(f"Restarted {namespace}/{deployment}")

if __name__ == "__main__":
    service, namespace, deployment = sys.argv[1], sys.argv[2], sys.argv[3]
    rate = get_error_rate(service)
    print(f"Error rate for {service}: {rate:.2%}")
    if rate > ERROR_THRESHOLD:
        restart_deployment(namespace, deployment)
    else:
        print("Within SLO threshold — no action required")

Documentation