chaos-engineer

SKILL.md

Chaos Engineer

When to Use This Skill

Designing and executing chaos experiments

Implementing failure injection frameworks (Chaos Monkey, Litmus, etc.)

Planning and conducting game day exercises

Building blast radius controls and safety mechanisms

Setting up continuous chaos testing in CI/CD

Improving system resilience based on experiment findings

Core Workflow

System Analysis - Map architecture, dependencies, critical paths, and failure modes

Experiment Design - Define hypothesis, steady state, blast radius, and safety controls

Execute Chaos - Run controlled experiments with monitoring and quick rollback

Learn & Improve - Document findings, implement fixes, enhance monitoring

Automate - Integrate chaos testing into CI/CD for continuous resilience

Reference Guide

Load detailed guidance based on context:

Topic
Reference
Load When

Experiments
references/experiment-design.md
Designing hypothesis, blast radius, rollback

Infrastructure
references/infrastructure-chaos.md
Server, network, zone, region failures

Kubernetes
references/kubernetes-chaos.md
Pod, node, Litmus, chaos mesh experiments

Tools & Automation
references/chaos-tools.md
Chaos Monkey, Gremlin, Pumba, CI/CD integration

Game Days
references/game-days.md
Planning, executing, learning from game days

Safety Checklist

Non-obvious constraints that must be enforced on every experiment:

Steady state first — define and verify baseline metrics before injecting any failure

Blast radius cap — start with the smallest possible impact scope; expand only after validation

Automated rollback ≤ 30 seconds — abort path must be scripted and tested before the experiment begins

Single variable — change only one failure condition at a time until behaviour is well understood

No production without safety nets — customer-facing environments require circuit breakers, feature flags, or canary isolation

Close the loop — every experiment must produce a written learning summary and at least one tracked improvement

Output Templates

When implementing chaos engineering, provide:

Experiment design document (hypothesis, metrics, blast radius)

Implementation code (failure injection scripts/manifests)

Monitoring setup and alert configuration

Rollback procedures and safety controls

Learning summary and improvement recommendations

Concrete Example: Pod Failure Experiment (Litmus Chaos)

The following shows a complete experiment — from hypothesis to rollback — using Litmus Chaos on Kubernetes.

Step 1 — Define steady state and apply the experiment

# Verify baseline: p99 latency < 200ms, error rate < 0.1%
kubectl get deploy my-service -n production
kubectl top pods -n production -l app=my-service

Step 2 — Create and apply a Litmus ChaosEngine manifest

# chaos-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: my-service-pod-delete
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=my-service"
    appkind: deployment
  # Limit blast radius: only 1 replica at a time
  engineState: active
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"          # seconds
            - name: CHAOS_INTERVAL
              value: "20"          # delete one pod every 20s
            - name: FORCE
              value: "false"
            - name: PODS_AFFECTED_PERC
              value: "33"          # max 33% of replicas affected

# Apply the experiment
kubectl apply -f chaos-pod-delete.yaml

# Watch experiment status
kubectl describe chaosengine my-service-pod-delete -n production
kubectl get chaosresult my-service-pod-delete-pod-delete -n production -w

Step 3 — Monitor during the experiment

# Tail application logs for errors
kubectl logs -l app=my-service -n production --since=2m -f

# Check ChaosResult verdict when complete
kubectl get chaosresult my-service-pod-delete-pod-delete \
  -n production -o jsonpath='{.status.experimentStatus.verdict}'

Step 4 — Rollback / abort if steady state is violated

# Immediately stop the experiment
kubectl patch chaosengine my-service-pod-delete \
  -n production --type merge -p '{"spec":{"engineState":"stop"}}'

# Confirm all pods are healthy
kubectl rollout status deployment/my-service -n production

Concrete Example: Network Latency with toxiproxy

# Install toxiproxy CLI
brew install toxiproxy   # macOS; use the binary release on Linux

# Start toxiproxy server (runs alongside your service)
toxiproxy-server &

# Create a proxy for your downstream dependency
toxiproxy-cli create -l 0.0.0.0:22222 -u downstream-db:5432 db-proxy

# Inject 300ms latency with 10% jitter — blast radius: this proxy only
toxiproxy-cli toxic add db-proxy -t latency -a latency=300 -a jitter=30

# Run your load test / observe metrics here ...

# Remove the toxic to restore normal behaviour
toxiproxy-cli toxic remove db-proxy -n latency_downstream

Concrete Example: Chaos Monkey (Spinnaker / standalone)

# chaos-monkey-config.yml — restrict to a single ASG
deployment:
  enabled: true
  regionIndependence: false
chaos:
  enabled: true
  meanTimeBetweenKillsInWorkDays: 2
  minTimeBetweenKillsInWorkDays: 1
  grouping: APP           # kill one instance per app, not per cluster
  exceptions:
    - account: production
      region: us-east-1
      detail: "*-canary"  # never kill canary instances

# Apply and trigger a manual kill for testing
chaos-monkey --app my-service --account staging --dry-run false

Documentation