gke-workload-scaling

SKILL.md

GKE Workload Scaling

This skill provides workflows and best practices for scaling applications on
Google Kubernetes Engine (GKE). It covers manual scaling, Horizontal Pod
Autoscaling (HPA), and Vertical Pod Autoscaling (VPA).

Workflows

1. Manual Scaling

Scale a deployment to a fixed number of replicas. Useful for immediate manual
intervention or testing.

Command:

kubectl scale deployment {deployment_name} --replicas={number} -n {namespace}

Verify the scale event

kubectl get deployment {deployment_name} -n {namespace}

### 2. Horizontal Pod Autoscaling (HPA)

Automatically scale the number of pods based on observed CPU utilization, memory
utilization, or custom metrics.

**Prerequisites:**

-   Metrics Server must be running (enabled by default on GKE).
-   Containers clearly define resource requests/limits.

**Quick Command:**

```bash
kubectl autoscale deployment {deployment_name} --cpu-percent=50 --min=1 --max=10

Manifest Approach (Recommended): Use a YAML manifest for version-controlled
configuration. See assets/hpa-example.yaml for a
template.

kubectl apply -f assets/hpa-example.yaml

# Verify HPA is created and fetching metrics
kubectl get hpa

Custom Metrics & External Metrics: For GKE, the modern and recommended
approach for scaling based on Cloud Monitoring metrics (e.g., Pub/Sub queue
length) is to use the External metric type, which is natively supported by
the GKE control plane without requiring the Custom Metrics Adapter. For
application-specific metrics exposed via Prometheus, you can use Google Cloud
Managed Service for Prometheus or the Prometheus Adapter.

3. Vertical Pod Autoscaling (VPA)

Automatically adjust the CPU and memory reservations for your pods to match
actual usage. This is critical for right-sizing workloads.

Prerequisites:

VPA must be enabled on the cluster.

Autopilot: Enabled by default.

Standard: Must be enabled manually.

Enable VPA on Standard Cluster:

gcloud container clusters update {cluster_name} --enable-vertical-pod-autoscaling --zone {zone}

Update Modes:

Off: Calculates recommendations but does not apply them. Good for "dry
run" analysis.

Initial: Assigns resources only at pod creation time.

Auto: Updates running pods by restarting them if recommendations differ
significantly from requests.

InPlaceOrRecreate: Attempts to update Pod resources without recreating the
Pod. If in-place update is not possible, it reverts to Auto mode (requires
GKE 1.34+).

Example: See assets/vpa-example.yaml for a
configuration template.

Best Practices

Define Resource Requests: HPA and VPA rely on accurate resource
requests. Always define them in your container specs.

Avoid Metric Conflicts: Do not configure HPA and VPA to use the same
metric (e.g., both CPU). This causes thrashing.

Typical Pattern: HPA on CPU, VPA on Memory.

Pod Disruption Budgets (PDBs): Define PDBs to ensure application
availability during scaling events or node upgrades.

HPA Lag: HPA has a stabilization window (default 5 mins) to prevent
rapid fluctuation.

VPA "Auto" Mode Risks: In "Auto" mode, VPA restarts pods to change
resources. Ensure your application handles restarts gracefully (e.g.,
handles SIGTERM).

Note: By default, VPA requires at least 2 replicas to perform
evictions (to prevent a situation where the only running replica is
evicted, causing downtime). In GKE 1.22+, you can override this by
setting minReplicas in PodUpdatePolicy.

Rightsizing Workflow

Deploy VPA in Off mode for 24+ hours

Read recommendations: kubectl describe vpa {deployment_name}-vpa -n {namespace}

Compare target values against current requests

Apply with 20% buffer: new_request = target * 1.2

Use patch format or update deployment manifest to apply new resource
requests

Condition
Recommendation
Risk

CPU request >5x P95 actual
Reduce to P95 * 1.2
Medium

Memory request >3x P95 actual
Reduce to P95 * 1.2
Medium

CPU request >2x P95 actual
Rightsizing with 20% buffer
Low

No resource limits set
Add limits to prevent noisy-neighbor
Low