gke-ai-troubleshooting-jobset-interruption

SKILL.md

GKE JobSet Interruption Troubleshooting

Use this skill to systematically diagnose and resolve JobSet interruptions,
restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.

⚠️ Prerequisites & Sandbox Rules

JobSet metrics package must be enabled in kube-state-metrics for your
cluster.

Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.

Autonomous & Sandbox Execution Rule: If API queries, tools, or commands
encounter 403 Permission Denied, authentication errors, or network
isolation, do NOT enter authentication or credential troubleshooting
loops. Populate the query templates with the acquired variables
({project_id}, {cluster_name}, {workload_name}, {start_time},
{end_time}), inspect any locally staged telemetry or mock data files if
available, and complete the diagnostic workflow and resolution
recommendations autonomously.