gke-ai-troubleshooting-tpu-vbar-oom

SKILL.md

TPU Connection Failure and VBAR OOM Troubleshooting

Use this skill to systematically diagnose and prevent vbar_control_agent
segfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.

⚠️ Prerequisites

Cloud Logging must be enabled for the project.

Access to the project and cluster via gcloud or equivalent tool.

🔍 Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Independently gather required context using available GCP/GKE tools or use the
provided {variable} placeholders:

{project_id}: The GCP Project ID (e.g., customer-ai-project-123).

{cluster_name}: The GKE Cluster Name (e.g., tpu-cluster-prod).

{node_name}: The Node Name or Instance ID (e.g., tpu-node-1).

{workload_name}: The Workload Name / JobSet Name (e.g.,
my-training-job-456).

{namespace}: The Workload Namespace.

{issue_time}: The timestamp of the issue (e.g., 2026-04-14T20:00:00Z).

Time Handling & Execution Rules

Window Calculation: If an issue timestamp {issue_time} is provided,
calculate the query time window as [{issue_time} - 30m] to
[{issue_time} + 30m].

Let {start_time} = {issue_time} - 30m

Let {end_time} = {issue_time} + 30m

Informational vs. Live Execution: If the user request is informational
or query-formulation (e.g. "How can I check...", "How do I determine..."),
or if live GCP project resources are not actively targetable, directly
output the calculated time window, log names, and Cloud Logging filter
templates without attempting live log execution commands.

Step 1: Check for vbar_control_agent OOMs

Look for specific out of memory messages from vbar_control_agent in serial
console logs (serialconsole.googleapis.com%2fserial_port_1_output).

Tool to use: query_logs (for live diagnostics)

Filter Templates:

Serial Console Logs (OOMs):

logName="projects/{project_id}/logs/serialconsole.googleapis.com%2fserial_port_1_output"
AND labels."compute.googleapis.com/resource_name"="{node_name}"
AND SEARCH(text_payload, "Memory cgroup out of memory: Killed process .* (vbar_control_ag)")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Presence of Memory cgroup out of memory messages related to
vbar_control_agent. Stack traces pointing to
libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR are a strong
indicator.

Automation: Proceed to next step automatically after reporting findings.

Reference: See references/failure_signatures.md for example log
patterns.

Step 2: Investigate tpu-device-plugin Metrics Fetch Failures [Low Risk]

Check if tpu-device-plugin is reporting metric fetch failures.

Tool to use: query_logs

Filter Template:

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND resource.labels.container_name="tpu-device-plugin"
AND severity=ERROR
AND textPayload:"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Errors indicating "metrics fetch failed" with "checksum didn't
match" suggest vBAR memory corruption.

Automation: Proceed to next step automatically after reporting findings.

Step 3: Check for Custom Metrics Collection Usage [Low Risk]

Inspect cluster configurations, workloads, or container specs to determine if
custom TPU metrics collection mechanisms are deployed.

Action: Check if custom scripts or agents (e.g., using
libtpu.sdk.tpumonitoring) are deployed that frequently query
GetHostMetrics from vBAR Control Agent.

Verification Commands:

Kubectl Search (Inspect workload env/specs):

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

Log Search Filter (query_logs):

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND textPayload:"libtpu.sdk.tpumonitoring"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

Logic: Confirmation of custom metrics collection helps confirm the race
condition hypothesis.

🛠️ Resolution Workflow

Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]

If a custom metrics collection agent is identified, recommend disabling it.

Action: Recommend disabling the custom metrics collector.

Justification: Prevents reads from vBAR during device resets, stopping
crashes and OOMs.

Resolution 2: Await vbar_control_agent Resiliency Update [Low Risk]

Advise that a permanent fix will be available in a future GKE version.

Action: Recommend upgrading GKE when the fix is available.

Justification: The updated agent will be resilient to memory corruption
and gracefully handle reads from unbound vBARs.

📋 copypaste checklist

 Acquire context and compute [{start_time}, {end_time}] window.

 Check for vbar_control_agent segfaults and OOMs using query_logs.

 Investigate tpu-device-plugin failures using query_logs.

 Inspect for custom metrics collection usage.

 Advise disabling custom metrics collection if applicable.

 Advise awaiting resiliency update.

2f:["$