Alibabacloud Pai Dlc Job Diagnostics

SKILL.md

---
name: alibabacloud-pai-dlc-job-diagnostics
description: |
  PAI-DLC job diagnostics and health inspection. Queuing-stuck root cause
  analysis, failed-job localization, cluster health checks.
  Companion to the `alibabacloud-pai-dlc-job` skill (read-only — no writes).
  Triggers: "diagnose", "diagnose job", "job stuck", "why queuing",
  "queue stuck", "stuck in queue", "job failed", "failure reason",
  "healthcheck", "health check", "inspect job", "inspection".
---

# PAI-DLC Job Diagnostics and Health Inspection

Read-only diagnostic analysis for PAI-DLC distributed training jobs, covering
three scenarios:

- **Queuing-stuck root cause analysis** — quota check, node scheduling
  diagnosis, hyper-node availability
- **Failed-job localization** — failure classification, logs/events evidence
  chain, root cause identification
- **Cluster health inspection** — training throughput, hang detection,
  SanityCheck, restart stability

**Architecture**: PAI-DLC Job (read-only queries) + PAI Studio Resource
Diagnosis API (queuing scenario).

## 0. Dependencies

This skill performs **read-only diagnostics** only. All write operations
(create / update / stop jobs, resource discovery, etc.) live in the companion
skill `alibabacloud-pai-dlc-job`. The two skills are complementary in
responsibility and share a common field contract.

| Prerequisite Skill | Role | When to switch to it |
|--------------------|------|----------------------|
| `alibabacloud-pai-dlc-job` | Write ops (create/update/stop) + AIWorkSpace resource discovery | Creating / modifying / stopping jobs, or discovering Image / Dataset / CodeSource |
| This skill | Read-only diagnostics (logs / events / sanity-check / queuing root cause) | Job already exists — troubleshooting or health inspection |

**Discover and install the prerequisite skill:**

```bash
# Discover available skills
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-find-skills
# Install the alibabacloud-pai-dlc-job skill itself
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-pai-dlc-job
```

**Cross-skill field contract:** The `--job-id` / `--pod-id` values this skill
consumes are produced verbatim by `alibabacloud-pai-dlc-job` via
`list-jobs` / `get-job --cli-query "Pods[0].PodId"` — no transformation needed.
`--region` / `--workspace-id` follow the same resolution rules in both skills.

## Installation Requirements

> **Pre-check: Aliyun CLI >= 3.3.1 required**
> Run `aliyun version` to verify >= 3.3.1. If not installed or version too low,
> see [references/cli-installation-guide.md](references/cli-installation-guide.md).
> Then [MUST] run `aliyun configure set --auto-plugin-install true`.

> **Note on `--user-agent`:** Every API-invoking `aliyun` command in this skill MUST
> include `--user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics`. Client-side helpers
> (`aliyun version`, `aliyun configure ...`, `aliyun plugin ...`,
> `aliyun <product> --help`) do not invoke remote APIs and therefore do not require
> the flag.

```bash
aliyun version
aliyun configure set --auto-plugin-install true
aliyun plugin update
aliyun pai-dlc --help
aliyun paistudio --help >/dev/null 2>&1 || aliyun plugin install --names aliyun-cli-paistudio

aliyun configure ai-mode enable
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics"
# After session: aliyun configure ai-mode disable
```

## Authentication

> **Pre-check: Alibaba Cloud Credentials Required**
>
> **Security Rules:**
> - **NEVER** read, echo, or print AK/SK values
> - **NEVER** ask the user to input AK/SK directly
> - **ONLY** use `aliyun configure list` to check credential status
>
> ```bash
> aliyun configure list
> ```
> Check the output for a valid profile (AK, STS, or OAuth identity).
>
> **If no valid profile exists, STOP here.**
> 1. Obtain credentials from [Alibaba Cloud Console](https://ram.console.aliyun.com/manage/ak)
> 2. Configure credentials **outside of this session**
> 3. Return and re-run after `aliyun configure list` shows a valid profile

## RAM Permissions

> **[MUST] Permission Failure Handling:** When any command fails due to permission errors:
> 1. Read `references/ram-policies.md` for the full permission list
> 2. Use `ram-permission-diagnose` skill to guide the user
> 3. Pause and wait until the user confirms permissions have been granted

| Product | Permissions | Purpose |
|---------|-------------|---------|
| pai-dlc | `pai:GetJob`, `pai:GetPodLogs`, `pai:GetJobEvents`, `pai:GetPodEvents`, `pai:ListJobSanityCheckResults` | Job information collection |
| paistudio | `paistudio:GetQuotaWorkloadDiagnosis` | Queuing resource diagnosis |

## Parameter Confirmation

> **IMPORTANT: Parameter Confirmation** — Before executing any command,
> ALL user-customizable parameters (RegionId, JobId, etc.) MUST be confirmed with the user.

| Parameter | Required | Description |
|-----------|----------|-------------|
| `region` | Yes | Region where the job runs |
| `job_id` | Yes | DLC job ID (e.g., `dlcXXX`) |

---

## Entry Routing

When a diagnostic request arrives, first call `get-job` to fetch job status,
then route by status:

| Job status | Route to scenario |
|------------|-------------------|
| `Queuing` / `Creating` | → Queuing-stuck root cause analysis |
| `Failed` | → Failed-job localization |
| `Running` | → Health inspection |
| `Stopped` | Inform the user "job was actively stopped", no diagnosis |
| `Succeeded` | → Historical review (follow Scenario 3 Execution steps) |

**Edge case — job was queuing but is now Stopped/Succeeded**: If the user
describes the job as "stuck in queue" but `get-job` shows `Stopped` or
`Succeeded`, still route to Scenario 1 (queuing analysis) but expect the
resource diagnosis API to return HTTP 400. Follow the "Fallback on API Failure"
procedure in Scenario 1.

Users may also directly request a specific scenario (e.g., "run a health
inspection" even when status is not Running).

---

## Diagnostic Toolbox

### PAI-DLC Read-Only Commands

```bash
aliyun pai-dlc get-job --region <r> --job-id <id> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-job-events --region <r> --job-id <id> --max-events-num 50 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-pod-events --region <r> --job-id <id> --pod-id <pod> --max-events-num 20 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-pod-logs --region <r> --job-id <id> --pod-id <pod> --max-lines 100 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc list-job-sanity-check-results --region <r> --job-id <id> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-job-sanity-check-result --region <r> --job-id <id> --sanity-check-number 1 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
```

### PAI Studio Resource Diagnosis (queuing scenario only)

```bash
aliyun paistudio GET /api/v1/quotas/{quota_id}/workloads/{job_id}/diagnosis \
  --region <r> --header "Content-Type=application/json" --force \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
```

**Hard constraint**: `quota_id` MUST come from `get-job`'s `ResourceId` field.
If `ResourceId` is empty (public pay-as-you-go), this API is unavailable.

Full API structure: see [references/resource-diagnosis-api.md](references/resource-diagnosis-api.md).

---

## Scenario 1: Queuing-Stuck Root Cause Analysis

**Trigger**: job status = `Queuing` / `Creating` and user reports it cannot be scheduled.

**Tools**: `get-job` → `paistudio resource diagnosis` → (optional) `get-job-events`.

**Hard constraints**:
- `ResourceId` empty → resource diagnosis unavailable; mine events for clues
- `ResourceId` non-empty → resource diagnosis API is the primary instrument

> **CRITICAL: pai-dlc vs paistudio — two different products**
>
> | Product | Scope | Commands |
> |---------|-------|----------|
> | **pai-dlc** | Job/Pod lifecycle (GetJob, GetJobEvents, ListJobs, GetPodLogs) | `aliyun pai-dlc get-job ...` |
> | **paistudio** | Platform-level services including resource diagnosis | `aliyun paistudio GET /api/v1/quotas/...` |
>
> The resource diagnosis API belongs to **paistudio**, NOT pai-dlc.
> Do **NOT** call `pai-dlc GetResourceQuota`, `pai-dlc ListResourceQuotas`,
> or any `pai-dlc GET /api/v1/resourcequotas/...` — these are wrong APIs
> and will fail. The correct command is:
> ```bash
> aliyun paistudio GET /api/v1/quotas/{quota_id}/workloads/{job_id}/diagnosis ...
> ```

**Pattern knowledge**: resource diagnosis returns 4 checks
(`self_quota` / `ancestor_quota` / `user_limit` / `queue_strategy`), plus node
scheduling and hyper-node analysis. Common patterns:
[references/diagnostic-patterns.md](references/diagnostic-patterns.md) §1.

**Agent latitude**: decide whether to compute the quota gap, whether to pull
events for corroboration, and how verbose the report should be.

### Fallback on API Failure

The PAI Studio resource diagnosis API may fail with HTTP 400/404 when the
job is no longer in an active queuing state (e.g., already `Stopped` by user).
In this case:

1. **Explicitly declare** the API failure in the report:
   `"Resource diagnosis API unavailable: HTTP {code} — {error message}"`
2. **Perform qualitative analysis** using only these data sources:
   - `get-job` → `ResourceRequest` (GPU/CPU/Memory demand per pod)
   - `get-job` → `PodCount` × per-pod resources = **total demand**
   - `get-job` → `EcsSpec` (instance type and its per-node capacity)
   - `get-job-events` → scheduling event timeline and queuing duration
3. **Prohibited language**: NEVER use hedging words such as "possibly",
   "might", "perhaps", "may not be sufficient". State only confirmed facts:
   - Total GPU demand = PodCount × RequestGPU = {computed value}
   - Instance type = {EcsSpec}
   - Queuing duration = {computed hours/minutes}
   - Job final status = {Stopped/Succeeded/etc.}
4. **Mandatory output structure** when API is unavailable:

```markdown
## Resource Diagnosis (API Unavailable — Configuration-Based Analysis)

- Diagnosis API: unavailable (HTTP {code}: {message})
- Resource demand: {PodCount} pods × {GPU/pod} GPU = {total} GPU cards
- Instance type: {EcsSpec}
- Queuing duration: {hours}h {minutes}m
- Quota ID: {ResourceId}
- Job final status: {status} ({ReasonCode})
- Conclusion: The job requested {total} GPU cards which could not be
  fulfilled within the queuing window before the job was {stopped/completed}.
```

---

## Scenario 2: Failed-Job Localization

**Trigger**: job status = `Failed`.

**Tools**: `get-job` → (as needed) `get-job-events` / `get-pod-events` / `get-pod-logs`.

**Hard constraints**:
- `Stopped` is not a failure — do not diagnose
- **[OUTPUT GUARD — RECOMMENDATIONS STRICTLY FORBIDDEN]** When the failure
  falls into any of the categories below, the diagnostic report **MUST strictly
  omit** any "Recommendations" / "Suggested fixes" / "Next steps" / "Solution"
  section, and output only the root cause and objective facts:
  - `ResourceAllocateFailed` (insufficient resources)
  - Job preempted / evicted (`ReasonMessage` contains `preempted` or `evicted`)
  - Spot instance reclamation
  - **Report MUST end on the diagnostic conclusion.** No sentence anywhere in
    the output may instruct the user to change configuration, request more
    quota, switch instance types, add retry logic, or modify any job parameter.
  - **Negative examples (ALL forbidden)**:
    - "Recommend using pay-as-you-go instances"
    - "Consider requesting more quota"
    - "Try switching to another zone"
    - "Disable preemptible jobs by setting EnablePreemptibleJob: false"
    - "Implement checkpointing and retry logic"
    - "Solution: Use guaranteed quota instead of oversold quota"
  - **Positive examples (allowed)**:
    - "Root cause: spot instance reclaimed by the cloud platform, triggering pod eviction and job failure."
    - "The job failed due to preemption/eviction of pods. ReasonCode: JobFailed."
  - **Only permitted user-facing suggestion**: "For quota policy adjustments, please contact your platform administrator."

**Mandatory output template for preemption/eviction/resource-shortage failures**:
```markdown
## Diagnosis Conclusion

- Failure reason: {classification} ({ReasonCode}: {ReasonMessage})
- Affected pods: {pod list with SubStatus}
- Timeline: {key timestamps from events}
- Evidence: {quoted ReasonMessage or event details}

For quota policy or resource allocation adjustments, please contact your
platform administrator.
```

**Pattern knowledge**: failure-classification priority
(network > image > runtime > resource > config > system), exit-code meanings,
keyword-matching rules. See
[references/diagnostic-patterns.md](references/diagnostic-patterns.md) §2.

**Agent latitude**: when `ReasonCode` is clear, logs may be unnecessary; when
logs already explain the issue, events may be unnecessary. Decide investigation
depth based on information sufficiency.

---

## Scenario 3: Health Inspection

**Trigger**: job status = `Running` and user requests inspection / health check.

**Tools**: `get-job` + `get-job-events` + `get-pod-logs`
+ `list-job-sanity-check-results`.

**Execution steps**:
1. `get-job` → obtain status, WorkspaceId, Pod list, whether `EnableSanityCheck` is set
2. `get-job-events` → event-chain analysis (scheduling, restarts, etc.)
3. `get-pod-logs` → target the **master pod** (rank=0) first; extract structured training metrics if present
4. `list-job-sanity-check-results` → execute only if `EnableSanityCheck=true` in job settings
5. **Generate console links (MANDATORY — must always output both links)**:
   - Job overview: `https://pai.console.aliyun.com/?regionId={region}&workspaceId={workspace_id}#/dlc/jobs/{job_id}/overview`
   - Monitoring dashboard: `https://pai.console.aliyun.com/?regionId={region}&workspaceId={workspace_id}#/dlc/jobs/{job_id}/monitor`
   - **Mandatory closing notice** (append verbatim to every inspection report):
     ```
     > GPU/memory real-time resource utilization metrics require the monitoring
     > dashboard above. This skill's CLI commands do not support fetching
     > runtime utilization data directly.
     ```
   This step is **mandatory** regardless of job status (Running, Succeeded, or
   any other state). Even if the job has already completed, the user needs
   the monitoring link to review historical resource utilization.

**Dimension matrix** (mandatory vs optional):

| Dimension | Mandatory/Optional | Notes |
|-----------|-------------------|-------|
| Training throughput | Optional | Extract from master pod logs |
| Hang detection | Running only | Skip for Succeeded jobs |
| Hardware health | Optional | Requires `EnableSanityCheck=true` |
| Restart stability | **Mandatory** | Read `RestartCount` directly from `get-job` |

**Note on resource utilization**: GPU/memory metrics are NOT available via this
skill's CLI commands. The monitoring dashboard link (step 5) and its closing
notice are **mandatory** in every inspection report — do not omit them.

Interpretation rules per dimension: see
[references/healthcheck-dimensions.md](references/healthcheck-dimensions.md).

**Agent latitude**: kilo-card jobs focus on Hang + SanityCheck; small jobs
look at training throughput from logs. Decide depth and report verbosity
from scale and user intent.

---

## Best Practices

1. **`get-job` first, route second** — never assume status; query and then route
2. **Cap response size** — `--max-lines 100` / `--max-events-num 50` to avoid context blow-up
3. **Find the problem pod** — among `get-job` Pods, focus on `Status=Failed/Unknown` or those with `ReasonMessage`
4. **Read log tail** — errors usually live in the last few dozen lines; no need to pull the whole log
5. **Exit codes are clues, not conclusions** — `exit 137` may be OOM or external kill; combine with context
6. **Don't over-diagnose** — when `ReasonCode` already states the cause, skip the full log dump
7. **Console links** — generate both overview and monitoring links so the user can jump into the console for details and resource utilization metrics
8. **Read-only** — this skill MUST NEVER execute `stop` / `update` / `create`
9. **Report summary** — present a per-dimension rating table at the end of the report:

| Dimension | Rating | Key Finding |
|-----------|--------|-------------|
| Training throughput | ✅/⚠️/❌/N/A | ... |
| Hang detection | ✅/⚠️/❌/N/A | ... |
| Hardware health | ✅/⚠️/❌/N/A | ... |
| Restart stability | ✅/⚠️/❌ | ... |

## Reference Links

| Document | Contents |
|----------|----------|
| [references/resource-diagnosis-api.md](references/resource-diagnosis-api.md) | PAI Studio resource diagnosis API full reference |
| [references/diagnostic-patterns.md](references/diagnostic-patterns.md) | Failure pattern knowledge base (queuing / failure / hang / restart) |
| [references/healthcheck-dimensions.md](references/healthcheck-dimensions.md) | Health inspection dimensions and interpretation rules |
| [references/ram-policies.md](references/ram-policies.md) | RAM permission policies |
| [references/related-commands.md](references/related-commands.md) | CLI command quick reference |
| [references/acceptance-criteria.md](references/acceptance-criteria.md) | Acceptance criteria |
| [references/verification-method.md](references/verification-method.md) | Verification method |
| [references/cli-installation-guide.md](references/cli-installation-guide.md) | CLI installation guide |