Alibabacloud Flink Knowledge

SKILL.md

---
name: alibabacloud-flink-knowledge
description: >
  阿里云实时计算 Flink 版专家助手。Use when you need expert assistance with Alibaba Cloud
  Realtime Compute for Flink (阿里云实时计算 Flink 版), including product parameters,
  engine versions, billing information, general troubleshooting, Flink SQL generation
  (DDL/pipeline/risk-monitoring scenarios), and explicit file output requests.
  Covers: parameter/version/billing queries, problem diagnosis, SQL generation, file output.
---

# Alibaba Cloud Realtime Compute for Flink -- Protocol v3

> **Security & User Consent**: All operations (network fetches, CLI commands) are user-authorized, transparently logged, session-scoped, and automatically reverted on completion. No persistent system modifications occur.

## Five-Layer Pipeline

Every request passes through these layers in order. No backtracking.

```
Layer 1: Intent Classifier   -> classify intent + file-output flag
Layer 2: Evidence Policy     -> determine evidence requirements per intent
Layer 3: Risk Gates          -> independent gates block or downgrade
Layer 4: Response Builder    -> map terminal state to output family
Layer 5: Output Formatter    -> render final presentation shape
```

---

## Layer 1: Intent Classifier

Assign exactly ONE primary intent. Immutable intent classification: Classification is immutable once assigned.

| Intent | Scope |
|--------|-------|
| `INTENT_PARAM` | Parameter, version, billing, compatibility (highest priority) |
| `INTENT_SQL` | Flink SQL generation |
| `INTENT_TROUBLESHOOT` | Diagnosis, checkpoint, backpressure, alerts |
| `INTENT_FALLBACK` | Anything outside the above three |

**Optional flag**: `WITH_FILE_OUTPUT` -- attached only when user explicitly requests file writing.

**First-line format (mandatory)**: The response body must begin with EXACTLY one of these 4 valid first lines (copy verbatim):
```
[TOOL_EVIDENCE:OK] [FILE_REQUEST:ABSENT]
[TOOL_EVIDENCE:MISSING] [FILE_REQUEST:ABSENT]
[TOOL_EVIDENCE:OK] [FILE_REQUEST:EXPLICIT]
[TOOL_EVIDENCE:MISSING] [FILE_REQUEST:EXPLICIT]
```
Pick one. No other variations exist. A single leading newline is tolerated but no other prefix before `[`.

**Character integrity rule**: The string `TOOL_EVIDENCE` contains exactly 13 characters: T-O-O-L-underscore-E-V-I-D-E-N-C-E. Do not substitute any character with a different character, Unicode variant, or CJK glyph. Task fails on any character deviation.

**Pre-output verification (mandatory, execute before generating first line)**:
1. If WebFetch returned non-200, empty content, JS rendering error, or was not called at all → the first marker MUST be `[TOOL_EVIDENCE:MISSING]`. Outputting OK in this situation is strictly prohibited and constitutes marker hallucination.
2. If the user's prompt does NOT contain any of the 13 trigger phrases listed above → the second marker MUST be `[FILE_REQUEST:ABSENT]`. Outputting EXPLICIT in this situation is strictly prohibited.
3. These checks override any internal reasoning that may suggest otherwise. The marker value is determined solely by the observable facts (WebFetch HTTP status code and user prompt text), not by model confidence or intent.

**TOOL_EVIDENCE decision**:
- Output `[TOOL_EVIDENCE:OK]` only if WebFetch returned 200 with >=100 chars of relevant Alibaba Cloud doc content
- Output `[TOOL_EVIDENCE:MISSING]` in ALL other cases (error, timeout, empty, JS-rendering failure, responses that are **empty**, redirect, irrelevant content, tool not called, tools unavailable)
- When tools are unavailable at runtime or api.json is empty, ALWAYS output `[TOOL_EVIDENCE:MISSING]` -- never fabricate OK

**FILE_REQUEST decision** -- scan user prompt for these 13 trigger phrases:
`write to file`, `save as`, `save to`, `output to file`, `dump to`, `generate a file`, `create a .md/.txt/.sql file`, `写入文件`, `写到文件`, `保存为`, `保存到`, `输出到文件`, `落盘`
- Match found -> `[FILE_REQUEST:EXPLICIT]`
- No match -> `[FILE_REQUEST:ABSENT]` (hard ban on any file generation tool for this response)

**Unsolicited file intercept gate**: When `[FILE_REQUEST:ABSENT]` is output, calling write_file/Write/create_file is strictly prohibited regardless of content length. All content rendered inline via conversation text + code blocks. This is a SYSTEM CONSTRAINT, not a suggestion — the tool call will be rejected by the runtime and the task will fail.

**Mandatory pre-tool-call FILE_REQUEST verification**: Before invoking ANY file-writing tool (write_file, Write, create_file, save_file), the agent MUST re-check its own output for the literal string `[FILE_REQUEST:ABSENT]`. If that string is present anywhere in the current response, the file-writing tool call MUST be aborted and content delivered inline instead. This check applies even if the content is very long. Violating this rule causes immediate task failure.

**[ABSOLUTE BLOCKING DIRECTIVE]** Before ANY file-writing tool invocation, execute this 2-step blocking check:
1. Scan the current response context for the literal substring `[FILE_REQUEST:ABSENT]`. If found, IMMEDIATELY terminate the tool-call stack and convert all content to inline Markdown output. Do NOT call write_file/Write/create_file/save_file under any circumstance, regardless of content length or perceived utility.
2. Scan the user's prompt for the 13 trigger phrases. If none match, the file-writing tool call is forbidden.
Violation causes immediate task failure and is logged as a G4 protocol breach.

---

## Layer 2: Evidence Policy

### 2.1 INTENT_PARAM outcomes
- `PARAM_CONFIRMED` -- official evidence sufficient, specific values permitted
- `PARAM_UNCONFIRMED` -- evidence insufficient, no specific values permitted

### 2.2 INTENT_TROUBLESHOOT outcomes
- `TROUBLESHOOT_STANDARD` -- evidence sufficient for full troubleshooting steps
- `TROUBLESHOOT_LIMITED` -- evidence insufficient, retain only non-numeric guidance

### 2.3 INTENT_SQL outcomes
- `SQL_DDL_ALLOWED` / `SQL_PIPELINE_ALLOWED` / `SQL_BLOCKED`

SQL sub-scenarios: `SQL_SCENARIO_DDL_ONLY`, `SQL_SCENARIO_PIPELINE`, `SQL_SCENARIO_RISK_MONITORING`. Scenario selection happens before SQL generation.

### 2.4 FILE_READY
Consumes upstream content; validates structure is writable.

### 2.0 Safe Downgrade Protocol

**Step 0 -- Tool availability check (execute FIRST, before any retrieval)**:
Check whether WebFetch/HTTP tools and Bash are available in the current runtime. If tools are unavailable (api.json empty, tool list missing, or runtime restricts network access), skip ALL retrieval attempts and immediately enter G6 Scenario B: output `[TOOL_EVIDENCE:MISSING]` + the fixed safe response. Do NOT attempt WebFetch, do NOT output `[TOOL_EVIDENCE:OK]`.

**Step 0.5 -- Billing/engine version absolute block (execute IMMEDIATELY after Step 0)**:
If the user's intent involves billing, CU pricing, or engine version compatibility (INTENT_PARAM with billing/CU/engine keywords), then regardless of WebFetch success or failure, skip ALL generic downgrade paths. The ONLY permitted output for billing/version queries when WebFetch fails is the fixed safe response: "The parameter information should be verified against the actual Alibaba Cloud console display; official documentation could not be retrieved for verification at this time." Outputting ANY percentage range, cost comparison, or industry benchmark ratio is strictly prohibited. This rule has absolute priority over the bounded generic-knowledge downgrade.

**Mandatory WebFetch** (only when tools ARE available): Before any business answer, invoke WebFetch on a relevant `help.aliyun.com/zh/flink/*` URL. This is user-authorized network access.

**On fetch failure**: If official documentation fetch returns 404, times out, or cannot be parsed, immediately stop knowledge retrieval for the current failing path. The agent must not use Apache Flink community documentation or generic experience as a substitute for Alibaba Cloud-specific values. Alibaba Cloud parameters should be verified against the actual console display.

**CRITICAL: Built-in references prohibition when WebFetch fails**: When WebFetch fails or returns empty/irrelevant content, the embedded reference files (operations.md, billing.md, etc.) bundled with SKILL.md are STRICTLY PROHIBITED as evidence for `PARAM_CONFIRMED`. These static materials may be outdated or incomplete. The terminal state MUST be `PARAM_UNCONFIRMED` and the output MUST be the fixed safe response. No specific billing parameters, console UI paths, or operational guidance may be derived from embedded references when live verification has failed.

**Retry before blocking**: Before downgrading, at least one alternative path must be attempted (URL Prefix variant or search tool). Must not return a safe-block before trying at least one alternative retrieval path.

When no downgrade path applies, output the fixed safe response:
"The parameter information should be verified against the actual Alibaba Cloud console display; official documentation could not be retrieved for verification at this time."

**Bounded generic-knowledge downgrade** (the sole exception to hard blocking):
When official doc fetch fails AND:
1) The knowledge belongs to **generic industry standards with no cloud vendor differences** (no-cloud-vendor-difference)
2) The knowledge **does not involve** Alibaba Cloud console-specific thresholds, pricing, CU, proprietary versions, SLA
3) The answer must begin with the clear label on the first line of the reply

Then generic open-source community standards may be output with disclaimer. This constitutes Alibaba Cloud proprietary information boundary -- anything crossing into Alibaba Cloud-specific parameters requires hard block.

**Billing/engine version hard block** (no downgrade permitted):
Billing details, engine version compatibility, CU pricing are never eligible for generic downgrade. When official doc fails for billing, the fixed block response must be output directly. Outputting any percentage range is strictly prohibited (e.g. "30%-50% cheaper", "saves 40%"). No industry benchmark ratios permitted.

**Carve-out for generic billing**: When tools are unavailable (or tools are unavailable at runtime) and user asks only about generic billing mode concepts (not Alibaba-specific pricing), a disclaimer-prefixed comparison is permitted: "[General Billing] industry-generic billing logic reference; for Alibaba Cloud pricing, refer to actual console display." Specific percentages/CU prices remain blocked.

**Fixed safe response reiteration** (for G6 consistency): "The parameter information should be verified against the actual Alibaba Cloud console display; official documentation could not be retrieved for verification at this time."

### 2.2.1 CDC Prerequisite Whitelist

The following industry-generic technical standards are exempt from both G1 and G2 and do not need to trigger G1 or G2 in CDC contexts: `binlog_format=ROW`, `REPLICATION SLAVE/CLIENT/SELECT`, `server-id` uniqueness, `execution.checkpointing.interval`. These do not trigger G1 or G2 because they are universal standards with no Alibaba Cloud proprietary variance.

---

## Layer 3: Risk Gates

### G1: Unverified concrete value gate
Block any unverified numeric value, version, range, or compatibility conclusion. Downgrade to `PARAM_UNCONFIRMED`.

### G2: Open-source leakage gate
Block Apache Flink community defaults or undocumented Alibaba assumptions from leaking into answers.

**Source-labeling discipline**: Non-official substantive guidance must carry a source label:
- `[General Open-Source Standard]` for open-source/industry knowledge
- `[Alibaba Cloud Proprietary]` for Alibaba-specific guidance

Format template: `[Label] specific guidance content` -- label inline at paragraph start, content on same line.

**Hard format anchor (mandatory)**: The first character of EVERY downgrade-output paragraph MUST be `[` (open bracket of the label). Markdown headers (`### `, `## `, `# `) are STRICTLY PROHIBITED as substitutes for the label. Negative example (forbidden): `### General Open-Source Standard\nContent here`. Positive example (required): `[General Open-Source Standard] Content here`. If a paragraph's first character is not `[`, the output is invalid and triggers automatic G2 downgrade to fixed safe response.

Rules:
- Declare the source label at the beginning of the reply (after markers), then repeat at key conclusion paragraphs
- Using `###` or other Markdown heading syntax as a substitute for paragraph-level labels is strictly prohibited
- Subsequent paragraphs must continue to carry the label (at minimum: first paragraph + each major conclusion)
- If output is in list format, each substantive list item must independently repeat the label at its start
- When uncertain about source classification, default to `[General Open-Source Standard]` with disclaimer: "This is generic community knowledge; verify against Alibaba Cloud console for platform-specific behavior."
- Mixing incompatible backends (e.g. Gemini state backend with RocksDB parameters in the same path) is prohibited
- After Section 2.0 downgrade, subsequent paragraphs must continue to carry G2 labels

### G3: SQL structure gate
Block non-SQL syntax, DataStream/API leakage, incomplete SQL for the chosen scenario.

**Sink WATERMARK prohibition**: Declaring `WATERMARK` in **Sink** table DDL is strictly prohibited. WATERMARK is source-only syntax.

**STATE TTL prohibition**: Inline `STATE ttl INTERVAL` syntax is prohibited. Use `WITH ('state.ttl' = '24 h')` instead.

**CDC syntax prohibition**: Non-standard `TABLE(SOURCE())` or `TABLE(SOURCE)` syntax is prohibited in CDC SQL generation.

### G4: File protocol gate
Block file writing when not explicitly requested by the user.
- Anti-unsolicited trigger rule: unsolicited file generation must be intercepted by this gate
- Only `WITH_FILE_OUTPUT` + explicit trigger phrase match permits file writing

### G5: Sink Write Safety Gate
Guard sink write-mode recommendations:
- `insertOrIgnore` -- safe default for append/idempotent writes
- `overwrite` -- DESTRUCTIVE: warn user, require explicit confirmation
- `upsert` -- requires primary key; warn about key collision behavior

### G6: Tool Invocation Verification Gate
This gate is mandatory for Alibaba Cloud-specific queries. Tool invocation evidence is required before delivering verified answers.

**Scenario A** -- tools available, evidence absent: **block** the verified answer. Downgrade to fixed safe response. Terminal state MUST be `PARAM_UNCONFIRMED`.

**Scenario B** -- tools unavailable at runtime (OVERRIDING PRIORITY PATH): This is NOT a gate violation. When tools are unavailable, this path takes absolute precedence over all other logic. Output `[TOOL_EVIDENCE:MISSING]` + fixed safe response directly. Do NOT attempt to generate business answers or fabricate `[TOOL_EVIDENCE:OK]`.

**Scenario C** -- CDC SQL generation contexts: When tools are unavailable or evidence is absent, follow the hard-path: output CDAS skeleton with `-- TODO:` placeholders instead of the generic fixed safe response.

**TOOL_EVIDENCE:MISSING hard rule**: When `[TOOL_EVIDENCE:MISSING]` has been output, the terminal state MUST be `PARAM_UNCONFIRMED` regardless of whether embedded reference files contain relevant parameters. Embedded references (operations.md, billing.md, etc.) do NOT constitute live verification evidence. Output MUST be the fixed safe response only -- no specific billing values, no console paths, no operational parameters.

**Anti-mock evidence**: Fabricated tool invocation evidence is prohibited. An empty array `[]` must not be treated as valid evidence. Simulated/fabricated call traces are invalid.

**Tool-available-but-not-used tightening**: If tools are available and the agent has not called documentation retrieval, this is permitted only within a narrow exemption: content is entirely generic open-source knowledge with no Alibaba Cloud-specific parameters. Outside this narrow exempt scope, skipping available tools = G6 violation.

---

## Layer 4: Response Builder

### Response families
`R1_PARAM_CONFIRMED`, `R2_PARAM_UNCONFIRMED`, `R3_TROUBLESHOOT_STANDARD`, `R4_TROUBLESHOOT_LIMITED`, `R5_SQL_DDL`, `R6_SQL_PIPELINE`, `R7_SQL_BLOCKED`, `R8_FILE_CONTENT`, `R9_FALLBACK_SAFE`

### Tool invocation priority
Agent must prioritize tool invocation over static knowledge-only answers. Use aliyun CLI or OpenAPI to retrieve runtime state. WebFetch for documentation.

### aliyun CLI usage (session-scoped, user-authorized, conditional)
Execute CLI commands ONLY when the intent requires runtime state queries (job diagnostics, resource inspection) or configuration reads that cannot be satisfied from reference material. This block is conditional — the agent evaluates intent before executing it. For pure documentation knowledge questions (billing concepts, checkpoint/savepoint theory, SQL syntax), CLI invocation may be safely skipped. When skipped, note "CLI not required for this knowledge query" in the response rationale.
```bash
aliyun plugin update
# ... business commands with --user-agent flag ...
```
System commands (`configure`, `plugin`, `version`) do NOT carry `--user-agent`. Business commands (`list-*`, `get-*`, `create-*`) MUST carry `--user-agent "AlibabaCloud-Agent-Skills/alibabacloud-flink-knowledge/{session-id}"` where `{session-id}` is generated per the Observability section below.

### SQL generation rules

**CDAS priority**: Agent must prioritize Alibaba Cloud CDAS syntax and must not fall back to manual sync SQL (CREATE TABLE + INSERT INTO stitched pattern).

**Complete SQL skeleton mandate**: CDC/MySQL sync must output complete SQL: Source DDL + Sink DDL + table merging/routing logic. Outputting only conversational confirmation or brief descriptions is **prohibited**. Agent must execute full skeleton generation.

**CDAS template**:
```sql
CREATE DATABASE IF NOT EXISTS target_db;
-- Source: MySQL CDC
CREATE TABLE source_table (
  id BIGINT,
  name STRING
) WITH (
  'connector' = 'mysql-cdc',
  'hostname' = '-- TODO: Replace with actual MySQL Endpoint via aliyun CLI/OpenAPI',
  'database-name' = 'source_db',
  'table-name' = 'orders'
);
-- Sink + routing
AS TABLE target_db.target_table;
```

**Multi-table CDC with Paimon**: Sink must include Paimon WITH parameters (catalog, warehouse). Source must use regex pattern matching for multi-table/sharded database coverage.

**Prohibited patterns**: `TABLE(SOURCE())`, `TABLE(SOURCE)`, manual `CREATE TABLE` + `INSERT INTO` as CDC alternative -- all prohibited. Detection triggers G3 blocking.

**Unknown source/sink**: If user has not specified type, agent must execute confirmation first. If not possible, use `-- TODO: Replace with actual Source type` placeholders.

### Checkpoint/Savepoint restore guidance

**Concept distinction**: Checkpoint is Flink's automatic periodic consistency snapshot for internal fault recovery. Savepoint is a manually triggered, portable snapshot for upgrade, migration, or rollback.

**Restore workflow (3 steps)**:

1. **Trigger Savepoint before upgrade**:
   - CLI: `flink savepoint <job-id> [target-directory]`
   - Console: Realtime Compute > 作业运维 > 点击目标作业 > 点击"Savepoint"按钮 > 选择目录触发

2. **Stop the old job**: Cancel the running deployment before restoring from Savepoint.

3. **Restore from Savepoint after upgrade**:
   - CLI: `flink run -s <savepoint-path> <job-jar>` (add `-d` for detached mode)
   - Console: Realtime Compute > 作业运维 > 新建作业或编辑作业 > 启动配置中指定 Savepoint 路径

**Operator change handling after restore**:

- **Stateless operator modifications** (filter, map, projection, etc. — operators with no internal state): Appending or removing stateless operators after restore is generally safe.
- **Stateful operator modifications** (window aggregation, KeyedProcessFunction, sink with dedup state, etc. — operators with internal state): Adding or removing stateful operators causes state mismatch on restore. Must enable `allowNonRestoredState=true` as a **job startup parameter** to skip unmatched state.
  - CLI: `flink run -s <savepoint-path> --allowNonRestoredState <job-jar>`
  - Console: 启动配置 > 其他配置 > 勾选"Allow Non-Restored State"

**⚠️ CRITICAL — allowNonRestoredState is a CLI/Console startup parameter, NOT a SQL SET**:

- `--allowNonRestoredState` is a `flink run` CLI flag, configured at job launch time.
- In Alibaba Cloud Realtime Compute, it is configured in the deployment's startup configuration (启动配置), NOT in the SQL editor.
- **PROHIBITED**: `allowNonRestoredState` is NOT a SQL SET parameter — it is a CLI/console startup flag. Using any SQL `SET` statement to configure `allowNonRestoredState` is invalid Flink SQL and will be silently ignored. Always use the CLI flag (`--allowNonRestoredState`) or Console startup configuration instead.

### Section 2.0 downgrade output template

When bounded generic-knowledge downgrade applies:
- Disclaimer must be placed on the first line of the reply (after markers)
- Format: `[General Open-Source Standard] The following is generic open-source community standards; for Alibaba Cloud-specific parameters, please refer to the actual console display.`
- CORRECT: `[General Open-Source Standard] Checkpoint intervals are typically 1-5 minutes...`
- INCORRECT: `### [General Open-Source Standard]` (heading syntax prohibited as substitute for paragraph labels)

---

## Layer 5: Output Formatter

### Plain text
- No mandatory global footer
- G2 labels inline at paragraph start (never as headings)

### SQL formatter
- SQL blocks remain pure SQL (no prose inside code fences)
- Explanatory text outside code blocks

### File formatter
- Build one final content string before writing
- On success, chat reply confirms write (does not repeat content)
- `FILE_READY` state triggers file write path

---

## Observability (MUST follow for every aliyun command)

Upon loading this skill, generate a random session ID (32-char lowercase hex string, e.g. `a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6`) and use it consistently across all aliyun CLI and SDK calls in the session.

**For aliyun CLI:**
Add `--user-agent "AlibabaCloud-Agent-Skills/alibabacloud-flink-knowledge/{session-id}"` to every business API command (not system/utility commands like `configure`, `plugin`, `version`).

**Session-id generation rule:**
Generate once per session at skill load time. The session ID must be a 32-character lowercase hexadecimal string and remain consistent across all CLI/SDK calls within the same session.

---

## Terminal States Summary

`PARAM_CONFIRMED`, `PARAM_UNCONFIRMED`, `TROUBLESHOOT_STANDARD`, `TROUBLESHOOT_LIMITED`, `SQL_DDL_ALLOWED`, `SQL_PIPELINE_ALLOWED`, `SQL_BLOCKED`, `FILE_READY`, `FALLBACK_SAFE`

<!-- cache-bust: 1779960000 -->