alibabacloud-datahub-manage

intent

manage the complete lifecycle of alibaba cloud datahub resources (projects, topics, subscriptions) using the aliyun cli. use this skill when you need to provision streaming data infrastructure, configure topic schemas, set up data subscriptions, or tear down datahub resources. covers creation, querying, updates, and deletion across the region → project → topic → subscription hierarchy.

inputs

external connections:

• aliyun cli: version >= 3.3.3 required. verify with aliyun version. install or upgrade via references/cli-installation-guide.md
• aliyun datahub plugin: automatic installation enabled via aliyun configure set --auto-plugin-install true. run aliyun plugin update before first use
• alibaba cloud credentials: ak/sk pair or sts token configured in ~/.aliyun/config.json or environment variables (ALIBABA_CLOUD_ACCESS_KEY_ID, ALIBABA_CLOUD_ACCESS_KEY_SECRET). verify with aliyun configure list (never echo credentials directly)
• ram permissions: account must have datahub actions: dhs:ListProjects, dhs:CreateProject, dhs:GetProject, dhs:UpdateProject, dhs:DeleteProject, dhs:ListTopics, dhs:CreateTopic, dhs:GetTopic, dhs:UpdateTopic, dhs:DeleteTopic, dhs:ListSubscriptions, dhs:CreateSubscription, dhs:GetSubscription, dhs:DeleteSubscription. see references/ram-policies.md

user-provided parameters (require explicit confirmation before execution):

session state: generate a single random 32-character lowercase hex string (session-id) at skill load time. use it for all --user-agent flags in this session.

procedure

step 1: verify prerequisites and authentication

1. run aliyun version to confirm cli >= 3.3.3

   • output: version string (e.g., 3.4.0)
   • if missing or too old, stop and direct user to references/cli-installation-guide.md

2. run aliyun configure set --auto-plugin-install true to enable auto-install

   • output: no error message indicates success

3. run aliyun plugin update to sync plugins

   • output: list of updated plugins or "up-to-date"

4. run aliyun configure list to verify credentials

   • output: json with profile name, ak/sk status (or sts/oauth)
   • if no valid profile in output, stop and instruct user to run aliyun configure outside this session, then return

5. confirm ram permissions via user attestation or ram-permission-diagnose skill

   • input: none
   • output: user confirms permission availability or skill pauses for permission grant

step 2: list existing projects (conflict check)

1. prompt user: "what region will you use?"

   • confirm region is valid aliyun region id
   • store as {region}

2. run list-projects to check for name conflicts:

   aliyun datahub list-projects \
     --region {region} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: {region}, {session-id}
   • output: json array of existing project names
   • if list is empty, proceed to step 3
   • if list contains user's intended project name, stop and notify user of conflict

step 3: create project

1. prompt user: "what project name (3-32 chars, starts with letter, alphanumeric + underscore)?"

   • validate input matches constraint
   • store as {project-name}

2. prompt user (optional): "add a project description or comment?"

   • store as {project-comment} (default: empty string)

3. run create-project:

   aliyun datahub create-project \
     --region {region} \
     --project-name {project-name} \
     --comment "{project-comment}" \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: {region}, {project-name}, {project-comment}, {session-id}
   • output: success message with project-id or error (e.g., auth failure, project exists)
   • on auth error: stop, run ram-permission-diagnose skill, ask user to confirm permissions granted, then retry
   • on project-exists error: stop, notify user to try different name, loop to step 3.1

4. verify project creation:

   aliyun datahub get-project \
     --region {region} \
     --project-name {project-name} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: {region}, {project-name}, {session-id}
   • output: json with project metadata (id, create-time, owner, region)
   • if get-project fails, stop and notify user creation may not have completed, retry step 3.3

step 4a: create topic (blob type)

1. prompt user: "create a topic now? (yes/no)"

   • if no, skip to step 5
   • if yes, continue

2. prompt user: "what topic name (3-128 chars)?"

   • validate input
   • store as {topic-name}

3. prompt user: "will this be BLOB (binary) or TUPLE (structured with schema)?"

   • if TUPLE, skip to step 4b
   • if BLOB, continue with step 4a.4

4. prompt user: "how many shards? (integer >= 1)"

   • validate input is integer
   • store as {shard-count}

5. prompt user: "data retention in days? (integer >= 1)"

   • validate input is integer
   • store as {lifecycle}

6. prompt user (optional): "add a topic description?"

   • store as {topic-comment} (default: empty)

7. run create-topic for BLOB:

   aliyun datahub create-topic \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --comment "{topic-comment}" \
     --shard-count {shard-count} \
     --record-type BLOB \
     --lifecycle {lifecycle} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all params above plus {session-id}
   • output: success message or error
   • on auth error: stop, run ram-permission-diagnose, wait for confirmation, retry
   • on topic-exists error: stop, ask user to use different topic name, loop to step 4a.2

8. verify topic creation:

   aliyun datahub get-topic \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all region/project/topic/session params
   • output: json with topic metadata (shards, lifecycle, record-type, create-time)
   • if get-topic fails, stop and notify user

step 4b: create topic (tuple type with schema)

1. steps 4a.1-4a.6 apply here (name, comment, shard-count, lifecycle)

2. prompt user: "define your tuple schema in json format. example: {"fields":[{"name":"user_id","type":"STRING","notnull":"false"}]}"

   • accept json string
   • store as {record-schema}
   • validate json is parseable; if not, prompt user to retry

3. run create-topic for TUPLE:

   aliyun datahub create-topic \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --comment "{topic-comment}" \
     --shard-count {shard-count} \
     --record-type TUPLE \
     --lifecycle {lifecycle} \
     --record-schema '{record-schema}' \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all params above
   • output: success or error
   • error handling: same as step 4a.7

4. verify topic:

   • run same get-topic as step 4a.8
   • output: json includes record-schema field

step 4c: update topic description (optional, post-creation)

1. prompt user: "update topic description? (yes/no)"

   • if no, skip to step 5
   • if yes, continue

2. prompt user: "new description?"

   • store as {new-comment}

3. run update-topic:

   aliyun datahub update-topic \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --comment "{new-comment}" \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all params above
   • output: success message or error
   • on error, notify user and offer retry

step 5: create subscription

1. prompt user: "create a subscription now? (yes/no)"

   • if no, skip to step 6
   • if yes, continue

2. prompt user: "subscription application name (max 256 chars)? (e.g., 'mobile-analytics-service')"

   • validate length <= 256
   • store as {application}

3. prompt user (optional): "subscription description or comment?"

   • store as {subscription-comment} (default: empty)

4. prompt user (optional): "manual subscription-id (4-40 chars, lowercase alphanumeric + dash), or auto-generate?"

   • if auto-generate, set {subscription-id} to empty string, skip to step 5.6
   • if manual, store as {subscription-id} and validate format

5. display summary:

   creating subscription with:
   - project: {project-name}
   - topic: {topic-name}
   - application: {application}
   - subscription-id: {subscription-id} (or auto if empty)
   - comment: {subscription-comment}
   
   confirm? (yes/no)
   

   • if no, loop to step 5.2
   • if yes, continue

6. run create-subscription:

   aliyun datahub create-subscription \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --application "{application}" \
     --comment "{subscription-comment}" \
     [--subscription-id {subscription-id}] \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all params (subscription-id flag omitted if empty)
   • output: success message with subscription-id (if auto-generated) or error
   • on auth error: stop, run ram-permission-diagnose, wait, retry
   • store returned subscription-id as {actual-subscription-id}

7. verify subscription:

   aliyun datahub get-subscription \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --subscription-id {actual-subscription-id} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all params
   • output: json with subscription metadata (create-time, application, owner)
   • if get-subscription fails, stop and notify user

step 6: cleanup (deletion, reverse dependency order)

1. prompt user: "delete resources? (yes/no)"

   • if no, exit skill with summary of created resources
   • if yes, continue to step 6.2

2. prompt user: "delete subscriptions, topics, or entire project?"

   • store user choice as {deletion-scope} (subscription / topic / project)

step 6a: delete subscription only

1. prompt user: "list subscriptions to delete? (yes/no)"

   • if yes, run:

     aliyun datahub list-subscriptions \
       --region {region} \
       --project-name {project-name} \
       --topic-name {topic-name} \
       --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
     

   • output: json array of subscriptions on this topic
   • display list to user, ask: "which subscription-id to delete?"
   • store as {target-subscription-id}

2. display confirmation prompt (required, do not skip):

   YOU ARE ABOUT TO DELETE (IRREVERSIBLE):
   - subscription: {target-subscription-id}
   - topic: {topic-name}
   - project: {project-name}
   
   THIS ACTION CANNOT BE UNDONE.
   type 'confirm-delete' to proceed, or 'cancel' to abort:
   

   • if user does not type exact string confirm-delete, abort deletion and exit step 6a

3. run delete-subscription:

   aliyun datahub delete-subscription \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --subscription-id {target-subscription-id} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all params
   • output: success message or error
   • on auth error: stop, run diagnose, wait, retry
   • on not-found error: stop, notify user subscription does not exist

4. notify user: "subscription deleted successfully"

step 6b: delete topic (and subscriptions)

1. pre-check: list subscriptions on topic:

   aliyun datahub list-subscriptions \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • output: json array of subscriptions
   • if non-empty, display list and prompt: "delete all these subscriptions first? (yes/no)"
   • if yes, loop through each subscription-id and run delete-subscription (step 6a.3) for each, skipping the confirmation prompt (since user already confirmed topic deletion)
   • if no, stop and ask user to delete manually via step 6a

2. display confirmation prompt (required):

   YOU ARE ABOUT TO DELETE (IRREVERSIBLE):
   - topic: {topic-name}
   - project: {project-name}
   - all subscriptions on this topic
   
   THIS ACTION CANNOT BE UNDONE.
   type 'confirm-delete' to proceed, or 'cancel' to abort:
   

   • if user does not type exact string confirm-delete, abort

3. run delete-topic:

   aliyun datahub delete-topic \
     --region {region} \
     --project-name {project-name} \
     --topic-name {topic-name} \
     --user-agent AlibabaCloud-Agent-Skills/alibabacloud-datahub-manage/{session-id}
   

   • input: all params
   • output: success or error
   • error handling: same as step 6a.3

4. notify user: "topic deleted successfully"

step 6c: delete project (and all children)

1. pre-check: list topics