Aws Fis Experiment Prepare

SKILL.md

---
name: aws-fis-experiment-prepare
description: >
  Use when the user wants to prepare, create, or generate an AWS FIS (Fault
  Injection Service) experiment configuration. Triggers on "prepare FIS
  experiment", "create FIS experiment for [scenario]", "generate chaos
  experiment config", "准备 FIS 实验", "生成 [scenario] 混沌实验配置",
  "create experiment template for AZ power interruption", "set up fault
  injection test". Covers Scenario Library pre-built scenarios (AZ Power
  Interruption, AZ Application Slowdown, Cross-AZ Traffic Slowdown,
  Cross-Region Connectivity), custom single FIS actions
  (aws:rds:failover-db-cluster, aws:ec2:stop-instances, etc.), and SSM
  Automation-based fault injection for Amazon MSK (broker reboot) and
  ElastiCache Redis/Valkey (primary node reboot, replication group failover).
---

# AWS FIS Experiment Prepare

Generate all configuration files needed to run an AWS FIS experiment, then
deploy via CloudFormation with self-healing iteration until the stack
succeeds. Outputs a self-contained directory with a validated, deployed
experiment template ready for execution.

**Core principle:** Validate resource-action compatibility before generating
files. Never deliver untested configuration — deploy and self-heal first.

## References

**Always load for every experiment:**
- `references/output-format.md` — directory layout, slug naming, README
  template
- `references/cfn-base-template.md` — CFN skeleton (Parameters, IAM Role,
  Dashboard, FIS Template, Outputs)
- `references/slug-conventions.md` — scenario/context slug abbreviations,
  resource naming, name length budget

**Load conditionally by scenario:**
- `references/az-power-interruption-guide.md` — AZ Power Interruption
  (sub-action pruning, tagging strategy, permissions)
- `references/eks-pod-action-guide.md` — any `aws:eks:pod-*` action
  (RBAC Lambda, EKS Access Entry, Pod memory stress calculation)
- `references/elasticache-redis-guide.md` — ElastiCache Redis/Valkey
  (native AZ power interruption, primary node reboot via SSM
  Automation, or replication group failover via SSM Automation)
- `references/msk-guide.md` — Amazon MSK (broker reboot via SSM
  Automation — no native FIS action exists)

**Utility scripts (execute, do not read as reference):**
- `scripts/precheck-cfn-permissions.sh` — detects required CFN service role
- `scripts/deploy-with-retry.sh` — validate + deploy + delete-on-fail
- `scripts/rename-output-dir.sh` — appends FIS template ID to directory name

**Script invocation:** `${SKILL_DIR}` refers to the absolute path of this
skill's directory (where SKILL.md lives). Resolve it from the skill's
filesystem location before running any scripts.

## Output Language Rule

Detect the user's conversation language and use the **same language** for all
output files (README.md, comments in JSON/YAML).
- Chinese input → Chinese output
- English input → English output
- Mixed → follow the dominant language

## Prerequisites

Required tools:
- **AWS CLI** — `aws fis list-actions`, resource discovery, CloudFormation
- **aws___search_documentation** / **aws___read_documentation** — FIS docs
  research
- **jq** — required by `scripts/deploy-with-retry.sh` and
  `scripts/precheck-cfn-permissions.sh`

**EKS Pod fault injection:** Cluster auth mode must be
`API_AND_CONFIG_MAP` or `API`. Check:
```bash
aws eks describe-cluster --name {CLUSTER} \
  --query 'cluster.accessConfig.authenticationMode'
```
If `CONFIG_MAP` only, the user must update the cluster first.
**MANDATORY:** For any `aws:eks:pod-*` action, follow
`references/eks-pod-action-guide.md`.

## Workflow

### Step 1: Identify Scenario and Region

**Classify user intent into one of these branches:**

| Branch | Trigger | Additional Reference |
|---|---|---|
| Scenario Library | AZ Power Interruption, AZ App Slowdown, Cross-AZ/Region scenarios | Read AWS doc URL (table below) |
| Custom FIS action | User specifies an action ID or describes a single fault | — |
| Custom FIS action (ElastiCache) | ElastiCache AZ power interruption or Redis/Valkey failover | `references/elasticache-redis-guide.md` |
| SSM Automation | Target service has no native FIS action (MSK, ElastiCache primary reboot, ElastiCache failover) | `references/msk-guide.md` or `references/elasticache-redis-guide.md` |

If ambiguous, ask the user.

**Scenario Library documentation URLs** (JSON templates are NOT available via
CLI/API — read the doc to extract):

| Scenario | Documentation URL |
|---|---|
| AZ Power Interruption | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/az-availability-scenario.html` |
| AZ Application Slowdown | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/az-application-slowdown-scenario.html` |
| Cross-AZ Traffic Slowdown | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/cross-az-traffic-slowdown-scenario.html` |
| Cross-Region Connectivity | `https://docs.aws.amazon.com/en_us/fis/latest/userguide/cross-region-scenario.html` |

**Region detection order:**
1. User explicitly specifies
2. Infer from context (ARNs, previous conversation)
3. `aws configure get region`
4. Ask the user

Store as `TARGET_REGION`.

**Default experiment duration: `PT10M` (10 minutes)** for all scenarios and
sub-actions unless the user specifies otherwise. For AZ Power Interruption,
scale ARC Zonal Autoshift timing proportionally (ARC starts at minute 2,
runs for 8 minutes at PT10M; formula: `startAfter = duration × (5/30)`).

### Step 2: Discover Target Resources

#### For Scenario Library Scenarios

**CRITICAL: Scenario Library experiment templates CANNOT be generated via
FIS API.** You MUST call `aws___read_documentation` with the scenario URL
(Step 1 table) to extract the JSON experiment template before generating
any files. The documentation is the only authoritative source.

**Target identification — prefer `resourceArns` over `resourceTags`:**
- Use `resourceArns` (exact ARNs) for most resource types — more precise,
  no pre-tagging needed
- Exception — these types do NOT support `resourceArns`, use
  `resourceTags` instead:
  - `aws:elasticache:replicationgroup`
  - `aws:ec2:autoscaling-group`
- EKS pod actions use Kubernetes namespace + pod labels (neither
  `resourceArns` nor `resourceTags`)

**`resourceArns` and `filters` are mutually exclusive.** FIS rejects targets
that specify both. For AZ-scoped targeting, either use `resourceArns` with
only the target AZ's ARNs, or use `resourceTags` + `filters` together.

**If scenario is AZ Power Interruption:** follow
`references/az-power-interruption-guide.md` for sub-action pruning, tagging
strategy, permissions, and one-Stack-per-AZ design.

**Ask the user:**
1. Which AZ to target (for AZ-level scenarios)
2. Which services to include (for AZ Power Interruption) — if user mentions
   specific services, include ONLY those + mandatory infrastructure sub-actions
3. Target resource identifiers (cluster IDs, instance IDs, etc.)

#### For Custom FIS Actions

```bash
aws fis get-action --id "ACTION_ID" --region TARGET_REGION
```

Extract required `targets` and `parameters`. Resolve user-provided
identifiers to ARNs via AWS CLI.

#### For Services Without Native FIS Actions (SSM Automation)

1. Confirm no native action exists:
   ```bash
   aws fis list-actions \
     --query "actions[?starts_with(id, 'aws:{SERVICE}:')]" \
     --region TARGET_REGION
   ```

2. If empty, follow the service-specific guide:
   - Amazon MSK → `references/msk-guide.md`
   - ElastiCache primary node reboot → `references/elasticache-redis-guide.md`
     (Scenario 2)
   - Other services → not yet documented. Stop and inform the user.

**Special case — ElastiCache:** Has a native FIS action for AZ-level impact
(`aws:elasticache:replicationgroup-interrupt-az-power`) but **no native
action for single-node reboot or replication group failover**. For primary
node reboot, use SSM Automation per
`references/elasticache-redis-guide.md` → Scenario 2. For replication group
failover (TestFailover), use SSM Automation per
`references/elasticache-redis-guide.md` → Scenario 3.

3. Discover resources via the target service's CLI (`aws kafka list-clusters`,
   etc.).

### Step 2.5: EKS Pod Action Setup Gate

**If the experiment includes ANY `aws:eks:pod-*` action, complete this gate
BEFORE Step 3.**

Applicable actions: `aws:eks:pod-cpu-stress`, `aws:eks:pod-delete`,
`aws:eks:pod-io-stress`, `aws:eks:pod-memory-stress`,
`aws:eks:pod-network-blackhole-port`, `aws:eks:pod-network-latency`,
`aws:eks:pod-network-packet-loss`.

1. Read the official documentation:
   ```
   aws___read_documentation:
     url: https://docs.aws.amazon.com/fis/latest/userguide/eks-pod-actions.html
   ```

2. Follow ALL requirements in `references/eks-pod-action-guide.md`:
   - Lambda-backed CFN Custom Resource for K8s RBAC (fixed names: `fis-sa`,
     `fis-experiment-role`, `fis-experiment-role-binding`)
   - EKS Access Entry for FIS Experiment Role (`Username: fis-experiment`)
   - Cluster auth mode check (`API_AND_CONFIG_MAP` or `API`)
   - Pod `readOnlyRootFilesystem: false` check
   - Network action limitations (no Fargate, no bridge mode)
   - **Pod memory stress threshold calculation** (if action is
     `aws:eks:pod-memory-stress`) — user's percent is total target, not
     injection value

Do NOT skip. EKS pod actions have complex setup requirements that differ
significantly from other FIS actions.

### Step 3: Validate Resource-Action Compatibility

**CRITICAL GATE.** Before generating any files, verify that the user's
actual resources are compatible with the chosen FIS action(s).

#### 3a. Inspect the Actual Resource

| User Says | CLI Command | Key Fields |
|---|---|---|
| RDS database | `aws rds describe-db-instances --db-instance-identifier {ID}` | `Engine`, `DBClusterIdentifier` |
| RDS/Aurora cluster | `aws rds describe-db-clusters --db-cluster-identifier {ID}` | `Engine`, `EngineMode`, `MultiAZ` |
| EC2 instance | `aws ec2 describe-instances --instance-ids {ID}` | `InstanceType`, `Placement.AvailabilityZone` |
| EKS cluster | `aws eks describe-cluster --name {NAME}` | `accessConfig.authenticationMode`, `version` |
| ElastiCache | `aws elasticache describe-replication-groups --replication-group-id {ID}` | `NodeGroupConfiguration`, `MultiAZ` |
| ASG | `aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names {NAME}` | `AvailabilityZones`, `Instances` |

#### 3b. Cross-Check Against FIS Action Requirements

```bash
aws fis get-action --id "ACTION_ID" --region TARGET_REGION \
  --query 'action.targets' --output json
```

**Common incompatibility traps:**

| FIS Action | Required resourceType | Incompatible With | Detection |
|---|---|---|---|
| `aws:rds:failover-db-cluster` | `aws:rds:cluster` | Standalone RDS (non-Aurora) | `DBClusterIdentifier` is null |
| `aws:rds:reboot-db-instances` | `aws:rds:db` | Aurora clusters | `Engine` starts with `aurora` |
| `aws:elasticache:replicationgroup-interrupt-az-power` | `aws:elasticache:replicationgroup` | Standalone ElastiCache nodes | No replication group |
| `aws:ec2:stop-instances` | `aws:ec2:instance` | Spot instances | `InstanceLifecycle` = `spot` |

#### 3c. Decision Gate

- **Compatible** → proceed to Step 4.
- **Incompatible** → explain the mismatch, suggest alternatives based on
  the actual resource type, ask the user to confirm or abort.

Example alternatives:
- Standalone RDS Multi-AZ → `aws:rds:reboot-db-instances` with
  `--force-failover`
- Aurora cluster → `aws:rds:failover-db-cluster`
- ElastiCache standalone → explain replication group is required

#### 3d. For Scenario Library Scenarios

Validate EACH included sub-action against its target resources. Only
validate sub-actions that remain after service-scoped pruning (Step 2).

### Step 4: Determine Monitoring Configuration

**Stop Conditions — default: `source: "none"` (no alarm).** Only create a
CloudWatch Alarm if the user explicitly provides one.

**Dashboard Metrics — comprehensive, per-service.** Group widgets by
service, 3 widgets per service (availability, performance, errors/latency).
Include only services actually affected by the experiment.

| Service | Metrics |
|---|---|
| EC2 | `StatusCheckFailed`, `CPUUtilization`, `NetworkIn/Out`, `NetworkPacketsIn/Out` |
| RDS/Aurora | `DatabaseConnections`, `ReadLatency`, `WriteLatency`, `AuroraReplicaLag`, `FreeableMemory` |
| EKS | `pod_number_of_running_pods`, `pod_number_of_container_restarts`, `node_cpu_utilization`, `node_memory_utilization` |
| ElastiCache | `ReplicationLag`, `EngineCPUUtilization`, `CurrConnections`, `CacheHitRate`, `Evictions`, `IsMaster` |
| ALB | `HealthyHostCount`, `UnHealthyHostCount`, `HTTPCode_ELB_5XX_Count`, `TargetResponseTime` |
| NLB | `ActiveFlowCount`, `TCP_Client_Reset_Count`, `TCP_Target_Reset_Count` |

### Step 5: Generate Configuration Files

**Create output directory:**

```bash
# ─── Fill in from user's request + references/slug-conventions.md ───
SCENARIO_SLUG="..."         # e.g., pod-delete, az-power-int, rds-failover
TARGET_RESOURCE_ID="..."    # e.g., my-aurora-cluster, i-0abc123def
CONTEXT_NAME=""             # optional (e.g., redis, msk); leave empty if N/A
# ────────────────────────────────────────────────────────────────────

# Derived values (do not edit):
TARGET_SLUG=$(echo "${TARGET_RESOURCE_ID}" | tr '[:upper:]' '[:lower:]' | tr ' :/' '-' | cut -c1-20)
CONTEXT_SLUG=$(echo "${CONTEXT_NAME}" | tr '[:upper:]' '[:lower:]' | tr ' :/' '-' | cut -c1-10)
TIMESTAMP=$(TZ=Asia/Shanghai date +%Y-%m-%d-%H-%M-%S)

if [ -n "${CONTEXT_SLUG}" ]; then
    OUTPUT_DIR="./${TIMESTAMP}-${SCENARIO_SLUG}-${TARGET_SLUG}-${CONTEXT_SLUG}"
else
    OUTPUT_DIR="./${TIMESTAMP}-${SCENARIO_SLUG}-${TARGET_SLUG}"
fi
mkdir -p "${OUTPUT_DIR}"
```

**REQUIRED:** Before generating `cfn-template.yaml`, read the
`AWS::FIS::ExperimentTemplate` CloudFormation resource documentation:

```
aws___read_documentation:
  url: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-fis-experimenttemplate.html
```

**ALSO REQUIRED:** Search for CloudFormation examples for the resources used:

```
aws___search_documentation:
  search_phrase: "<CFN resource types in this experiment>"
  topics: ["cloudformation"]
```

**Generate files:**

1. **cfn-template.yaml** — use `references/cfn-base-template.md` as the
   skeleton. Extend with scenario-specific resources per:
   - `references/az-power-interruption-guide.md` (if AZ Power Interruption)
   - `references/eks-pod-action-guide.md` (if EKS pod actions)
   - `references/msk-guide.md` (if MSK)
   - `references/elasticache-redis-guide.md` (if ElastiCache)

2. **README.md** — use the template in `references/output-format.md`.

### Step 5.5: CFN Permission Pre-Check

Run the precheck script to detect whether a CFN service role is required:

```bash
CFN_ROLE_ARN=$("${SKILL_DIR}/scripts/precheck-cfn-permissions.sh")
```

If the caller lacks CloudFormation permissions, the script exits 1 with
guidance — **stop and inform the user**. Otherwise, `CFN_ROLE_ARN` is either
empty (no service role needed) or contains the required role ARN.

### Step 6: Deploy CFN Template (Self-Healing Loop)

**Generate deployment parameters:**

```bash
# See references/slug-conventions.md for the ExperimentName composition rule
RANDOM_SUFFIX=$(LC_ALL=C tr -dc 'a-z0-9' < /dev/urandom | head -c6)

if [ -n "${CONTEXT_SLUG}" ]; then
    EXPERIMENT_NAME="${SCENARIO_SLUG}-${TARGET_SLUG}-${CONTEXT_SLUG}-${RANDOM_SUFFIX}"
else
    EXPERIMENT_NAME="${SCENARIO_SLUG}-${TARGET_SLUG}-${RANDOM_SUFFIX}"
fi
STACK_NAME="fis-${EXPERIMENT_NAME}"
```

**Deploy with self-healing retry loop** (maximum 5 attempts driven by the
agent). The `deploy-with-retry.sh` script performs **one attempt** — the
agent drives the loop externally. On each attempt:

1. Run `scripts/deploy-with-retry.sh`:
   ```bash
   "${SKILL_DIR}/scripts/deploy-with-retry.sh" \
     "${OUTPUT_DIR}/cfn-template.yaml" \
     "${STACK_NAME}" \
     "${TARGET_REGION}" \
     "${CFN_ROLE_ARN}" \
     "ExperimentName=${EXPERIMENT_NAME}" \
     "RandomSuffix=${RANDOM_SUFFIX}"
   ```
2. Exit 0 → deployment succeeded, proceed to "On Successful Deployment".
3. Exit 1 (validation failed) or 2 (deployment failed, stack deleted) →
   analyze stderr output, fix `cfn-template.yaml`, increment attempt
   counter, re-invoke the script.
4. After 5 failed attempts → stop and report to the user with the last
   error, all fixes attempted, and the current `cfn-template.yaml`.

**Common CFN errors and fixes:**

| Error Pattern | Root Cause | Fix |
|---|---|---|
| `Property validation failure` | Invalid CFN property name/value | Fix the resource property |
| `Template format error` | YAML syntax issue | Fix indentation/structure |
| `Resource type not supported` | Resource unavailable in region | Check regional availability |
| `Circular dependency` | Resources reference each other | Use `DependsOn` or restructure |
| `RoleArn ... is invalid` | IAM role not yet propagated | Add `DependsOn` for IAM role |
| Empty `logConfiguration` | AZ Power Interruption doc artifact | Remove the `logConfiguration` block |

#### On Successful Deployment

1. Extract stack outputs:
   ```bash
   aws cloudformation describe-stacks \
     --stack-name "${STACK_NAME}" \
     --query 'Stacks[0].Outputs' \
     --region "${TARGET_REGION}" --output table
   ```

2. Update `README.md` with actual stack name, template ID, dashboard URL,
   and cleanup command. Replace ALL `{STACK_NAME}` placeholders — do NOT
   leave placeholders in the final output.

### Step 7: Rename Output Directory with Template ID

Run the rename script:

```bash
NEW_OUTPUT_DIR=$("${SKILL_DIR}/scripts/rename-output-dir.sh" \
    "${OUTPUT_DIR}" \
    "${STACK_NAME}" \
    "${TARGET_REGION}")
OUTPUT_DIR="${NEW_OUTPUT_DIR}"
```

Update `README.md`'s `**Directory:**` field with the full absolute path of
the renamed directory. If CFN deployment failed (Step 6 exceeded max
retries), skip this step.

Print a brief summary to the terminal:
- Experiment output directory (with template ID)
- CFN stack name and deployment status
- Experiment template ID
- Next step instruction

## Important Guidelines

- **Scenario Library templates come from documentation.** Call
  `aws___read_documentation` on the scenario's doc URL (Step 1 table) before
  generating any files. The documentation is the only authoritative source.
- **Never start the FIS experiment in this skill.** Starting the experiment
  is handled by `aws-fis-experiment-execute` or manually by the user.
- **Validate resource-action compatibility BEFORE generating files** (Step 3).
  The most common source of wasted effort is deploying a template that
  targets an incompatible resource.
- **Always deploy and validate.** Do not just generate files — deploy the CFN
  template and iterate until it succeeds (Step 6). The user should receive a
  working, deployed experiment template ready to start.
- **Self-heal on CFN errors.** Read stack events, diagnose, fix the template,
  delete the failed stack, retry. Do not ask the user to fix CFN errors.
- **Verify FIS action availability** (`aws fis list-actions` /
  `aws fis get-action`) before generating templates. Don't fabricate action
  IDs.
- **Prefer `resourceArns` over `resourceTags` for targets.** Exceptions:
  `aws:elasticache:replicationgroup`, `aws:ec2:autoscaling-group`. Never
  combine `resourceArns` with `filters`.
- **IAM policy must be least-privilege.** Only include permissions for the
  specific actions in the experiment.
- **CFN template must be self-contained.** Deploy the CFN template and get a
  working experiment without any other steps.
- **Sequential MCP calls.** All `aws___read_documentation` and
  `aws___search_documentation` calls must be sequential, never parallel.
  Retry up to 10 times on rate limit errors.
- **Keep local files in sync.** After successful deployment, update README.md
  with real ARNs and stack outputs.