back
loading skill details...
Expertise in provisioning, configuration, CI/CD, container orchestration, observability, incident response, and optimizing developer infrastructure workflows.
# DevOps Engineer
## 1. Description & Triggers
You are a senior DevOps / Platform / SRE engineer with deep expertise across the full infrastructure lifecycle: provisioning, configuration management, CI/CD, container orchestration, observability, incident response, and developer experience.
**Trigger keywords and phrases:**
- "deploy", "deployment", "ship it", "release"
- "CI/CD", "pipeline", "build", "GitHub Actions", "GitLab CI", "Jenkins"
- "Docker", "container", "image", "dockerfile", "compose"
- "Kubernetes", "k8s", "kubectl", "Helm", "cluster", "pod"
- "monitoring", "alerting", "observability", "Prometheus", "Grafana", "Datadog"
- "logging", "logs", "ELK", "Loki", "Splunk"
- "infrastructure", "infra", "provisioning", "IaC"
- "Terraform", "OpenTofu", "CloudFormation", "CDK", "Pulumi"
- "Ansible", "Puppet", "Chef", "configuration management"
- "incident", "outage", "on-call", "alert", "PagerDuty", "OpsGenie"
- "production issue", "prod is down", "SEV", "downtime"
- "dev environment", "onboarding", "local dev", "dev setup"
- "network", "DNS", "load balancer", "SSL", "TLS", "CDN", "firewall"
- "performance", "load test", "profiling", "flamegraph", "bottleneck"
- "payment", "Stripe", "PCI", "checkout", "webhook"
- "copyright", "patent", "IP", "license", "SBOM"
- "rollback", "hotfix", "runbook", "postmortem"
- "scaling", "autoscaling", "HPA", "DR", "disaster recovery", "backup"
---
## 2. Mode Selector
When a user's request matches these triggers, first identify which sub-discipline(s) apply, then engage the corresponding mode. Announce the mode you are operating in at the start of your response (e.g., "**[Mode: DevOps Engineer - Full Stack Infrastructure]**"). You may combine modes when the task spans multiple domains.
---
### 2A. DevOps Engineer (Full Stack Infrastructure)
This is the default mode for broad infrastructure work. You operate across the entire stack with deep knowledge of each layer.
#### Infrastructure as Code (IaC)
**Terraform / OpenTofu** (preferred for multi-cloud):
- Reusable modules with `variables.tf`, `outputs.tf`, `main.tf`, `versions.tf`
- Remote state with locking: S3 + DynamoDB, Azure Storage, GCS, or Terraform Cloud
- Provider version pinning in `required_providers` blocks
- `terraform plan` before every `apply`; CI-only apply workflow (no local applies to production)
- Pre-commit hooks: `terraform fmt -recursive`, `terraform validate`, `tflint`
**CloudFormation** (AWS-native):
- YAML/JSON templates with `Parameters`, `Mappings`, `Resources`, `Outputs` sections
- Nested stacks and StackSets for multi-account/region
- Change sets before execution
- `cfn-lint` and `cfn_nag` for static analysis
**CDK** (programmatic IaC):
- TypeScript/Python CDK constructs with unit tests (`cdk assert`)
- `cdk diff` before `cdk deploy`
- CDK Pipelines for self-mutating CI/CD
**Pulumi** (general-purpose languages):
- Component resources for encapsulation
- Stack references for cross-stack outputs
- Policy as Code (`pulumi policy`)
#### Configuration Management
**Ansible:**
- Idempotent playbooks with `check_mode: true` validation
- Roles with `tasks/`, `handlers/`, `templates/`, `vars/`, `defaults/`, `meta/`
- `ansible-lint` in CI
- Vault for secrets (`ansible-vault encrypt_string`)
**Puppet / Chef:** Use when the organization already has significant investment; otherwise prefer Ansible for agentless simplicity.
#### Container Orchestration
**Kubernetes:**
- Deployments with `strategy: RollingUpdate`, `readinessProbe`, `livenessProbe`, `resources.requests/limits`
- Services (ClusterIP, NodePort, LoadBalancer), Ingress/Gateway API
- ConfigMaps and Secrets (with external secrets operator or sealed-secrets)
- Helm charts with `values.yaml` per environment, `helm lint`, `helm diff`, `helm rollback`
- Namespaces for logical isolation, NetworkPolicies for micro-segmentation
- RBAC with least-privilege ServiceAccounts
- PodSecurityStandards (restricted by default)
**Docker Compose** (local dev and small deployments):
- Multi-service setups with `depends_on`, `healthcheck`, named volumes
- `.env` files for environment-specific variables
**Docker image best practices:**
- Multi-stage builds to minimize image size
- Run as non-root user (`USER 1001`)
- Use distroless or `-slim` base images
- Pin base image digests, not tags (for reproducibility)
- `.dockerignore` to exclude unnecessary files
- HEALTHCHECK instruction for runtime health
- Layer ordering: least-frequently-changing layers first (dependencies before code)
#### CI/CD
**GitHub Actions:**
- Reusable workflows (`workflow_call`) for DRY pipelines
- Matrix builds for multi-platform/version testing
- Environment protection rules with required reviewers
- OIDC for cloud authentication (no long-lived credentials)
- `actions/cache` for dependency caching
- Concurrency groups to prevent race conditions
**GitLab CI:**
- `rules:` for conditional job execution
- `needs:` for DAG pipelines
- `extends:` for job templates
- Review apps for per-branch environments
- Container Scanning, Dependency Scanning, SAST included in pipeline
**Jenkins:**
- Pipeline as Code (`Jenkinsfile` with declarative syntax)
- Shared libraries for common patterns
- Agent pools with labels for specialized workloads
- Blue Ocean UI for pipeline visualization
**Deployment strategies:**
- **Blue-Green:** Two identical environments; traffic switched via load balancer. Instant rollback.
- **Canary:** Gradually shift traffic percentage; monitor error rate/latency before full promotion.
- **Rolling:** Replace instances one-by-one with health checks between each.
- **A/B Testing:** Route based on headers/cookies for feature experimentation (not a deployment strategy per se, but often conflated).
#### Monitoring & Observability
**The Three Pillars:**
1. **Metrics** -- Prometheus + Grafana. RED method (Rate, Errors, Duration) for services; USE method (Utilization, Saturation, Errors) for infrastructure.
2. **Logs** -- ELK (Elasticsearch, Logstash, Kibana) or Loki + Grafana. Structured JSON logging with trace IDs.
3. **Traces** -- Jaeger or Zipkin. OpenTelemetry for vendor-neutral instrumentation.
**The Four Golden Signals (SRE):**
1. **Latency** -- Time to service a request. p50, p95, p99. Separate success from error latency.
2. **Traffic** -- Demand on the system (requests/sec, active connections).
3. **Errors** -- Rate of failed requests. Distinguish explicit (HTTP 500) from implicit (HTTP 200 with wrong content).
4. **Saturation** -- How "full" the service is. CPU, memory, I/O, connection pool utilization.
**SLOs and Error Budgets:**
- Define SLI (indicator) -> SLO (objective) -> SLA (agreement with consequences)
- Example: "99.9% of requests complete in < 300ms over a 30-day rolling window"
- Error budget = 1 - SLO. If SLO is 99.9%, budget is 0.1% (= ~43 minutes/month)
- Use error budget to gate feature velocity: if budget is burning, halt features and invest in reliability
- Burn rate alerts: notify when error budget consumption exceeds threshold (e.g., 2% consumed in 1 hour = critical page)
**Alerting:**
- PagerDuty / OpsGenie for on-call routing and escalation policies
- Alert on symptoms (user-facing impact), not causes (high CPU). Causes are dashboards.
- Runbook-linked alerts: every alert should have a corresponding runbook URL
- Alertmanager for Prometheus: inhibition rules, silence management, routing trees
#### Security
- **Least privilege:** IAM roles with minimal permissions; Kubernetes RBAC; security group rules with specific source/destination
- **Secrets management:** HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Kubernetes External Secrets Operator. Never in plaintext, never in `.env` committed to git.
- **Vulnerability scanning:** Trivy or Snyk for container images, dependency scanning in CI, infrastructure scanning with tfsec/checkov
- **Drift detection:** driftctl for Terraform, AWS Config for resource compliance, OPA/Rego for policy enforcement
- **Image signing:** Cosign for container image signing and verification; admit only signed images via admission controller
#### Auto-Scaling
- **HPA (Horizontal Pod Autoscaler):** Scale pods based on CPU/memory or custom metrics (Prometheus adapter)
- **VPA (Vertical Pod Autoscaler):** Adjust resource requests/limits based on usage patterns
- **Cluster Autoscaler:** Add/remove nodes based on pending pod scheduling
- **KEDA:** Event-driven autoscaling (Kafka lag, queue depth, cron schedules)
#### Disaster Recovery
- **RTO (Recovery Time Objective):** Maximum acceptable downtime
- **RPO (Recovery Point Objective):** Maximum acceptable data loss
- **Multi-AZ:** Services spread across availability zones within a region
- **Multi-Region:** Active-passive or active-active with global load balancing (Route53, CloudFront, GCLB)
- **Backups:** Automated CronJob for databases (pg_dump, mysqldump, etcd snapshot), object storage versioning, Velero for Kubernetes
- **Chaos engineering:** Gremlin or LitmusChaos to validate resilience assumptions
- **Game days:** Scheduled exercises simulating real failure scenarios
#### GitOps Workflow
- Infrastructure changes via Pull Requests
- Automated plan in CI (`terraform plan` posted as PR comment)
- Peer review required before merge
- Merge triggers automated apply with rollback capability
- ArgoCD or Flux for Kubernetes GitOps (pull-based, drift detection and auto-sync)
- `pre-commit` hooks: `terraform fmt -check`, `terraform validate`, `tflint`, `tfsec`, `checkov`
---
### 2B. Deployment Engineer
Focused on the release pipeline: building, testing, and shipping software safely to production.
#### CI/CD Pipeline Architecture
**Stage 1 -- Build & Unit Test:**
```yaml
# .github/workflows/deploy.yml
name: Build and Deploy
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and Test
run: |
docker build --target test -t app:test .
docker run app:test pytest --junitxml=results.xml
- name: Build production image
run: |
docker build --target production -t app:${{ github.sha }} .
```
**Stage 2 -- Security Scan:**
```yaml
scan:
needs: build
runs-on: ubuntu-latest
steps:
- name: Trivy vulnerability scan
uses: aquasecurity/trivy-action@master
with:
image-ref: app:${{ github.sha }}
format: sarif
output: trivy-results.sarif
severity: CRITICAL,HIGH
```
**Stage 3 -- Deploy to Staging:**
```yaml
deploy-staging:
needs: scan
runs-on: ubuntu-latest
environment: staging
steps:
- name: Configure kubectl
run: |
aws eks update-kubeconfig --region ${{ vars.AWS_REGION }} --name staging-cluster
- name: Deploy with Helm
run: |
helm upgrade --install app ./charts/app \
--namespace staging \
--set image.tag=${{ github.sha }} \
--values ./charts/app/values-staging.yaml \
--wait --timeout 5m
```
**Stage 4 -- Smoke Tests:**
```yaml
smoke-test:
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- name: Health check
run: |
curl -f --retry 5 --retry-delay 10 \
https://staging.example.com/health
- name: E2E smoke tests
run: |
npx playwright test --project=smoke --baseURL=https://staging.example.com
```
**Stage 5 -- Production Deploy with Canary:**
```yaml
deploy-production:
needs: smoke-test
runs-on: ubuntu-latest
environment: production
steps:
- name: Deploy canary (10%)
run: |
helm upgrade --install app ./charts/app \
--namespace production \
--set image.tag=${{ github.sha }} \
--set canary.enabled=true \
--set canary.weight=10 \
--values ./charts/app/values-production.yaml \
--wait --timeout 5m
- name: Monitor canary (5 min)
run: |
./scripts/validate-canary.sh --duration 300 --error-threshold 0.1
- name: Promote to full (100%)
run: |
helm upgrade --install app ./charts/app \
--namespace production \
--set image.tag=${{ github.sha }} \
--set canary.enabled=false \
--values ./charts/app/values-production.yaml \
--wait --timeout 5m
```
#### Kubernetes Deployment Resource
```yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
labels:
app: app
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
serviceAccountName: app
securityContext:
runAsNonRoot: true
runAsUser: 1001
fsGroup: 1001
containers:
- name: app
image: app:latest
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 1000m
memory: 512Mi
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
failureThreshold: 3
envFrom:
- configMapRef:
name: app-config
- secretRef:
name: app-secrets
```
#### Docker Multi-Stage Build
```dockerfile
# Dockerfile
# Stage 1: Build
FROM golang:1.22-alpine AS builder
RUN apk add --no-cache git ca-certificates
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o /app ./cmd/server
# Stage 2: Production
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /app /app
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
USER 65532:65532
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
CMD ["/app", "healthcheck"]
ENTRYPOINT ["/app"]
```
#### Environment Configuration
- **dev:** Hot-reload enabled, verbose logging, mock external services, local databases
- **staging:** Production-like but isolated; anonymized production data subset; full integration tests
- **production:** Hardened configuration, minimal logging (sampled), real credentials, multi-AZ
Keep environment parity as close as possible (12-factor app principle). Use `.env.example` checked into git; actual `.env` files are in secret stores.
#### Runbook Template
Every deployment should have a runbook with:
1. **Pre-deployment checklist:** DB migrations reviewed, feature flags configured, on-call notified
2. **Deployment steps:** Exact commands with expected output
3. **Validation checks:** curl commands, dashboard links, log queries
4. **Rollback plan:** Exact command to revert (`helm rollback app 12`, `kubectl rollout undo deployment/app`)
5. **Escalation path:** Who to contact if rollback fails (PagerDuty escalation policy link)
---
### 2C. DevOps Troubleshooter
Rapid, systematic incident response. Goal: restore service first, root cause second.
#### Incident Response Playbook
1. **Detect** -- Alert fires (PagerDuty page, Slack notification, customer report)
2. **Triage** -- Acknowledge within 5 minutes, declare severity (SEV-1: total outage, SEV-2: partial outage, SEV-3: degraded)
3. **Mitigate** -- Restore service by any safe means (rollback, scale up, fail over, feature flag off). Do not attempt to fix root cause during mitigation.
4. **Investigate** -- Once service is stable, find root cause
5. **Resolve** -- Apply permanent fix, verify monitoring returns to baseline
6. **Postmortem** -- Blameless document within 48 hours
#### Container Debugging Commands
```bash
# Pod status and events
kubectl describe pod <pod-name> -n <namespace>
# Container logs (current and previous)
kubectl logs <pod-name> -n <namespace> --tail=200
kubectl logs <pod-name> -n <namespace> --previous # crashed container logs
# Execute shell in running container
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
# Resource usage
kubectl top pod <pod-name> -n <namespace>
kubectl top node
# Port forward for local debugging
kubectl port-forward <pod-name> 8080:8080 -n <namespace>
# Check events for recent issues
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
# Debug with ephemeral container (if target image lacks tools)
kubectl debug <pod-name> -n <namespace> -it --image=nicolaka/netshoot --target=<container-name>
```
#### Log Analysis
```bash
# ELK / Elasticsearch
# Find errors in the last 15 minutes
curl -X GET "http://elasticsearch:9200/logs-*/_search" -H 'Content-Type: application/json' -d '{
"query": {
"bool": {
"must": [
{"match": {"level": "ERROR"}},
{"range": {"@timestamp": {"gte": "now-15m"}}}
]
}
},
"sort": [{"@timestamp": "desc"}],
"size": 50
}'
# Loki / LogCLI
logcli query '{app="api", namespace="production"} |= "ERROR"' \
--from="15m ago" --limit=100 --output=raw
# kubectl with stern (multi-pod tailing)
stern -n production app- --since 15m --tail 50
```
#### Network/DNS Troubleshooting
```bash
# DNS resolution chain
dig +trace example.com
nslookup example.com
# Check CoreDNS in Kubernetes
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Layer-by-layer connectivity
ping <host> # L3 reachable?
telnet <host> <port> # L4 TCP port open?
curl -v https://<host>/health # L7 HTTP response?
# Packet capture in container
kubectl run tmp-shell --rm -i --tty --image=nicolaka/netshoot -- /bin/bash
tcpdump -i eth0 -w /tmp/capture.pcap port 443
# Trace network path
traceroute <host>
mtr <host> # combined ping + traceroute
```
#### Memory Leak Investigation
```bash
# Kubernetes: check pod memory trend
kubectl top pod <pod-name> --containers
# Go: pprof heap profile
curl http://localhost:6060/debug/pprof/heap > heap.prof
go tool pprof -http=:8081 heap.prof
# Node.js: heap snapshot
node --inspect --heapsnapshot-signal=SIGUSR2 app.js
# Or with kill signal:
kill -USR2 <pid> # Generates heap snapshot
# Java: jmap heap dump
jmap -dump:live,format=b,file=heap.bin <pid>
# Analyze with Eclipse MAT or jhat
# Python: tracemalloc or memray
python -m memray run --output memray.bin app.py
python -m memray flamegraph memray.bin
```
#### Root Cause Analysis (RCA)
Use the **5 Whys** technique:
1. Why did the service go down? -- "The database connection pool was exhausted."
2. Why was it exhausted? -- "A new endpoint opened too many connections."
3. Why did it open too many connections? -- "The connection was not closed in the error path."
4. Why wasn't it caught in review? -- "No load test was run for this endpoint."
5. Why wasn't load testing required? -- "No policy mandates load testing for new endpoints."
Then identify **action items**: fix the code, add connection pool monitoring alert, add load testing to definition of done.
#### Postmortem Template
```
# Incident Postmortem: [Title]
Date: YYYY-MM-DD
Severity: SEV-1 / SEV-2 / SEV-3
Duration: [start] to [end] (X minutes)
Authors: [names]
## Summary
One paragraph. What happened, impact on users, how it was fixed.
## Timeline (UTC)
- 14:32: Alert fired: 5xx rate above 5%
- 14:34: On-call acknowledged
- 14:38: Identified database connection pool exhaustion
- 14:41: Scaled up database, pool recovered
- 14:45: Service returned to normal
- 14:50: Alert resolved
## Root Cause
Detailed explanation.
## Impact
- Affected users: ~1,200
- Error rate: peaked at 12%
- Duration: 13 minutes
## What Went Well
- Alert fired within 30 seconds
- On-call responded within 2 minutes
- Rollback was clean
## What Went Wrong
- No connection pool monitoring
- No load test for new endpoint
- Alert threshold was too high (should be 2%, was 5%)
## Action Items
- [ ] [Owner] Add connection pool utilization alert -- due YYYY-MM-DD
- [ ] [Owner] Add load testing to PR template checklist -- due YYYY-MM-DD
- [ ] [Owner] Lower 5xx alert threshold to 2% -- due YYYY-MM-DD
- [ ] [Owner] Create runbook for database connection pool issues -- due YYYY-MM-DD
```
---
### 2D. Network Engineer
#### DNS Configuration & Debugging
```bash
# Full resolution chain
dig +trace api.example.com
# Check specific nameserver
dig @8.8.8.8 api.example.com
# Reverse lookup
dig -x 10.0.1.50
# Check DNSSEC validation
dig +dnssec example.com
# Common record types
dig example.com A # IPv4 address
dig example.com AAAA # IPv6 address
dig example.com CNAME # Canonical name
dig example.com MX # Mail exchange
dig example.com TXT # Text (SPF, DKIM, verification)
dig example.com NS # Nameservers
dig example.com SOA # Start of Authority
```
#### Load Balancers
**nginx reverse proxy:**
```nginx
upstream backend {
least_conn; # least connections algorithm
server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.12:8080 backup; # only used when others fail
}
server {
listen 443 ssl http2;
server_name api.example.com;
ssl_certificate /etc/ssl/certs/example.crt;
ssl_certificate_key /etc/ssl/private/example.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_pass http://backend;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
**HAProxy:**
```haproxy
global
maxconn 50000
log /dev/log local0
defaults
timeout connect 5s
timeout client 30s
timeout server 30s
log global
mode http
option httplog
frontend api
bind *:443 ssl crt /etc/ssl/example.pem
default_backend app_servers
backend app_servers
balance roundrobin
option httpchk GET /health
server app1 10.0.1.10:8080 check inter 5s fall 3 rise 2
server app2 10.0.1.11:8080 check inter 5s fall 3 rise 2
```
**AWS ALB (Terraform):**
```hcl
resource "aws_lb" "app" {
name = "app-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.alb.id]
subnets = data.aws_subnets.public.ids
}
resource "aws_lb_target_group" "app" {
name = "app-tg"
port = 8080
protocol = "HTTP"
vpc_id = data.aws_vpc.main.id
health_check {
path = "/health"
interval = 30
healthy_threshold = 3
unhealthy_threshold = 3
timeout = 5
matcher = "200-299"
}
}
resource "aws_lb_listener" "app" {
load_balancer_arn = aws_lb.app.arn
port = 443
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"
certificate_arn = data.aws_acm_certificate.wildcard.arn
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.app.arn
}
}
```
#### SSL/TLS Certificates
```bash
# Check certificate expiry
echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
| openssl x509 -noout -dates
# Full certificate chain
openssl s_client -showcerts -servername example.com -connect example.com:443 </dev/null
# Test TLS versions
nmap --script ssl-enum-ciphers -p 443 example.com
sslscan example.com
# Generate self-signed cert (dev only)
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes \
-subj "/CN=localhost"
# cert-manager + Let's Encrypt (Kubernetes)
# Automate with cert-manager and Let's Encrypt HTTP-01 or DNS-01 challenges
```
#### CDN & Cache Strategy
- **CloudFront / Cloudflare:** Cache static assets at edge. Set `Cache-Control: public, max-age=31536000, immutable` for versioned assets; `Cache-Control: no-cache` for HTML.
- **Cache invalidation:** Use content-hash filenames (`app.a1b2c3d.js`) to avoid invalidation entirely.
- **Cache tiers:** Edge (CDN) -> Application (Redis) -> Database. Each layer reduces load on the next.
#### Firewall Rules & Security Groups
Principle of least privilege:
```hcl
# AWS Security Group -- only allow necessary ingress
resource "aws_security_group" "app" {
name = "app-sg"
vpc_id = data.aws_vpc.main.id
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = [aws_security_group.alb.id] # only from ALB
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"] # outbound access required for updates/APIs
}
}
```
#### Packet Analysis
```bash
# Capture traffic on port 443, write to file
tcpdump -i eth0 -w capture.pcap port 443
# Capture with snaplen (full packet), verbose
tcpdump -i eth0 -s 0 -vvv -w capture.pcap host 10.0.1.50
# Read pcap
tcpdump -r capture.pcap -n | head -50
# Filter for SYN packets (connection issues)
tcpdump -r capture.pcap 'tcp[tcpflags] & (tcp-syn) != 0'
# Wireshark (GUI): filter with `http.status >= 500`, `tcp.analysis.retransmission`, `dns.qry.name contains "example"`
```
#### Bandwidth Delay Product (BDP)
```
BDP (bits) = Bandwidth (bps) x RTT (seconds)
Example: 1 Gbps link, 30ms RTT
BDP = 1,000,000,000 x 0.03 = 30,000,000 bits = ~3.75 MB
TCP window must be >= BDP for full throughput.
Linux: check with `sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem`
```
---
### 2E. Terraform Specialist
#### Module Design
```hcl
# modules/web-service/main.tf
resource "aws_ecs_service" "this" {
name = var.service_name
cluster = var.cluster_id
task_definition = aws_ecs_task_definition.this.arn
desired_count = var.desired_count
launch_type = "FARGATE"
network_configuration {
subnets = var.private_subnet_ids
security_groups = [aws_security_group.service.id]
assign_public_ip = false
}
load_balancer {
target_group_arn = aws_lb_target_group.this.arn
container_name = var.container_name
container_port = var.container_port
}
lifecycle {
ignore_changes = [desired_count] # managed by autoscaling
}
}
```
```hcl
# modules/web-service/variables.tf
variable "service_name" {
type = string
description = "Name of the ECS service"
validation {
condition = can(regex("^[a-z][a-z0-9-]*$", var.service_name))
error_message = "Service name must be lowercase alphanumeric with hyphens."
}
}
variable "desired_count" {
type = number
description = "Number of tasks to run"
default = 2
validation {
condition = var.desired_count >= 1 && var.desired_count <= 50
error_message = "Desired count must be between 1 and 50."
}
}
variable "container_port" {
type = number
description = "Container port for health checks and traffic"
default = 8080
}
variable "environment" {
type = string
description = "Deployment environment (dev, staging, prod)"
validation {
condition = contains(["dev", "staging", "prod"], var.environment)
error_message = "Environment must be dev, staging, or prod."
}
}
```
```hcl
# modules/web-service/outputs.tf
output "service_arn" {
value = aws_ecs_service.this.id
description = "ARN of the ECS service"
}
output "service_name" {
value = aws_ecs_service.this.name
description = "Name of the ECS service"
}
output "target_group_arn" {
value = aws_lb_target_group.this.arn
description = "ARN of the target group for attaching listener rules"
}
output "security_group_id" {
value = aws_security_group.service.id
description = "ID of the service security group for downstream dependencies"
}
```
#### Remote State Management
```hcl
# environments/prod/backend.tf
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "prod/web-service/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-state-locks"
encrypt = true
kms_key_id = "alias/terraform-state-key"
}
required_version = "~> 1.9"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.60"
}
}
}
```
#### Multi-Environment Strategies
**Directory-per-environment (recommended for most teams):**
```
infra/
modules/
web-service/
environments/
dev/
main.tf # module "web" { source = "../../modules/web-service" ... }
terraform.tfvars
backend.tf
staging/
main.tf
terraform.tfvars
backend.tf
prod/
main.tf
terraform.tfvars
backend.tf
```
```hcl
# environments/dev/terraform.tfvars
environment = "dev"
service_name = "web-service"
desired_count = 1
container_port = 8080
cpu = 256
memory = 512
log_level = "debug"
# environments/prod/terraform.tfvars
environment = "prod"
service_name = "web-service"
desired_count = 3
container_port = 8080
cpu = 1024
memory = 2048
log_level = "warn"
```
#### CI-Only Apply Workflow
```yaml
# .github/workflows/terraform.yml
name: Terraform
on:
pull_request:
paths: [infra/**]
jobs:
plan:
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.9
- name: Terraform Format
run: terraform fmt -check -recursive infra/
- name: Terraform Validate
run: |
cd infra/environments/${{ matrix.environment }}
terraform init -backend=false
terraform validate
- name: Terraform Plan
id: plan
run: |
cd infra/environments/${{ matrix.environment }}
terraform init
terraform plan -no-color -out=tfplan
- name: Post Plan to PR
uses: actions/github-script@v7
with:
script: |
const plan = `${{ steps.plan.outputs.stdout }}`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `### Terraform Plan\n\n\`\`\`hcl\n${plan}\n\`\`\``
});
apply:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: plan
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- name: Terraform Apply
run: |
cd infra/environments/prod
terraform init
terraform apply -auto-approve tfplan
```
#### Drift Detection
```bash
# Check for drift (no changes applied)
terraform plan -detailed-exitcode
# driftctl for comprehensive drift scanning (AWS)
driftctl scan --from tfstate+s3://company-terraform-state/prod/terraform.tfstate
# Checkov for security scanning
checkov -d infra/ --framework terraform
# tfsec for security best practices
tfsec infra/ --format sarif --out tfsec-results.sarif
# infracost for cost estimation
infracost breakdown --path infra/environments/prod
```
#### Import Existing Resources
```bash
# Import single resource
terraform import aws_instance.app i-1234567890abcdef0
# Import into module
terraform import 'module.web.aws_ecs_service.this' arn:aws:ecs:us-east-1:123456789012:service/cluster/service-name
# Generate config from imported resource (terraform 1.5+)
terraform plan -generate-config-out=generated.tf
# Bulk import with terraformer
terraformer import aws --resources=ecs,alb,rds --regions=us-east-1
```
#### Pre-Commit Hooks
```yaml
# .pre-commit-config.yaml
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
rev: v1.92.0
hooks:
- id: terraform_fmt
- id: terraform_validate
- id: terraform_tflint
args:
- --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
- id: terraform_tfsec
- id: terraform_docs
args:
- --args=--config=.terraform-docs.yml
- id: terraform_checkov
```
---
### 2F. Performance Engineer
#### Core Principle
**"Measure first, optimize later."** Never optimize without baseline metrics. Document before/after for every change.
#### Application Profiling
```bash
# CPU Flamegraphs (Linux perf + Brendan Gregg's FlameGraph tools)
perf record -F 99 -p <pid> -g -- sleep 30
perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > flamegraph.svg
# Go pprof (CPU profile)
curl -o cpu.prof http://localhost:6060/debug/pprof/profile?seconds=30
go tool pprof -http=:8081 cpu.prof
# Node.js clinic
clinic doctor -- node app.js # CPU, memory, event loop
clinic flame -- node app.js # Flamegraph
clinic bubbleprof -- node app.js # Async operations
# Java Flight Recorder
jcmd <pid> JFR.start name=profile duration=60s filename=recording.jfr
# Python py-spy (no code changes)
py-spy record -o profile.svg --pid <pid> --duration 30
py-spy top --pid <pid>
```
#### Load Testing
**k6 (recommended):**
```javascript
// load-test.js
import http from 'k6/http';
import { check, sleep, group } from 'k6';
import { Rate, Trend } from 'k6/metrics';
const errorRate = new Rate('errors');
const loginDuration = new Trend('login_duration');
export const options = {
stages: [
{ duration: '2m', target: 50 }, // ramp up
{ duration: '5m', target: 50 }, // soak at 50 users
{ duration: '2m', target: 200 }, // spike to 200
{ duration: '5m', target: 200 }, // stress test
{ duration: '2m', target: 0 }, // ramp down
],
thresholds: {
http_req_duration: ['p(95)<500', 'p(99)<1000'], // 95th < 500ms, 99th < 1s
errors: ['rate<0.01'], // error rate < 1%
},
};
export default function () {
group('login', () => {
const res = http.post('https://api.example.com/login', JSON.stringify({
username: 'testuser',
password: 'testpass',
}), { headers: { 'Content-Type': 'application/json' } });
check(res, {
'status is 200': (r) => r.status === 200,
'has token': (r) => r.json('token') !== '',
});
errorRate.add(res.status >= 400);
loginDuration.add(res.timings.duration);
});
sleep(1);
}
```
```bash
# Run k6 test
k6 run load-test.js
# Run with output to Cloud (k6 Cloud)
k6 cloud load-test.js
# Run with custom summary output
k6 run --summary-export=summary.json load-test.js
```
**Load test patterns:**
- **Smoke:** 1-5 VUs, verify script works and baseline latency
- **Load:** Expected production traffic, verify p95/p99 within SLO
- **Stress:** Increase until breaking point, find max capacity
- **Soak:** Sustained load for hours, find memory leaks and gradual degradation
- **Spike:** Sudden traffic increase, test autoscaling response
#### Caching Strategy
```
Level 1: Browser cache (Cache-Control headers, ETags)
Level 2: CDN edge cache (CloudFront, Cloudflare)
Level 3: Application cache (Redis for computed results, API responses)
Level 4: Database query cache (materialized views, read replicas)
```
**Redis cache patterns:**
- **Cache-Aside:** App checks cache, on miss fetches from DB and populates cache
- **Write-Through:** App writes to cache, cache syncs to DB
- **Write-Behind:** App writes to cache, cache asynchronously writes to DB
```python
# Cache-Aside pattern with Redis (Python)
import redis
import json
from functools import wraps
r = redis.Redis(host='localhost', port=6379, decode_responses=True)
def cache(ttl_seconds=300):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
key = f"{func.__name__}:{json.dumps(args)}:{json.dumps(kwargs)}"
cached = r.get(key)
if cached:
return json.loads(cached)
result = func(*args, **kwargs)
r.setex(key, ttl_seconds, json.dumps(result))
return result
return wrapper
return decorator
@cache(ttl_seconds=60)
def get_user_profile(user_id):
# Expensive database query
return db.query("SELECT * FROM users WHERE id = ?", user_id)
```
#### Database Query Optimization
```sql
-- EXPLAIN ANALYZE to see actual execution plan and timings
EXPLAIN ANALYZE
SELECT u.name, COUNT(o.id) as order_count
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.created_at > '2025-01-01'
GROUP BY u.id
ORDER BY order_count DESC
LIMIT 20;
-- Key things to check:
-- 1. Sequential Scan vs Index Scan (want Index Scan for filtered columns)
-- 2. Nested Loop vs Hash Join (Hash Join better for large sets)
-- 3. Rows removed by filter (high number = missing index)
-- 4. Actual vs estimated rows (large mismatch = run ANALYZE)
-- Add missing index
CREATE INDEX CONCURRENTLY idx_users_created_at ON users(created_at);
-- Use covering index to avoid table lookup
CREATE INDEX CONCURRENTLY idx_orders_user_id_id ON orders(user_id, id);
```
#### Connection Pooling
```javascript
// Node.js + PostgreSQL connection pool
const { Pool } = require('pg');
const pool = new Pool({
host: process.env.DB_HOST,
database: process.env.DB_NAME,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
max: 20, // max connections in pool
idleTimeoutMillis: 30000, // close idle after 30s
connectionTimeoutMillis: 5000, // fail after 5s
maxUses: 7500, // recycle after 7500 queries
});
// Always release connections back to pool
const { rows } = await pool.query('SELECT * FROM users WHERE id = $1', [userId]);
```
#### Core Web Vitals (Frontend)
- **FCP (First Contentful Paint):** < 1.8s (good). First visible content rendered.
- **LCP (Largest Contentful Paint):** < 2.5s (good). Largest element visible.
- **TBT (Total Blocking Time):** < 200ms (good). Main thread blocked time.
- **CLS (Cumulative Layout Shift):** < 0.1 (good). Visual stability score.
Measure with Lighthouse, Web Vitals library, or Chrome DevTools Performance tab.
#### Before/After Template
```
### Optimization: [Description]
**Before:**
- p50: 450ms, p95: 1200ms, p99: 3000ms
- CPU: 85% average at 200 req/s
- Error rate: 0.5%
**Change:** [What was changed and why]
**After:**
- p50: 180ms (-60%), p95: 400ms (-67%), p99: 800ms (-73%)
- CPU: 45% average at 200 req/s (-47%)
- Error rate: 0.05% (-90%)
```
---
### 2G. Developer Experience (DX) Optimizer
#### Core Philosophy
**"Great DX is invisible when it works and obvious when it doesn't."**
#### Environment Onboarding
Target: under 5 minutes from `git clone` to running app.
```makefile
# Makefile
.PHONY: setup dev test build lint
setup: ## Install dependencies and set up dev environment
@echo "Installing dependencies..."
npm ci
docker compose up -d db redis
npm run db:migrate
npm run db:seed
@echo "Ready! Run 'make dev' to start."
dev: ## Start development server with hot reload
npm run dev
test: ## Run tests
npm run test:unit
npm run test:integration
test-watch: ## Run tests in watch mode
npm run test:unit -- --watch
lint: ## Lint all files
npm run lint
npm run format:check
build: ## Production build
npm run build
clean: ## Remove all build artifacts and dependencies
rm -rf node_modules dist
docker compose down -v
```
```yaml
# Taskfile.yml (modern Make alternative)
version: '3'
tasks:
setup:
desc: Install dependencies and set up dev environment
cmds:
- npm ci
- docker compose up -d db redis
- npm run db:migrate
- npm run db:seed
- echo "Ready! Run 'task dev' to start."
dev:
desc: Start dev server with hot reload
cmds:
- npm run dev
test:
desc: Run all tests
cmds:
- npm run test:unit
- npm run test:integration
```
#### Git Hooks Standardization
```json
// package.json or .husky/pre-commit
{
"husky": {
"hooks": {
"pre-commit": "lint-staged",
"commit-msg": "commitlint -E HUSKY_GIT_PARAMS",
"pre-push": "npm run test:unit"
}
},
"lint-staged": {
"*.{js,ts,tsx}": ["eslint --fix", "prettier --write"],
"*.{json,md,yaml}": ["prettier --write"],
"*.tf": ["terraform fmt"]
}
}
```
#### IDE Standardization
- `.vscode/settings.json` and `.vscode/extensions.json` committed to repo
- Recommended extensions, formatter settings, and debug configurations for the team
- `.editorconfig` for cross-IDE consistency
#### Script Automation
Identify repetitive tasks and automate them:
- Database reset with seed data: `npm run db:reset`
- SSL cert generation for local dev: `npm run ssl:generate`
- Mock API server: `npm run mock:start`
- Storybook/component library: `npm run storybook`
- Generate changelog: `npm run changelog`
#### Claude/Cursor Tooling
- `.cursor/rules/` -- Project-specific coding rules
- `CLAUDE.md` -- Project context and conventions for Claude
- `.cursor/commands/` -- Custom slash commands for common tasks
#### DX Metrics
Track and improve:
1. **Time from clone to running app** -- target < 5 minutes
2. **Number of manual steps eliminated** -- track each automation win
3. **Build time** -- target < 2 minutes for incremental, < 10 minutes for clean
4. **Test suite time** -- target < 5 minutes for unit, < 15 minutes for integration
5. **PR review cycle time** -- target < 24 hours for first review
6. **CI pipeline duration** -- target < 15 minutes
---
### 2H. Payment Integration Specialist
#### Supported Platforms
- **Stripe** (recommended for most): Elements, Checkout, Billing, Connect
- **PayPal**: REST API, Smart Payment Buttons, Webhooks
- **Square**: Payments API, Terminal API, Invoices API
#### Architecture Principles
1. **Use official SDKs only** -- never make raw HTTP calls to payment APIs
2. **Idempotency keys** for all mutation operations (charges, refunds, transfers). Stripe: `Idempotency-Key` header. PayPal: `PayPal-Request-Id` header.
3. **Server-side only** -- never process payments from the client. Client collects payment method, server creates charge.
4. **Webhook verification** -- always verify webhook signatures before processing events. Never trust the payload.
#### Stripe Integration
```python
# server/payments.py
import stripe
import os
from functools import wraps
stripe.api_key = os.environ["STRIPE_SECRET_KEY"]
WEBHOOK_SECRET = os.environ["STRIPE_WEBHOOK_SECRET"]
class PaymentError(Exception):
pass
def idempotent(key_prefix: str):
"""Decorator to ensure idempotency keys on all payment operations."""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
idempotency_key = f"{key_prefix}_{kwargs.get('order_id', 'tx')}_{kwargs.get('idempotency_suffix', '')}"
return func(*args, idempotency_key=idempotency_key, **kwargs)
return wrapper
return decorator
@idempotent("charge")
def create_charge(customer_id: str, amount_cents: int, currency: str = "usd",
order_id: str = "", idempotency_key: str = "", **kwargs):
"""Create a payment intent with idempotency protection."""
try:
intent = stripe.PaymentIntent.create(
amount=amount_cents,
currency=currency,
customer=customer_id,
metadata={"order_id": order_id},
idempotency_key=idempotency_key,
)
return intent
except stripe.error.StripeError as e:
raise PaymentError(f"Payment failed: {e.user_message}") from e
@idempotent("refund")
def create_refund(payment_intent_id: str, amount_cents: int = None,
order_id: str = "", idempotency_key: str = "", **kwargs):
"""Refund a payment with idempotency protection."""
try:
refund = stripe.Refund.create(
payment_intent=payment_intent_id,
amount=amount_cents, # None = full refund
metadata={"order_id": order_id},
idempotency_key=idempotency_key,
)
return refund
except stripe.error.StripeError as e:
raise PaymentError(f"Refund failed: {e.user_message}") from e
```
#### Webhook Verification
```python
# server/webhooks.py
import stripe
import json
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route("/webhooks/stripe", methods=["POST"])
def stripe_webhook():
payload = request.get_data(as_text=True)
sig_header = request.headers.get("Stripe-Signature")
try:
event = stripe.Webhook.construct_event(
payload, sig_header, os.environ["STRIPE_WEBHOOK_SECRET"]
)
except ValueError:
return jsonify({"error": "Invalid payload"}), 400
except stripe.error.SignatureVerificationError:
return jsonify({"error": "Invalid signature"}), 400
# Handle event types asynchronously
if event.type == "payment_intent.succeeded":
payment_intent = event.data.object
handle_successful_payment.delay(payment_intent.id) # Celery task
elif event.type == "payment_intent.payment_failed":
payment_intent = event.data.object
handle_failed_payment.delay(payment_intent.id)
elif event.type == "charge.dispute.created":
dispute = event.data.object
handle_dispute.delay(dispute.id)
return jsonify({"status": "ok"}), 200
```
#### PCI Compliance
- **Target SAQ A** (the least burdensome): Use Stripe Elements/Checkout, never handle card data on your servers.
- **Never SAQ D:** Avoid storing, processing, or transmitting cardholder data on your infrastructure.
- Embed Stripe.js on your checkout page; sensitive data goes directly to Stripe.
- Use tokenization: card details become a `pm_xxx` token on Stripe's servers.
#### Subscription Billing
```python
def create_subscription(customer_id: str, price_id: str, trial_days: int = 0):
"""Create a subscription with optional trial period."""
subscription_params = {
"customer": customer_id,
"items": [{"price": price_id}],
"payment_behavior": "default_incomplete",
"expand": ["latest_invoice.payment_intent"],
}
if trial_days > 0:
subscription_params["trial_period_days"] = trial_days
subscription = stripe.Subscription.create(**subscription_params)
return subscription
```
#### Error Handling & Retry Logic
```python
import time
from typing import Callable
def retry_with_backoff(
max_retries: int = 3,
base_delay: float = 1.0,
max_delay: float = 60.0,
backoff_factor: float = 2.0,
):
"""Exponential backoff retry decorator for payment operations."""
def decorator(func: Callable):
@wraps(func)
def wrapper(*args, **kwargs):
last_exception = None
for attempt in range(max_retries + 1):
try:
return func(*args, **kwargs)
except stripe.error.RateLimitError as e:
last_exception = e
if attempt == max_retries:
raise
delay = min(base_delay * (backoff_factor ** attempt), max_delay)
time.sleep(delay)
except stripe.error.APIConnectionError as e:
last_exception = e
if attempt == max_retries:
raise
delay = min(base_delay * (backoff_factor ** attempt), max_delay)
time.sleep(delay)
except stripe.error.CardError:
raise # Card errors are not retryable
except stripe.error.StripeError as e:
raise # Other Stripe errors are not retryable
raise last_exception
return wrapper
return decorator
```
#### Testing & Sandbox to Production
1. Use Stripe test mode (`sk_test_xxx`) exclusively during development
2. Use test card numbers (`4242 4242 4242 4242` for success, `4000 0000 0000 0002` for decline)
3. Webhook testing: `stripe listen --forward-to localhost:5000/webhooks/stripe`
4. Before production go-live:
- Verify all webhook endpoints respond with 200
- Confirm idempotency on all mutation endpoints
- Test full checkout flow with test cards
- Verify email receipts are configured
- Confirm CVC/AVS checks are enabled
- Set up monitoring alerts for payment failure rate
---
### 2I. Software Copyright & Patent Specialist
#### Intellectual Property Identification
**Patentable inventions** (novel, non-obvious, useful):
- Novel algorithms with measurable performance improvements
- Unique data processing pipelines
- New cryptographic or security methods
- Novel UI interaction patterns
- Hardware-software integration methods
**Copyrightable works:**
- Source code (registered with Copyright Office for statutory damages)
- Documentation, API specifications
- Training data curation and preparation
- Database schema designs (if creative)
**Trade secrets** (confidential business information):
- Proprietary algorithms not disclosed publicly
- Customer data and analytics methods
- Business processes and pricing models
- ML model weights and training methodology
#### Open-Source License Audit
```bash
# Generate SBOM with Syft
syft dir:. -o spdx-json > sbom.spdx.json
syft dir:. -o cyclonedx-json > sbom.cdx.json
# License compliance check with FOSSA / ScanCode
scancode --license --copyright --package --info \
--json-pp scan-results.json .
# Check for copyleft licenses (GPL, AGPL, SSPL) in production dependencies
npm-license-crawler --production --csv licenses.csv
# Review: GPL/AGPL in production code may require releasing source
```
#### SBOM Generation
```bash
# CycloneDX SBOM (OWASP standard)
cyclonedx-npm --output-file sbom.json
# Software Package Data Exchange (SPDX)
spdx-sbom-generator -o .
# Container image SBOM
syft registry.example.com/app:latest -o spdx-json > image-sbom.json
```
#### Patent Application Documentation
**Required sections for each submission:**
```
patent-submissions/
YYYY-MM-algorithm-name/
01-problem-statement.md -- What problem does this solve? (Prior art search results)
02-solution-overview.md -- How does it work? (Include system diagrams)
03-novelty-claims.md -- What is new? (Be specific: "Unlike X which does Y, our method does Z")
04-performance-data.md -- Benchmarks, comparisons, experimental results
05-implementation.md -- Detailed description sufficient for "one skilled in the art"
06-code-inventory.md -- List of relevant source files with paths
07-filing-checklist.md -- USPTO/EPO requirements, fees, deadlines
```
**Novelty documentation template:**
```markdown
## Invention: [Name]
**Date of conception:** YYYY-MM-DD
**Inventors:** [Names]
### Problem
[Clear problem statement. What existing solutions exist and why are they insufficient?]
### Prior Art
| Reference | Key Limitation | Our Improvement |
|-----------|---------------|-----------------|
| US Patent 12345678 | Requires X | Eliminates X by doing Y |
| Paper: "Title" (2024) | O(n^2) complexity | O(n log n) via Z |
### Solution
[How does the invention work? Include technical details.]
### Novelty
[What is new, specifically? Be precise about the novel elements.]
### Performance
| Metric | Prior Art (Best) | Our Method | Improvement |
|--------|-------------------|------------|-------------|
| Speed | 100ms | 45ms | 55% faster |
| Memory | 2GB | 800MB | 60% reduction |
| Accuracy | 92% | 96.5% | +4.5% |
### Implementation
[Code references with file paths and line numbers.]
```
#### Continuous Code Documentation
```bash
# Generate merged source document with line numbers
find src/ -name "*.py" -print0 | sort -z | xargs -0 cat -n > patent-code-appendix.txt
# Split by module
for file in src/**/*.py; do
echo "=== $file ===" >> patent-code-appendix.txt
cat -n "$file" >> patent-code-appendix.txt
echo "" >> patent-code-appendix.txt
done
```
#### IP Submission Checklist
```
[ ] Invention disclosure form completed
[ ] Prior art search conducted (Google Patents, USPTO, EPO, Google Scholar)
[ ] Novelty claims documented with specific comparisons
[ ] Performance benchmarks recorded
[ ] Source code appendix generated
[ ] All inventors identified and listed
[ ] Employer IP assignment agreement confirmed
[ ] Filing deadline noted (1 year from first public disclosure in US)
[ ] Provisional vs. non-provisional decision made
[ ] Legal counsel engaged for final filing
```
---
## 3. Universal Quality Standards
These principles apply across all modes:
### Automate Everything
- No manual steps in production. If a human touches it, it's a bug in the automation.
- Use CI/CD for all deployments, including infrastructure changes.
- Script all operational tasks (restarts, scaling, failovers, backups).
- If you run a command twice, put it in a script. If you run a script twice, put it in CI.
### Infrastructure as Code
- All infrastructure defined in version-controlled code (Terraform, CloudFormation, Pulumi, CDK).
- No click-ops. Console access is read-only for debugging.
- Remote state with locking. No local state files for shared infrastructure.
- Peer review required for all infrastructure changes.
### Immutable Infrastructure
- Never modify running servers. Replace them.
- New AMI/container image for every deployment. Roll forward or roll back by swapping images.
- Configuration changes trigger new deployment, not in-place edits.
- Pets vs. Cattle: treat all servers as disposable.
### Fast Feedback Loops
- Linting, formatting, and unit tests in < 2 minutes (pre-commit/pre-push).
- CI pipeline complete in < 15 minutes for most changes.
- Staging deployment and smoke tests in < 5 minutes.
- Fail early: lint before build, build before test, test before deploy.
### Security by Default
- Least privilege IAM roles and security groups. Start with deny-all, add only what's needed.
- Encryption in transit (TLS 1.2+) and at rest (AES-256) for all data stores.
- Secrets never in plaintext, never in git, never in environment variables committed to source.
- Container images scanned for vulnerabilities (Trivy, Snyk) on every build.
- Regular dependency updates (Renovate, Dependabot) with automated merge for passing CI.
### Observability
- Every service emits: structured logs (JSON with trace ID), metrics (RED/USE), and traces (OpenTelemetry).
- Dashboards for every service (Grafana). Alerting for every SLO.
- Logs are queryable, not just tail-able. Ship to centralized system (Loki, ELK, Datadog).
- Correlation IDs propagate across service boundaries (trace context via W3C Trace Context headers).
### Runbooks & Rollback Plans
- Every deployment includes a runbook: steps, validation, rollback command.
- Rollback should be a single command and take < 5 minutes.
- Practice rollbacks in staging. If you haven't tested rollback, you haven't tested deploy.
- Alert-to-runbook linkage: every alert links to its resolution procedure.
### Cost Awareness
- Right-size resources. Start with minimal requests/limits; scale based on metrics.
- Use spot/preemptible instances for fault-tolerant workloads.
- Tag all resources with `Environment`, `Service`, `Team`, `CostCenter`.
- Auto-shutdown non-production environments outside business hours (CronJob or Instance Scheduler).
- Review cloud bills monthly. Set budget alerts at 50%, 80%, 100% of expected spend.
- Use `infracost` in CI to estimate cost impact of infrastructure changes before apply.
---
## Quick Reference: Common Commands
```bash
# Terraform
terraform init -backend-config=backend.hcl
terraform plan -out=tfplan
terraform apply tfplan
terraform state list
terraform state show <resource>
terraform import <resource_type>.<name> <id>
terraform destroy -target=<resource>
terraform fmt -recursive
terraform validate
# Kubernetes
kubectl get pods -n <namespace> --sort-by=.status.startTime
kubectl describe pod <name> -n <namespace>
kubectl logs -f <pod> -n <namespace> --tail=100
kubectl exec -it <pod> -n <namespace> -- /bin/sh
kubectl rollout status deployment/<name> -n <namespace>
kubectl rollout undo deployment/<name> -n <namespace>
kubectl top pod -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
kubectl port-forward svc/<name> 8080:80 -n <namespace>
kubectl auth can-i create deployments --as=system:serviceaccount:ns:sa
kubectl cluster-info dump # full cluster state for debugging
# Docker
docker build -t app:latest --target production .
docker buildx build --platform linux/amd64,linux/arm64 -t app:latest --push .
docker scan app:latest # Snyk vulnerability scan
docker run --rm -p 8080:8080 --read-only --tmpfs /tmp app:latest
docker compose up -d --build
docker compose logs -f
docker system prune -a --volumes # clean everything unused
# Git
git log --oneline --graph --all
git bisect start HEAD <known-good-commit> # find breaking commit
git revert <commit> # safe undo
git stash push -m "WIP: description"
git stash pop
# Network
dig +trace example.com
nslookup example.com
curl -v -o /dev/null -s -w "%{http_code}\n%{time_total}s\n" https://example.com
tcping <host> <port>
nc -zv <host> <port>
openssl s_client -connect <host>:443 -servername <host>
tcpdump -i eth0 -w capture.pcap port 443
# Performance
k6 run load-test.js --out json=results.json
perf record -F 99 -p <pid> -g -- sleep 30
strace -c -p <pid> # system call summary
lsof -p <pid> # open files and sockets
htop -p <pid> # per-process monitor
# Disk / Memory
df -h
du -sh /* 2>/dev/null | sort -rh | head -20
free -h
vmstat 1
iostat -x 1
```
---
## Operational Principles
1. **The on-call engineer's first priority is to restore service, not find root cause.** Mitigate first, investigate after.
2. **Every alert that fires without a human action is noise.** Tune or remove it.
3. **If it's not monitored, it's not in production.** Every service ships with dashboards and alerts.
4. **Configuration changes are deployments.** Treat them with the same rigor as code changes.
5. **Documentation lives with the code.** Runbooks, architecture decisions, and setup guides in the repo.
6. **Blameless culture.** Incidents are learning opportunities. Focus on process improvements, not individual fault.
7. **"And then what?"** -- for every automation, ask what happens next. What if it fails? What if it succeeds? What cleans up after it?
8. **Boring is a feature.** Production infrastructure should be predictable and uneventful. Novelty belongs in development.
---
## 延伸阅读
本技能专注于 DevOps 流程与基础设施指导。以下 ClawHub 社区技能可提供互补能力，均为可选的第三方工具：
- **qa-security** — 代码漏洞扫描、依赖安全分析、上线前安全审查。与本技能配合可覆盖"部署 + 安全"完整链路。
- **database-specialist** — Schema 设计、SQL 优化、索引策略，适合运维中涉及数据库排查的场景。
以上技能均为独立项目，与本技能无捆绑关系，按需选择安装。
don't have the plugin yet? install it then click "run inline in claude" again.