Generic Quality Gateways for Unattended Agent Development

SKILL.md

# openClaw Skill: Quality Gateways (Generic Web + API Applications)

## Purpose
This skill defines and applies **6 universal quality gateways** for typical application projects that include:
- Backend API services (any stack)
- Web frontends (any stack)
- CI/CD pipelines (any provider)

The gateways are written in **LLM-friendly operational language**: how to **check**, **calculate**, **evaluate**, and **document** results consistently.

This skill is **language-agnostic** and can be used on any repository. It relies on a central configuration file:
- `.defs/quality-gateway-definition.json` (MUST be stored in the repository, not the workspace)

## Non-Negotiable Storage Rules (openClaw)
- The gateway definition file MUST be placed in: `REPO_ROOT/.defs/quality-gateway-definition.json`
- Temporary files MUST go to: `REPO_ROOT/.tmp/quality-gates/` (do not create or delete other workspace directories)
- Reports MUST be written to repository paths defined in the JSON config (default suggested below)

## Inputs
- Repository root path (REPO_ROOT)
- Optional CI artifacts path (if provided by the runtime)
- Optional commit range (for PR-focused evaluation)
- Optional environment notes (target load, environments, risk level)

## Outputs
1. A human-readable report (Markdown)
2. A machine-readable report (JSON) containing raw metrics + per-check scores
3. Evidence references (paths, snippets, CI links if available)

Recommended default output paths (override via JSON config):
- `docs/quality/quality-gate-report.md`
- `docs/quality/quality-gate-report.json`
- Evidence directory: `docs/quality/evidence/`

---

# The 6 Quality Gateways

Each gateway produces:
- **Score**: 0–100
- **Status**: PASS / WARN / FAIL
- **Blocking behavior**: some gateways are “blocking” (FAIL blocks release)

All gateway thresholds and weights come from:
- `.defs/quality-gateway-definition.json`

---

## Gateway 1 — Build & Dependency Health
### Goal
Ensure the system can be built and packaged reliably, and dependencies are manageable and safe to ship.

### What to Check (typical checks)
- CI pipeline status (green on default branch / PR)
- Reproducible build or deterministic packaging indicators
- Dependency freshness (stale/outdated dependencies)
- License policy compliance (allowlist/denylist)
- SBOM presence (if required)

### How to Measure / Calculate
- Boolean checks: PASS=100, FAIL=0
- Ratio checks (e.g., “outdated deps %”): scale 0–100 using thresholds
- Policy checks: hard FAIL if a forbidden license is detected (if enabled)

### Evidence to Collect
- CI job summary (or local build logs)
- Dependency list report output (tool-specific, but keep the report file)
- SBOM artifact path (if present)
- License scan output (if used)

### How to Document
In the report, include:
- Build command/pipeline name
- Artifact identifiers / versions
- Summary of dependency deltas and policy results

---

## Gateway 2 — Automated Testing & Coverage
### Goal
Prove correctness through automated tests and prevent regression.

### What to Check
- Unit tests pass
- Integration/API tests pass (or contract tests)
- E2E/smoke tests pass (for web apps)
- Code coverage meets thresholds (overall + critical components)
- Flaky test rate is controlled (if CI provides retries/flakes)

### How to Measure / Calculate
- Test pass: boolean
- Coverage: numeric percentage
  - Score mapping example:
    - >= target: 100
    - between warn and target: linear 70–99
    - below warn: linear 0–69
- Optional “critical path coverage” gets extra weight

### Evidence to Collect
- Test run outputs (JUnit/TRX/etc.)
- Coverage summary files
- List of failed tests (if any) + links

### How to Document
- Test suites executed
- Coverage numbers (overall + key areas)
- Notes on skipped tests (if allowed) and rationale

---

## Gateway 3 — Security & Supply-Chain
### Goal
Prevent known vulnerabilities, secrets leakage, insecure configs, and supply-chain risks.

### What to Check
- Dependency vulnerabilities (Critical/High/Medium counts)
- Secret scanning results (must be zero leaked secrets)
- Basic secure configuration checks (CSP, TLS, auth boundaries) where applicable
- SAST findings severity counts (if tooling exists)
- Container image scan (if containers exist)

### How to Measure / Calculate
- Vulnerability gating (typical):
  - Critical = 0 required (FAIL otherwise)
  - High = 0 required (or <= allowedHigh)
  - Medium allowed up to a budget (WARN if above warn)
- Secrets: any secret finding => FAIL (blocking)
- Score: start at 100 and subtract penalties by severity and count (config-driven)

### Evidence to Collect
- Vulnerability scan report files
- Secret scan output (including file paths and fingerprint IDs, not actual secrets)
- SAST report snippet/summary

### How to Document
- Severity counts and whether exceptions exist
- Any exception MUST include: reason, owner, expiry date (if your org uses waivers)

---

## Gateway 4 — Performance & Efficiency (API + Web)
### Goal
Ensure the system meets baseline performance and user experience targets.

### What to Check
API (typical):
- p95 latency under target
- Error rate under target
- Throughput meets expected load (if known)

Web (typical):
- Core Web Vitals (LCP, CLS, INP) on a reference device/profile
- Bundle size / asset weight thresholds (optional)

### How to Measure / Calculate
- Latency score:
  - p95 <= target: 100
  - between target and warn: linear 70–99
  - > warn: 0–69 (linear), with hard FAIL if beyond “max”
- Error rate:
  - <= target: 100
  - <= warn: 70–99
  - > warn: 0–69, FAIL if beyond max
- Web vitals:
  - Each metric scored independently; weighted into a single web score

### Evidence to Collect
- Load test or benchmark outputs (k6/JMeter/etc.)
- APM snapshots (if available)
- Lighthouse or Web Vitals report exports

### How to Document
- Test conditions: environment, dataset size, concurrency, device profile
- Key p95 / error rate / vitals values
- Notable regressions vs baseline

---

## Gateway 5 — Maintainability & Code Health
### Goal
Keep the codebase understandable, changeable, and reviewable over time.

### What to Check
- Static analysis quality (lint errors, rule violations)
- Complexity thresholds (cyclomatic complexity, large functions/classes)
- Duplication rate
- “Change risk” signals (hotspots: frequent churn + complexity)
- Documentation coverage for public APIs (e.g., endpoint docs, component docs)

### How to Measure / Calculate
- Issue density: findings per KLOC (or per file for smaller repos)
- Complexity score: percentage of units exceeding complexity threshold
- Duplication: % duplicated lines
- Score: weighted average of normalized sub-scores (config-driven)

### Evidence to Collect
- Static analysis summaries
- Complexity and duplication reports (any tool is fine; store outputs)
- List of top hotspots and why (files + metrics)

### How to Document
- Top 10 problems by impact
- Concrete refactoring suggestions only if asked; otherwise just findings

---

## Gateway 6 — Release Readiness & Operability (Observability + Runbooks)
### Goal
Make sure the system can be operated safely in production.

### What to Check
- Health endpoints exist and are meaningful
- Logging is structured and includes correlation IDs
- Metrics and dashboards exist for key signals (latency, error rate, saturation)
- Alerts configured for SLO breaches / error budget burn (if applicable)
- Runbooks for major failure modes exist (deploy rollback, incident triage)
- Versioning and changelog/release notes exist

### How to Measure / Calculate
Mostly “presence + completeness” scoring:
- Each required artifact is a boolean check
- Optional maturity rubric: 0 (missing), 50 (partial), 100 (complete)
- Blocking if “minimum operability” is not met (config-driven)

### Evidence to Collect
- Paths to runbooks, dashboards-as-code, alert configs
- Sample log/metric/tracing docs
- On-call/ops notes (if present)

### How to Document
- List missing operational artifacts
- Minimum go-live checklist status

---

# Standard Evaluation Algorithm (LLM-Executable)

## Step 1: Load configuration
- Read `REPO_ROOT/.defs/quality-gateway-definition.json`
- Validate it against the schema description (see below)
- If fields are missing, use documented defaults from the JSON

## Step 2: Collect metrics per check
For each gate:
- For each check:
  - Identify data source:
    - Prefer CI artifacts if provided
    - Otherwise use repository files and local commands (if allowed by runtime)
  - Produce a metric value (number/boolean/string) and evidence references

## Step 3: Score each check (0–100)
Use the scoring method defined per check:
- `boolean`: pass => 100, fail => 0
- `threshold_range`: linear scoring between warn and target
- `penalty_by_count`: start at 100 and subtract per issue
- `rubric`: map {missing/partial/complete} to {0/50/100}

## Step 4: Score each gateway
- Compute weighted average of its checks
- Determine gateway status using configured thresholds:
  - Score >= passScore => PASS
  - Score >= warnScore => WARN
  - else => FAIL
- If gateway is marked `blockingOnFail=true`, any FAIL blocks release

## Step 5: Produce reports
Write:
1) Markdown report (human)
2) JSON report (machine)
Include:
- per-gateway score/status
- per-check metrics + evidence paths
- overall score and overall status
- explicit “BLOCKERS” list if any

---

# Report Template (Markdown)
Use this outline in `docs/quality/quality-gate-report.md` unless JSON overrides paths:

## Summary
- Overall Score:
- Overall Status:
- Blocking Failures:
- Date/Commit:

## Gateway Results
| Gateway | Score | Status | Key Metrics | Evidence |
|---|---:|---|---|---|

## Details (per Gateway)
### <Gateway Name>
- Score/Status
- Checks:
  - <Check>: metric=..., score=..., evidence=...
- Notes / Exceptions

---

# quality-gateway-definition.json — JSON Schema Description

The configuration file is a normal JSON document with:

## Root
- `schemaVersion` (string) — version of this config layout
- `projectProfile` (object) — context used for defaults
- `scoring` (object) — global pass/warn thresholds and aggregation rules
- `reporting` (object) — output paths and evidence folder
- `gates` (array) — list of gateway definitions (exactly 6 for this skill)

## projectProfile (object)
- `applicationType` (string) — e.g. `"web_api_and_web"`
- `riskLevel` (string) — `"low"|"medium"|"high"`
- `releaseCadence` (string) — e.g. `"daily"|"weekly"|"monthly"`
- `expectedLoad` (object, optional)
  - `apiRps` (number)
  - `concurrency` (number)

## scoring (object)
- `passScore` (number 0–100)
- `warnScore` (number 0–100)
- `overallAggregation` (string) — `"weighted_average"`
- `blockIfAnyBlockingGateFails` (boolean)

## reporting (object)
- `markdownReportPath` (string, repo-relative)
- `jsonReportPath` (string, repo-relative)
- `evidenceDir` (string, repo-relative)
- `tempDir` (string, repo-relative; MUST be inside `.tmp/quality-gates/`)

## gates (array of objects)
Each gate:
- `id` (string) — stable identifier
- `name` (string)
- `description` (string)
- `weight` (number) — relative importance in overall score
- `blockingOnFail` (boolean)
- `checks` (array)

## checks (array of objects)
Each check:
- `id` (string)
- `name` (string)
- `description` (string)
- `weight` (number)
- `metricType` (string) — `"boolean"|"percentage"|"count"|"duration_ms"|"rubric"`
- `scoringMethod` (string) — `"boolean"|"threshold_range"|"penalty_by_count"|"rubric"`
- `thresholds` (object) — depends on scoringMethod:
  - for `threshold_range`:
    - `target` (number)
    - `warn` (number)
    - `max` (number, optional hard-fail)
    - `direction` (string) — `"higher_is_better"|"lower_is_better"`
  - for `penalty_by_count`:
    - `allowed` (number)
    - `warnAbove` (number)
    - `failAbove` (number)
    - `penaltyPerUnit` (number)
- `evidenceHints` (array of strings) — where to find evidence in a generic repo/CI
- `notes` (string, optional)

---

# Operational Notes
- If a metric cannot be measured, do NOT invent numbers.
  - Mark the check as `"unknown"` in the JSON report and score it using the config’s fallback rule (recommended: treat unknown as WARN with score 70 unless the check is security/secrets, where unknown should be FAIL).
- Always include evidence references (paths or CI artifact names).
- Keep all temp work inside `.tmp/quality-gates/`.

# JSON references
- `templ/quality-gateway-definition-template.json` (template settings file. Can be copied to `REPO_ROOT/.defs/quality-gateway-definition.json` if missing)