sf-ai-agentforce-testing: Agentforce Test Execution & Coverage Analysis

intent

use this skill when you need to formally test a published Agentforce agent. this covers multi-turn conversation validation, topic and action routing verification, guardrail checks, escalation flows, and coverage analysis. the goal is to catch routing bugs, action failures, context preservation issues, and guardian behavior before or after go-live. use this skill after the agent is already published and activated in the target org. do not use this if you're building or editing the agent itself (delegate to sf-ai-agentforce or sf-ai-agentscript), running Apex unit tests (use sf-testing), or analyzing session telemetry and STDM traces (use sf-ai-agentforce-observability).

inputs

required org and agent metadata

• agent API name / developer name: internal identifier for the agent you're testing
• target org alias: which Salesforce org contains the agent (e.g., default, staging)
• testing goal: smoke test, regression, coverage expansion, or bug reproduction
• publish / activation state: confirm the agent is already published and activated

optional external connections and auth

• ECA credentials (External Client Application): required only for Track A (multi-turn API testing). set via env var SF_ECA_CLIENT_ID, SF_ECA_CLIENT_SECRET, SF_ECA_INSTANCE_URL, and SF_ECA_SCOPE=agent:invoke. see references/eca-setup-guide.md for setup. do not store secrets in repo files.
• Agent Testing Center availability: check whether the target org supports CLI Testing Center (available in Spring '26+). required for Track B (CLI-driven testing).
• CLI authentication: standard sf org login web for agent discovery and test-spec generation. does not require ECA.

prerequisite dependencies

• agent's backing Flows, Apex actions, and seed data must exist and be functional
• any data required by agent actions must be pre-seeded in the org (delegate to sf-data if needed)
• Flow and Apex fixes must be completed before test execution (delegate to sf-flow or sf-apex)

script location

• pre-approved scripts are under ~/.claude/skills/sf-ai-agentforce-testing/hooks/scripts/
• do not recreate or modify these scripts; use them as-is

procedure

1. discover and verify the agent

• run sf agent list --target-org <alias> to list all agents in the target org
• identify the agent by API name / developer name
• record the agent's current state (published or unpublished, activated or deactivated)
• decision point: if agent is not published or activated, tell the user to publish/activate first, then return to step 1

2. confirm dependencies and test readiness

• navigate to the agent configuration in Salesforce Setup and verify:
  • all topics are defined and assigned
  • all actions (Flows, Apex, or built-in) are created and linked
  • all guardrails and routing rules are in place
• verify that any data required by agent actions exists in the org (e.g., product records, customer data)
• decision point: if critical dependencies are missing, delegate to sf-data, sf-flow, or sf-apex; do not proceed until all are ready

3. choose testing track based on user request and org capabilities

• Track A (multi-turn API testing): choose this if the user needs multi-turn conversation validation, topic re-matching, context preservation checks, or escalation analysis across turns
  • requires ECA credentials to be set up and valid
  • provides fine-grained per-turn failure classification
• Track B (CLI Testing Center): choose this if the user wants org-native sf agent test workflows, test-spec YAML execution, or single-utterance validation, and the org supports Agent Testing Center
  • does not require ECA
  • integrates with CI/CD pipelines via CLI
• manual preview: choose this for quick validation; use sf agent preview commands first, then escalate to Track A or B if deeper testing is needed
• decision point: if user has not specified a preference and ECA is not available, default to Track B if Testing Center is available; if neither is available, use manual preview and escalate to Track A once ECA is ready

4. plan test scenarios

• identify the agent's main topics and create at least one test utterance per topic
• for each topic, include:
  • happy-path phrasing (standard customer language)
  • variation phrases (synonyms, informal speech)
  • edge cases (incomplete info, ambiguous intent)
  • guardrail scenarios (harmful, off-topic, refusal expected)
  • escalation triggers (e.g., "I want to speak to a human")
• include at least one multi-turn conversation per topic to test context preservation and topic re-matching
• record expected outcomes: which topic should match, which action should invoke, whether guardrails should block, expected escalation behavior

5a. execute Track A (multi-turn API testing)

• validate ECA credentials:
  • run the credential validation script at ~/.claude/skills/sf-ai-agentforce-testing/hooks/scripts/validate-eca.sh or equivalent
  • decision point: if validation fails, debug auth with the credential tooling (do not use raw curl); see references/eca-setup-guide.md
• retrieve agent metadata and topic list:
  • run sf agent get --name <agent-api-name> --target-org <alias> --json to fetch agent config
  • run sf topic list --target-org <alias> --json to list all topics
• generate multi-turn test scenarios using the provided Python script at ~/.claude/skills/sf-ai-agentforce-testing/hooks/scripts/generate-scenarios.py
  • input: agent metadata, topic list, and manually-authored test plan
  • output: JSON file with structured multi-turn conversation flows
• run multi-turn tests using the Agent Runtime API execution script at ~/.claude/skills/sf-ai-agentforce-testing/hooks/scripts/run-multi-turn.py
  • input: scenario JSON, ECA credentials, agent runtime endpoint
  • output: test results JSON with per-turn success/failure, action invocation logs, and context traces
• analyze results:
  • for each turn, record whether the topic matched correctly, the action invoked correctly, and context was preserved
  • classify failures by type: topic mismatch, action failure, guardrail block, context loss, escalation failure
  • compute coverage percentage: (successful turns / total turns) * 100
• clean up test sessions: run the cleanup script to remove test sessions from the Agent Runtime audit log

5b. execute Track B (CLI Testing Center)

• generate or refine a test-spec YAML file:
  • use the template at references/test-spec-reference.md
  • include name, description, utterance, expected topic, expected action, and expected guardrail behavior for each test case
• run tests with the CLI:
  • execute sf agent test --file <spec.yaml> --target-org <alias> --json to run the test spec
  • output: structured test results showing pass/fail per test case
• inspect detailed logs:
  • run sf agent test --file <spec.yaml> --target-org <alias> --verbose to see action invocation traces, topic routing decisions, and guardrail evaluations
• analyze results:
  • tally pass/fail counts
  • identify topics or actions that consistently fail
  • note guardrail false positives or false negatives
  • compute coverage as a percentage of test cases passed

6. classify failures and root causes

• topic not matched: utterance did not trigger any topic. likely causes: topic keywords too specific, topic description too narrow, or utterance phrasing too different from training examples.
• wrong topic matched: utterance triggered an incorrect topic. likely causes: topic descriptions too similar, guardrails too permissive, or intent ambiguity.
• action not invoked: topic matched but action did not execute. likely causes: action configuration missing, action name mismatch, or action preconditions not met.
• wrong action selected: topic matched but wrong action was chosen. likely causes: action ordering or ranking incorrect in topic config, or action selection rules ambiguous.
• action invocation failed: action executed but returned error or no result. likely causes: required input fields missing, data preconditions not met, Flow or Apex bug.
• context preservation failure: agent did not remember prior turns in conversation. likely causes: context storage misconfigured, or session timeout.
• guardrail failure: guardrail did not block harmful or off-topic utterance, or incorrectly blocked benign utterance. likely causes: guardrail rule too permissive, false positives in guardrail model.
• escalation failure: escalation action did not trigger when expected. likely causes: escalation precondition not met, or escalation action misconfigured.

7. run fix loop

• for each failure category, determine if the issue is in the agent configuration (topic/action routing) or in the underlying system (Flow, Apex, data):
  • agent configuration issues: fixes are delegated to sf-ai-agentscript; do not fix here
  • Flow or Apex issues: delegate to sf-flow or sf-apex
  • missing or bad data: delegate to sf-data
  • guardian rule or escalation issues: delegate to sf-ai-agentscript
• after fixes are deployed, re-publish and re-activate the agent if needed
• re-run focused tests on the failure categories to confirm fixes
• run full regression if the fix was broad (e.g., changed topic routing logic)

8. document and report results

• record final test metrics: total tests, passed, failed, coverage percentage, and score (see output contract)
• list all failures by root cause and criticality
• identify coverage gaps (topics, actions, guardrails not yet tested)
• recommend next steps: additional test scenarios, fix deployments, or go-live decision

decision points

track selection

• if user requests multi-turn conversation testing or context preservation checks: use Track A (multi-turn API)
  • if ECA credentials are not set up: guide user through references/eca-setup-guide.md before proceeding
  • if ECA credentials are invalid or expired: use the credential tooling to refresh; do not use raw curl
• if user requests org-native CLI testing or CI/CD integration: use Track B (CLI Testing Center)
  • if org does not support Agent Testing Center: fall back to Track A or manual preview
• if neither Track A nor Track B is available: use manual sf agent preview for quick validation and escalate to full testing once prerequisites are met

agent state

• if agent is not published: tell user to publish the agent first; do not run tests on unpublished agents
• if agent is not activated: tell user to activate the agent first; do not run tests on inactive agents

dependencies

• if required actions (Flows, Apex) are missing or broken: delegate to sf-flow or sf-apex; do not run tests
• if required seed data is missing: delegate to sf-data; do not run tests
• if the agent itself needs authoring or script changes: delegate to sf-ai-agentscript; do not edit the agent here

failure classification

• if failures are in topic/action routing or guardrails: escalate to sf-ai-agentscript for agent authoring fixes
• if failures are in action execution (Flows or Apex): escalate to sf-flow or sf-apex
• if failures are related to session telemetry or trace analysis: escalate to sf-ai-agentforce-observability

test coverage and scoring

• if score is 90+: agent is production-ready; approve go-live
• if score is 80-89: strong coverage with minor gaps; acceptable for go-live with caveats
• if score is 70-79: coverage expansion recommended before go-live
• if score is 60-69: partial validation only; recommend additional testing before go-live
• if score is < 60: insufficient test confidence; block go-live and expand test coverage

output contract

test execution report

report results in this format after all tracks are complete:

Agent: <agent-api-name>
Track: Multi-turn API | CLI Testing Center | Manual Preview
Executed: <N> scenarios or <N> test cases
Result: Passed | Partial | Failed
Breakdown: <M> passed, <N> failed
Coverage: <X>% of topics, <Y>% of actions, <Z>% of guardrails
Issues (by frequency):
  - <root-cause-1>: <count> failures
  - <root-cause-2>: <count> failures
Score: <0-100 points>
Next step: <fix and republish | rerun tests | expand coverage | ready for go-live>

test result artifacts

• Track A (multi-turn API):

  • JSON file with test execution log: test-results-<timestamp>.json
    • format: array of turn objects, each with utterance, expected/actual topic, expected/actual action, guardrail decision, context state, and pass/fail status
  • coverage analysis document: coverage-analysis-<timestamp>.md
    • format: table of topics/actions/guardrails with test count and pass rate

• Track B (CLI Testing Center):

  • CLI output file (JSON): test-output-<timestamp>.json
    • format: test results as emitted by sf agent test --json
  • verbose log file: test-verbose-<timestamp>.log
    • format: human-readable action traces and routing decisions

scoring system

• 100 points distributed across 7 categories:

  • topic coverage: 15 points (at least one test per main topic)
  • topic accuracy: 15 points (correct topic matched on happy-path tests)
  • action invocation: 15 points (action executed without error on valid inputs)
  • guardrail enforcement: 15 points (harmful/off-topic utterances correctly blocked; no false positives on benign utterances)
  • context preservation: 15 points (multi-turn conversations maintain context and topic re-match correctly)
  • escalation correctness: 10 points (escalation triggers fire as configured)
  • phrasing variation: 10 points (agent handles synonyms and informal speech for each topic)

• score calculation: (points earned / 100) * 100 = overall score

• scoring rules:

  • each category is scored 0-100 based on pass rate within that category
  • final score is the weighted average across all categories
  • if a category is not tested (e.g., no guardrails defined), it does not count toward the final score

file locations

• test result artifacts are stored in: ~/.claude/skills/sf-ai-agentforce-testing/results/<agent-api-name>/
• timestamped folders ensure no overwrites: results/<agent-api-name>/<YYYYMMDD-HHMMSS>/
• all logs and reports are retained for audit; clean up manually if needed

outcome signal

how you know the skill worked

• test execution completed: test command(s) ran without crashes or auth errors; results JSON or CLI output was generated
• results are meaningful: test output includes per-turn or per-case success/failure and reason (not just "error")
• coverage is clear: coverage percentage and gap analysis are documented; you know which topics, actions, and guardrails were tested
• root causes are identified: failures are bucketed by type (topic mismatch, action failure, guardrail, context loss, escalation); you can trace a failure to a specific utterance and agent behavior
• score is computed: a single 0-100 number is assigned with transparent calculation (e.g., "15/15 topic coverage, 12/15 topic accuracy..." = 92 points)
• next steps are clear: report recommends either "fix and republish", "rerun tests after fix", "expand coverage", or "ready for go-live" with no ambiguity
• no false confidence: if score is < 80, you acknowledge gaps and do not claim the agent is ready for production without additional testing

---

core operating rules

• testing comes after deploy / publish / activate. do not test unpublished agents.
• use multi-turn API testing as the primary path when conversation continuity and context preservation matter.
• use CLI Testing Center as the secondary path for single-utterance validation and org-supported test-center workflows.
• interactive and programmatic CLI preview use standard sf org login web authentication. ECA is only required for Agent Runtime API testing, not for live preview.
• fixes to the agent should be delegated to sf-ai-agentscript when Agent Script changes are needed.
• do not use raw curl for OAuth token validation in the ECA flow; use the provided credential tooling.

script path rule

use the existing scripts under ~/.claude/skills/sf-ai-agentforce-testing/hooks/scripts/. these scripts are pre-approved. do not recreate them.

---

recommended workflow

1. discover and verify

• locate the agent in the target org
• confirm it is published and activated
• confirm required actions, Flows, Apex, and data exist
• decide whether Track A or Track B fits the request

2. plan tests

cover at least:

• main topics (at least one test per topic)
• expected actions (verify each action fires correctly)
• guardrails and off-topic handling
• escalation behavior
• phrasing variation (synonyms, informal speech)

3. execute the right track

Track A (multi-turn API testing)

• validate ECA credentials with the provided tooling
• retrieve metadata needed for scenario generation
• run multi-turn scenarios with the provided Python scripts
• analyze per-turn failures and coverage

Track B (CLI Testing Center)

• generate or refine a flat YAML test spec
• run sf agent test commands
• inspect structured results and verbose action output

4. classify failures

typical failure buckets:

• topic not matched
• wrong topic matched
• action not invoked
• wrong action selected
• action invocation failed
• context preservation failure
• guardrail failure
• escalation failure

5. run fix loop

when failures imply agent-authoring issues:

• delegate fixes to sf-ai-agentscript
• re-publish / re-activate if needed
• re-run focused tests before full regression

---

testing guardrails

never skip these:

• test only after publish/activate
• include harmful, off-topic, and refusal scenarios
• use multiple phrasings per important topic
• clean up test sessions after API tests
• keep swarm execution small and controlled

avoid these anti-patterns:

• testing unpublished agents
• treating one happy-path utterance as full coverage
• storing ECA secrets in repo files
• debugging auth with brittle shell-expanded curl commands
• changing both tests and agent simultaneously without isolating the cause

---

cross-skill integration

need	delegate to	reason
fix Agent Script logic	sf-ai-agentscript	authoring and deterministic fix loops
create test data	sf-data	action-ready data setup
fix Flow-backed actions	sf-flow	Flow repair
fix Apex-backed actions	sf-apex	Apex repair
set up ECA / OAuth for Agent Runtime API	sf-connected-apps	auth and app configuration
analyze session telemetry	sf-ai-agentforce-observability	STDM / trace analysis

---

reference map

start here

• references/interview-wizard.md
• references/multi-turn-testing.md
• references/cli-commands.md
• references/test-spec-reference.md

execution / auth

• references/execution-protocol.md
• references/multi-turn-execution.md
• references/eca-setup-guide.md
• references/credential-convention.md
• references/connected-app-setup.md

coverage / fix loops

• references/coverage-analysis.md
• references/agentic-fix-loops.md
• references/results-scoring.md
• references/known-issues.md

advanced / specialized

• references/agentscript-agents.md
• references/agentscript-testing-patterns.md
• references/cli-testing-details.md
• references/deep-conversation-history-patterns.md
• references/swarm-execution.md
• references/trace-analysis.md
• references/agent-api-reference.md

templates / assets

• references/test-templates.md
• references/test-plan-format.md
• assets/

---

score guide

score	meaning
90+	production-ready test confidence
80-89	strong coverage with minor gaps
70-79	acceptable but coverage expansion recommended
60-69	partial validation only
< 60	insufficient confidence; block release

---

credits: original author Jag Valaiyapathy. skill maintainer: clawhub.