smoke-test

SKILL.md

Smoke Test Skill

Creates a new Mastra project using create-mastra@<tag> and performs smoke testing of the Mastra Studio in Chrome.

This skill is for Claude Code with Chrome MCP server. For MastraCode with built-in browser tools, use mastracode-smoke-test instead.

Usage

/smoke-test --directory <path> --name <project-name> --tag <version> [--pm <package-manager>] [--llm <provider>]
/smoke-test -d <path> -n <project-name> -t <version> [-p <package-manager>] [-l <provider>]

Parameters

Parameter
Short
Description
Required
Default

--directory
-d
Parent directory where project will be created
Yes
-

--name
-n
Project name (will be created as subdirectory)
Yes
-

--tag
-t
Version tag for create-mastra (e.g., latest, alpha, 0.10.6)
Yes
-

--pm
-p
Package manager: npm, yarn, pnpm, or bun
No
npm

--llm
-l
LLM provider: openai, anthropic, groq, google, cerebras, mistral
No
openai

Examples

# Minimal (required params only)
/smoke-test -d ~/projects -n my-test-app -t latest

# Full specification
/smoke-test --directory ~/projects --name my-test-app --tag alpha --pm pnpm --llm anthropic

# Using short flags
/smoke-test -d ./projects -n smoke-test-app -t 0.10.6 -p bun -l openai

Step 0: Parameter Validation (MUST RUN FIRST)

CRITICAL: Before proceeding, parse the ARGUMENTS and validate:

Parse arguments from the ARGUMENTS string provided above

Check required parameters:

--directory or -d: REQUIRED - fail if missing

--name or -n: REQUIRED - fail if missing

--tag or -t: REQUIRED - fail if missing

Apply defaults for optional parameters:

--pm or -p: Default to npm if not provided

--llm or -l: Default to openai if not provided

Validate values:

pm must be one of: npm, yarn, pnpm, bun

llm must be one of: openai, anthropic, groq, google, cerebras, mistral

directory must exist (or will be created)

name should be a valid directory name (no spaces, special chars)

If validation fails: Stop and show usage help with the missing/invalid parameters.

If -h or --help is passed: Show this usage information and stop.

Prerequisites

This skill requires the Chrome MCP server (Claude-in-Chrome) for browser automation. Ensure it's configured and running.

The Chrome MCP server provides tools like tabs_create_mcp, tabs_context_mcp, navigate_mcp, click_mcp, type_mcp, and screenshot_mcp.

Execution Steps

Step 1: Create the Mastra Project

Run the create-mastra command with explicit parameters to avoid interactive prompts:

# For npm
npx create-mastra@<tag> <project-name> -c agents,tools,workflows,scorers -l <llmProvider> -e

# For yarn
yarn create mastra@<tag> <project-name> -c agents,tools,workflows,scorers -l <llmProvider> -e

# For pnpm
pnpm create mastra@<tag> <project-name> -c agents,tools,workflows,scorers -l <llmProvider> -e

# For bun
bunx create-mastra@<tag> <project-name> -c agents,tools,workflows,scorers -l <llmProvider> -e

Flags explained:

-c agents,tools,workflows,scorers - Include all components

-l <provider> - Set the LLM provider

-e - Include example code

Being explicit with all parameters ensures the CLI runs non-interactively.

Wait for the installation to complete. This may take 1-2 minutes depending on network speed.

Step 2: Verify Project Structure

After creation, verify the project has:

package.json with mastra dependencies

src/mastra/index.ts exporting a Mastra instance

.env file (may need to be created)

Step 2.5: Add Browser Agent for Browser Testing

To test browser functionality, add a browser-enabled agent:

Install browser packages:

<pm> add @mastra/stagehand
# or for deterministic browser automation:
<pm> add @mastra/agent-browser

Create browser-agent.ts in src/mastra/agents/:

import { Agent } from '@mastra/core/agent';
import { Memory } from '@mastra/memory';
import { StagehandBrowser } from '@mastra/stagehand';

export const browserAgent = new Agent({
  id: 'browser-agent',
  name: 'Browser Agent',
  instructions: `You are a helpful assistant that can browse the web to find information.`,
  model: '<provider>/<model>', // e.g., 'openai/gpt-4o'
  memory: new Memory(),
  browser: new StagehandBrowser({
    headless: false,
  }),
});

Update index.ts to register the browser agent:

import { browserAgent } from './agents/browser-agent';

// In Mastra config:
agents: { weatherAgent, browserAgent },

Step 3: Configure Environment Variables

Based on the selected LLM provider, check for the required API key:

Provider
Required Environment Variable

openai
OPENAI_API_KEY

anthropic
ANTHROPIC_API_KEY

groq
GROQ_API_KEY

google
GOOGLE_GENERATIVE_AI_API_KEY

cerebras
CEREBRAS_API_KEY

mistral
MISTRAL_API_KEY

Check in this order:

Check global environment first: Run echo $<ENV_VAR_NAME> to see if the key is already set globally

If set globally, the project will inherit it - no .env file needed

Skip to Step 4

Check project .env file: If not set globally, check if .env exists in the project and contains the key

Ask user only if needed: If the key is not available globally or in .env:

Ask the user for the API key

Create the .env file with the provided key

Only check for the ONE key matching the selected provider - don't check for all providers.

Step 4: Start the Development Server

Navigate to the project directory and start the dev server:

cd <directory>/<project-name>
<packageManager> run dev

The server typically starts on http://localhost:4111. Wait for the server to be ready before proceeding.

Step 5: Smoke Test the Studio

Use the Chrome browser automation tools to test the Mastra Studio.

5.1 Initial Setup

Get browser context using tabs_context_mcp

Create a new tab using tabs_create_mcp

Navigate to http://localhost:4111

5.2 Test Checklist

Perform the following smoke tests using the Chrome automation tools:

Navigation & Basic Loading

 Studio loads successfully (page contains "Mastra Studio" or shows agents list)

 Take a screenshot of the home page

Agents Page (/agents)

 Navigate to agents page

 Verify at least one agent is listed (the example agent from --default)

 Take a screenshot

Agent Detail (/agents/<agentId>/chat)

 Click on an agent to view details

 Verify the agent overview panel loads

 Verify model settings panel is visible

 Take a screenshot

Agent Chat

 Send a test message to the agent (e.g., "What's the weather in Tokyo?")

 Wait for response

 Verify response appears in the chat

 Take a screenshot of the conversation

Browser Agent (/agents/browser-agent/chat) - if browser agent was added

 Navigate to the browser-agent

 Send a message: "Go to example.com and tell me what you see"

 Verify the agent launches a browser and extracts content

 Verify response includes page content

 Take a screenshot

Tools Page (/tools)

 Navigate to tools page

 Verify tools list loads (should show get-weather tool)

 Take a screenshot

Tool Execution (/tools/get-weather)

 Click on the get-weather tool to open detail page

 Find the city input field and enter a test city (e.g., "Tokyo")

 Click Submit button

 Wait for execution to complete

 Verify JSON output appears with weather data (temp, condition, etc.)

 Take a screenshot

Workflows Page (/workflows)

 Navigate to workflows page

 Verify workflows list loads (should show weather-workflow)

 Take a screenshot

Workflow Execution (/workflows/weather-workflow)

 Click on the weather-workflow to open detail page

 Verify visual graph displays (shows workflow steps)

 Find the city input field and enter a test city (e.g., "London")

 Click Run button

 Wait for execution to complete

 Verify steps show success (green checkmarks)

 Click to view JSON output modal

 Verify execution details with timing appear

 Take a screenshot

Settings Page (/settings)

 Navigate to settings page

 Verify settings page loads

 Take a screenshot

Observability Page (/observability)

 Navigate to observability page

 Verify traces list shows recent activity (from previous tests)

 Click on a trace to view details

 Verify timeline view shows steps and timing

 Take a screenshot

Scorers Page (/evaluation?tab=scorers)

 Navigate to /evaluation?tab=scorers (NOT /scorers - that route doesn't exist)

 Verify scorers list loads (shows 3 example scorers)

 Take a screenshot

Additional Pages (verify load only)

 Templates page (/templates) - Gallery of starter templates

 Request Context page (/request-context) - JSON editor

 Processors page (/processors) - Empty state OK

 MCP Servers page (/mcps) - Empty state OK

5.3 Report Results

After completing all tests, provide a summary:

Total tests passed/failed

Any errors encountered

Screenshots captured

Recommendations for issues found

Quick Reference

Step
Action

Create Project
cd <directory> && npx create-mastra@<tag> <name> -c agents,tools,workflows,scorers -l <provider> -e

Install Deps
Automatic during creation

Set Env Vars
Check global env first, then .env, ask user only if needed

Start Server
cd <directory>/<name> && npm run dev

Studio URL
http://localhost:4111

Troubleshooting

Server won't start

Verify .env has required API key

Check if port 4111 is available

Try <pm> install to reinstall dependencies

Browser can't connect

Wait a few seconds for server to fully start

Check terminal for server ready message

Verify no firewall blocking localhost

Agent chat fails

Verify API key is valid

Check server logs for errors

Ensure LLM provider API is accessible

Browser agent fails

Ensure Playwright browsers are installed: pnpm exec playwright install chromium

Check that no other browser instance is blocking

Studio Routes

Feature
Route

Agents
/agents

Workflows
/workflows

Tools
/tools

Evaluation
/evaluation

Scorers
/evaluation?tab=scorers

Observability
/observability/traces

Logs
/observability/logs

MCP Servers
/mcps

Processors
/processors

Templates
/templates

Request Context
/request-context

Settings
/settings

Notes

The -e flag includes example agents, making smoke testing meaningful

If the user doesn't specify an LLM provider, default to OpenAI as it's most common

Take screenshots at each major step for documentation/debugging

Keep the dev server running in the background during testing

Always use explicit flags (-c, -l, -e) to ensure non-interactive execution

Browser agent testing validates the new browser automation features

Observability traces appear automatically after running agents or workflows