Add Directories

SKILL.md

---
name: add-directories
description: Use when adding new AI/startup directories to directories.json from a URL, pasted text, or GitHub awesome-list. Parses, deduplicates, appends new entries, then runs classify pipeline (analyze, cleanup, browser verify). Does NOT cover submission — use submit-directories skill for that.
---

# Add Directories

## Workflow

1. **Parse** the input (URL or pasted text) into a list of directories
2. **Deduplicate** against existing entries in `directories.json`
3. **Append** new entries with required fields
4. **Classify** by running the analysis and verification pipeline
5. **Discover forms** for submission targets
6. **Submit** via automation or manual browser interaction

## Step 1: Parse Input

### From URL

Fetch the page and extract directory entries. Look for patterns like:
- Name + URL pairs in lists, tables, or cards
- Structured data (JSON-LD, markdown tables, CSV)
- Repeated DOM patterns with links

### From GitHub Topics/Repos

Use `gh` CLI to explore curated lists:
- `gh repo clone <owner>/<repo>` to clone awesome-lists
- Parse README.md for directory links (markdown link format)
- Check for JSON/YAML data files with directory entries
- Can also create PRs to add your product to these lists

### From Pasted Text

Parse lines/rows. Common formats:
- `Name - https://url.com` or `Name | https://url.com`
- Markdown links: `[Name](https://url.com)`
- Markdown tables with Name and URL columns
- Plain URLs (one per line) — derive name from domain
- CSV/TSV with headers

Extract at minimum: **name** and **url** (submission or homepage).

## Step 2: Deduplicate

Load `directories.json` and check each parsed entry against existing ones by:
- Exact URL match (normalize: strip trailing slash, lowercase domain)
- Domain match (same domain = likely duplicate)
- Name match (case-insensitive)

Report duplicates to the user and skip them.

## Step 3: Append New Entries

For each new directory, create an entry with this structure:

```json
{
  "categories": ["General"],
  "description": "",
  "is_active": true,
  "name": "Directory Name",
  "pricing_type": "free",
  "slug": "directory-name",
  "submission_url": "https://example.com/submit",
  "url": "https://example.com/submit"
}
```

Field rules:
- `slug`: lowercase name, spaces to hyphens, strip special chars
- `submission_url` and `url`: use the submission/signup URL if available, otherwise homepage
- `description`: leave empty string (will be filled later or by user)
- `categories`: default `["General"]` unless context provides a category
- `pricing_type`: default `"free"` unless explicitly marked paid
- `is_active`: always `true` for new entries

Save the updated `directories.json`.

## Step 4: Classify

Run the pipeline scripts in order using the project venv at `.venv/`:

```bash
# 1. HTTP-level analysis (auth, captcha, pricing signals, dead domains)
.venv/bin/python analyze_directories.py

# 2. Cleanup obvious failures + build browser check list
.venv/bin/python cleanup_and_categorize.py

# 3. Browser verification with Playwright (10 concurrent workers)
.venv/bin/python browser_verify.py

# 4. Deep recheck any remaining unknowns
.venv/bin/python browser_verify.py --recheck-unknown
```

Each script reads/writes `directories.json`. Steps 3-4 use `browser_check_list.json` as intermediate state (generated by step 2).

After completion, report the summary: how many added, and the auth/status breakdown for the new entries.

## Step 5: Discover Forms

For directories that are active and have `auth_type = none` or `auth_type = email_password`:

```bash
# Discover form fields on submission pages
.venv/bin/python discover_forms.py
```

This visits each submission URL with Playwright, extracts form fields via DOM queries, and updates `submission_plan.json` with discovered fields and form paths.

## Step 6: Submit

### Automated Submission

Configure the PRODUCT dict in `submit_directories.py` with your details (search for `YOUR_` placeholders), then:

```bash
# Auto-submit to all discovered directories
.venv/bin/python submit_directories.py
```

The script uses heuristic field mapping (matching field names/labels to product data) and handles file uploads for logo/screenshot.

### Manual Browser Submission (via Playwright MCP)

For directories that need manual interaction (captcha, OAuth, complex forms), use the Playwright browser tools:

1. Navigate to the submission URL
2. Take a snapshot to understand the page structure
3. Fill form fields using `browser_fill_form` or `browser_type`
4. Handle OAuth flows by switching tabs when Google login popups open
5. Upload files via `browser_file_upload`
6. Click submit and verify confirmation

### GitHub PR Submissions

Some directories accept submissions via GitHub PRs to awesome-lists:

1. Fork the repo: `gh repo fork <owner>/<repo>`
2. Clone and create a branch
3. Add your product entry following the repo's format
4. Push and create PR: `gh pr create`

## Notes

### Pipeline Scripts
- `analyze_directories.py` uses `ThreadPoolExecutor` with plain HTTP — fast first pass
- `cleanup_and_categorize.py` triages errors (dead domains, invalid URLs, Facebook groups) and builds `browser_check_list.json`
- `browser_verify.py` uses async Playwright with 10 concurrent tabs; `--recheck-unknown` does a deep DOM pass on active unknowns only
- `discover_forms.py` uses async Playwright with 10 concurrent tabs; extracts form field names, types, labels, and paths
- `submit_directories.py` uses async Playwright with 5 concurrent tabs; heuristic field mapping with file upload support
- All scripts are idempotent — safe to re-run

### Common Submission Blockers

When evaluating or submitting to directories, watch for these issues:

| Blocker | Frequency | How to Detect |
|---|---|---|
| **Paid listing required** | ~20% | Look for pricing page, Stripe/PayPal links, "$" on submit page |
| **reCAPTCHA / Turnstile** | ~10% | `iframe[src*=recaptcha]` or `[data-turnstile]` elements |
| **Broken captcha** | ~2% | "Invalid site key" errors, disabled submit buttons |
| **Login/account required** | ~15% | Redirect to `/login` or `/register` on submit URL |
| **Business email required** | ~3% | Rejects gmail/yahoo domains (e.g., SoftwareSuggest) |
| **Reciprocal link required** | ~5% | Old web directories require backlink before listing |
| **Newsletter-only forms** | ~10% | Page looks like submit but is actually email signup |
| **Backend API broken** | ~2% | Form submits but returns GraphQL/API errors |
| **Domain parked/dead** | ~8% | No content, parking page, DNS failure |
| **Cloudflare blocked** | ~3% | Challenge page, 403 errors |

### Automation Tips

- **Simple HTML forms** have highest auto-submit success rate
- **reCAPTCHA v3** (invisible) sometimes passes; **v2** (checkbox) never does automatically
- **Google Forms** are reliably automatable
- **Rich text editors** (TinyMCE, Quill) need `browser_evaluate` to set content
- **Cloudinary/custom upload widgets** often break automation — use manual browser
- **Cross-origin OAuth popups**: Switch tabs with `browser_tabs` action to handle Google login
- **Combobox/select fields**: Use `browser_click` on the dropdown, then click the option
- **Multi-step forms**: Take snapshot after each step to see new fields

### Submission Plan Structure

Each entry in `submission_plan.json` contains:

```json
{
  "directory_name": "Example AI",
  "submission_url": "https://example.com/submit",
  "status": "discovered",
  "copy": {
    "title": "Product Title Variation",
    "description": "Product description variation for this directory."
  },
  "discovered_fields": [...],
  "form_path": "form#submit-form",
  "credentials": {
    "email": "YOUR_EMAIL",
    "name": "YOUR_NAME",
    "username": "YOUR_USERNAME",
    "password": "YOUR_PASSWORD"
  }
}
```

Status values: `discovered`, `submitted`, `skipped`, `skipped_paid`, `timeout`, `no_form_found`, `no_fields_matched`, `submit_timeout`, `captcha`, `cloudflare_blocked`, `domain_parked`, `skipped_login_required`, `deferred`.

### Best ROI Directory Types (for AI/SaaS products)

1. **AI tool directories** with simple forms (FutureTools, SaaSHub, AItools.inc, etc.)
2. **Startup directories** with Google Form submissions
3. **GitHub awesome-lists** accepting PRs (free, high-quality backlinks)
4. **NoCode/SaaS aggregators** (NoCodeList, NoCodeDevs)
5. **General web directories** with DA≥30 (for SEO value)

### Security Note

Before pushing to GitHub, ensure all personal data is stripped:
- Search for `YOUR_` placeholders in `submission_plan.json` and `submit_directories.py`
- Never commit real emails, passwords, or API keys
- The `.playwright-mcp/` folder may contain console logs with personal data — add to `.gitignore`