Dataify Github Builder

SKILL.md

---
name: "dataify-github-repository-by-repo-url"
description: "Collect structured GitHub repository information from one or more known repository URLs. Do not use for GitHub code search or arbitrary webpages."
---

# Dataify Builder Skill

Use this skill to prepare Dataify builder requests for the scraper family rooted at `github_repository_by-repo-url` on `github.com`.


## Quick Start

**Input:** a GitHub repository URL.

```bash
python3 scripts/build-dataify-request.py --tool-sign github_repository_by-repo-url --params-json '[{"repo_url":"https://github.com/dataify-server/skills"}]'
```

This submits the task, waits for completion, downloads the final result, and returns it. Add `--no-wait` only when submission-only behavior is requested.
## Workflow

1. Check whether `DATAIFY_API_TOKEN` exists in the environment.
2. If the token is missing, stop and tell the user to sign in at [Dataify Dashboard](https://dashboard.dataify.com?utm_source=skill)  to obtain it.
3. Ask the user to choose exactly one tool from the following Chinese list:
- 通过仓库URL采集 (github_repository_by-repo-url)
- 通过搜索URL采集 (github_repository_by-search-url)
- 通过URL采集 (github_repository_by-url)
4. Read `references/tool-params.json` and find the chosen tool by `tool_sign` or Chinese tool name.
5. For each parameter in the chosen tool:
   - If `input_mode` is `user_input`, ask the user for the value.
   - If `input_mode` is `select`, present the saved options to the user.
6. Use `scripts/build-dataify-request.py` as the default cross-platform helper.
7. Use `scripts/build-dataify-request.ps1` as the Windows PowerShell helper when needed.
8. When a selectable parameter has a human-readable Chinese label, keep that label in `spider_parameters`. Do not replace it with a code such as `HK` unless the user explicitly asks for the coded value.
9. Build `spider_parameters` as a JSON array.
10. If every parameter has only one final value, build one object such as `[{"searchurl":"...","country":"Hong Kong"}]`.
11. If one or more parameters have multiple aligned values, zip them by index and build one object per row. Example: `[{"search_url":"url1","page_turning":"1","max_num":"15"},{"search_url":"url2","page_turning":"1","max_num":"15"}]`.
12. If a parameter has one value while another parameter has multiple values, reuse the single value across every generated row.
13. Set `spider_name` to `github.com`.
14. Set `spider_id` to the selected tool's `tool_sign`.
15. Always include `spider_errors=true` and `file_name={{TasksID}}`.
16. Return a curl command for `https://scraperapi.dataify.com/builder`.

## Set DATAIFY_API_TOKEN

Prefer a permanent environment-variable setup instead of setting the token only for the current terminal session.

Windows PowerShell, permanent for the current user:
```powershell
[Environment]::SetEnvironmentVariable("DATAIFY_API_TOKEN", "your_token_here", "User")
```

Then reopen PowerShell. If the current session also needs the token immediately, run:
```powershell
$env:DATAIFY_API_TOKEN = "your_token_here"
```

macOS or Linux, permanent for bash:
```bash
echo 'export DATAIFY_API_TOKEN="your_token_here"' >> ~/.bashrc
source ~/.bashrc
```

macOS or Linux, permanent for zsh:
```bash
echo 'export DATAIFY_API_TOKEN="your_token_here"' >> ~/.zshrc
source ~/.zshrc
```

## Script usage

Python:
```bash
python scripts/build-dataify-request.py --tool-sign <selected_tool_sign> --values-file values.json
```

PowerShell:
```powershell
& ".\scripts\build-dataify-request.ps1" -ToolSign "<selected_tool_sign>" -ValuesFile ".\values.json"
```

The `values.json` file should contain either one object or an array of objects. Example:
```json
[{"searchurl":"https://www.airbnb.com/s/Greece/homes?...","country":"Hong Kong"}]
```

## Required output shape

Generate a curl command in this form:

```bash
curl -X POST 'https://scraperapi.dataify.com/builder' \
  -H "Authorization: Bearer $DATAIFY_API_TOKEN" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'spider_name=github.com' \
  -d 'spider_id=<selected_tool_sign>' \
  -d 'spider_parameters=[{"param":"value"}]' \
  -d 'spider_errors=true' \
  -d 'file_name={{TasksID}}'
```

## Reference usage

- `references/tool-params.json` stores the full saved parameter catalog for every available tool in this scraper family.
- `scripts/build-dataify-request.py` is the portable implementation and should be preferred.
- `scripts/build-dataify-request.ps1` mirrors the same behavior for Windows users.
- If a parameter has no options, the user must provide the value.
- Do not assume `spider_parameters` always contains exactly one object. Multi-value tools may require multiple objects zipped by index.
- Use the saved `url_example` only as a reference example. Do not assume the user wants the example values unless they explicitly confirm them.

## Default completion behavior

The default deliverable is the collected result, not only a `task_id`.

1. Submit the Builder task once and capture its `task_id`.
2. Immediately continue with `$dataify-task-operations` and monitor the same task ID.
   - Use the default 600-second wait for ordinary collections.
   - Use `--timeout 1800` for media downloads or clearly high-volume, multi-page, or multi-input collections.
3. When the task succeeds, download and return the final JSON result. Summarize large payloads while preserving access to the raw result.
4. If monitoring times out or is interrupted, return the task ID and a resume command. Do not resubmit the paid task.
5. Stop after submission only when the user explicitly asks for submission only, a task ID, or `--no-wait` behavior.

## Parameter interaction policy

- For a clear, low-risk, read-only, and low-cost request, apply safe defaults and execute immediately. A short execution summary is optional; do not pause for confirmation.
- Ask only for a missing required input, a material ambiguity, a high-volume or multi-page scope, a media download, a choice that materially changes credit usage, an irreversible action, or an explicit user request to review parameters.
- When confirmation is required, show only user-facing values that affect the target, scope, output, or cost. Prefer one concise sentence; use a compact table only when three or more consequential values are easier to compare.
- Never show fixed fields, empty optional fields, unchanged defaults, credentials, or internal implementation parameters such as engine selectors, response-format flags, offsets, spider IDs, and file-name templates.
- Keep advanced filters hidden unless the user asks for them or they are needed to resolve ambiguity. Never substitute documentation example values for missing required user input.
- After returning results, offer relevant refinements instead of forcing all optional decisions before the first result.

## Account CTA policy

- Show a prominent Dataify account CTA only when the API token is missing, rejected/invalid, or the account has insufficient credits.
- For a missing token, offer https://dashboard.dataify.com/login?utm_source=skill and state: New accounts get 50 free credits, enough for about 6,000 trial results, valid for 7 days, and only successful requests are billed. Never ask the user to paste the token into chat.
- Detect the current operating system and shell. Show only the matching session-scoped setup command first (`export` for macOS/Linux shells, `$env:` for Windows PowerShell, or `set` for Windows Command Prompt). Show other platforms or persistent setup only when detection is ambiguous or the user asks.
- After the user says the token is configured, verify only whether `DATAIFY_API_TOKEN` is present; never print its value. If verification succeeds, continue the original task without asking the user to repeat it.
- Explain that persistent shell changes may require a new terminal or restarting the agent application. Do not recommend a project `.env` unless the execution path explicitly loads it, and ensure `.env` is ignored by version control.
- For an invalid token, direct the user to API-key management without implying that a new registration is required. For insufficient credits, direct the user to balance or recharge management.
- During normal submission, processing, and successful completion, do not promote registration or the Dashboard. Never expose the token or include it in CTA attribution parameters.