DataLens Web Scraper

intent

extract structured data (lists, tables, comments, products, reviews, or repeating items) from any website using the DataLens Chrome extension and MCP tools. use this when you need to pull repeating data from a live webpage, expand nested content like replies or load-more buttons, analyze the structure, then run a full or preview scrape with polling and result export. this skill handles the entire workflow from detection through job management.

inputs

external connections:

• DataLens Chrome extension: must be installed and active in your Chrome browser. no API key or OAuth needed.
• Node.js ≥ 18: required to run datalens-mcp-call or npx.
• datalens-mcp-server npm package: install via npm install -g datalens-mcp-server or use npx datalens-mcp-call (no global install required).
• Chrome browser: target webpage must be open in a tab, or you can pass a URL and the extension will open it.

parameters per tool:

• url (optional): full URL of the target webpage. if omitted, the extension uses the currently active Chrome tab.
• prompt (string): natural language description of what you're scraping (e.g. "article list", "product reviews"). used by the AI backend to identify relevant fields.
• rootSelector (string): CSS selector for the container holding all repeating items. obtained from scrape_detect_tables output.
• itemSelector (string): CSS selector for each individual item within the root. obtained from scrape_detect_tables output.
• documentInfoPath (string): JSONPath to the data object inside each item. obtained from scrape_detect_tables output.
• jobId (string): unique identifier returned by scrape_start. used to poll status, pause, resume, stop, or export results.
• jobDraft (object): complete job configuration returned by scrape_analyze_columns. must not be fabricated.
• maxRecords (integer): maximum rows to scrape in one job. use 10 for preview runs, higher for full export.
• outputDir (string): filesystem path where exported files will be saved (e.g. "/tmp/datalens").
• format (string): output format, one of "json", "csv", "xlsx". default is "json".
• waitMs (integer): milliseconds to wait between status polls (e.g. 3000 for 3 seconds).
• expandButtonUids (array): list of expand button identifiers found in scrape_get_table_tree output. used to click nested content before re-detecting.
• limit (integer): max rows to return in inline preview. use 50 for quick inspection.
• cursor (string): pagination token from prior result fetch. omit on first call.

environment variables:

• DATALENS_TIMEOUT: optional. set to milliseconds (e.g. 180000 for 3 minutes) if tools take longer than the default 120 seconds. useful for slow pages or large scrapes.

procedure

1. detect table structure: run datalens-mcp-call scrape_detect_tables '{"url":"<target_url>","prompt":"<what_you_want>"}' to find repeating data on the page. the tool returns a list of detected tables, each with rootSelector, itemSelector, and documentInfoPath. copy the selectors that best match your target data. input: url (optional if tab is open), prompt (required). output: list of table candidates with selectors.

2. (optional) inspect nested content: if the page has collapsible sections, nested replies, or "load more" buttons, run datalens-mcp-call scrape_get_table_tree '{"rootSelector":"<from_step_1>","itemSelector":"<from_step_1>","documentInfoPath":"<from_step_1>"}' to visualize the DOM tree with _uid-annotated expand buttons. input: the three selectors from step 1. output: annotated tree structure showing which buttons can be clicked.

3. (optional) expand and re-detect: if you found expand buttons in step 2, run datalens-mcp-call scrape_click_expand_and_redetect '{"rootSelector":"<from_step_1>","itemSelector":"<from_step_1>","documentInfoPath":"<from_step_1>","expandButtonUids":[{"type":"reply","uids":["<uid_1>","<uid_2>"]}]}' to click those buttons, wait for new content to load, and re-detect the updated structure. input: selectors from step 1 plus array of expand button uids. output: updated selectors to use in subsequent steps. skip this step if there are no nested items.

4. analyze columns and build config: run datalens-mcp-call scrape_analyze_columns '{"rootSelector":"<from_step_1_or_3>","itemSelector":"<from_step_1_or_3>","documentInfoPath":"<from_step_1_or_3>","url":"<target_url>","prompt":"<what_you_want>"}' to call the backend AI, which identifies field names, data types, and pagination strategy. this returns a scraperConfig object and a jobDraft. always confirm the field list matches what you expect before proceeding. input: three selectors (updated if step 3 was used), url, prompt. output: jobDraft (required for step 5) and scraperConfig.

5. start scrape job: run datalens-mcp-call scrape_start '{"jobDraft":<paste_jobDraft_json>,"maxRecords":<number>}' to launch the scrape. use maxRecords: 10 for a preview run first. do not fabricate or modify the jobDraft. input: jobDraft from step 4, maxRecords. output: jobId (string), used for all subsequent operations.

6. poll job status: repeatedly run datalens-mcp-call scrape_status '{"jobId":"<jobId>","waitMs":3000}' until status field is COMPLETED, FAILED, or STOPPED. check the scrapedCount field to monitor progress and error field for failure reasons. input: jobId from step 5, waitMs. output: status (enum), scrapedCount (integer), error (string, if failed).

7. export results: choose one export method per job:

   • to file (recommended for large sets): run datalens-mcp-call scrape_export_to_file '{"jobId":"<jobId>","outputDir":"<path>","format":"<json|csv|xlsx>"}'. input: jobId, outputDir (must exist or be creatable), format. output: file path where results were saved.
   • inline preview (small sets only): run datalens-mcp-call scrape_result '{"jobId":"<jobId>","limit":50}'. input: jobId, limit. output: array of rows plus optional cursor for pagination. re-run with the cursor value to fetch next page.
   • in-memory base64 (for passing to other tools): run datalens-mcp-call scrape_export '{"jobId":"<jobId>","format":"<json|csv>"}'. input: jobId, format. output: base64-encoded file content (string).

8. (optional) manage running jobs: pause with datalens-mcp-call scrape_pause '{"jobId":"<jobId>"}', resume with datalens-mcp-call scrape_resume '{"jobId":"<jobId>"}', or stop with datalens-mcp-call scrape_stop '{"jobId":"<jobId>"}'. input: jobId. output: updated job status.

9. (optional) manage browser tabs: list open tabs with datalens-mcp-call browser_list_tabs, open a new tab with datalens-mcp-call browser_open_tab '{"url":"<url>"}', switch to a tab with datalens-mcp-call browser_use_tab '{"tabId":<id>}', or close a tab with datalens-mcp-call browser_close_tab '{"tabId":<id>}'. input: url (for open) or tabId (for use/close). output: tab list or confirmation. note: scrape_detect_tables with a url arg handles tab opening automatically, so this step is usually not needed.

decision points

• if scrape_detect_tables returns an empty list: the page may require login, be dynamically loaded (JavaScript-heavy), or have no repeating structure. ask the user to log in in Chrome if needed, scroll down to trigger lazy-loading, then retry the detect command.

• if the page has nested/collapsible content: run step 2 (inspect tree) and step 3 (expand and re-detect) before analyzing columns. skip if the page is flat (no nested replies, no load-more buttons).

• if scrape_status stays at QUEUED for over 30 seconds: the Chrome extension may be inactive, the target URL tab may be closed, or the browser may be hung. verify the extension icon is visible in Chrome, verify a tab for the target URL is open, then check debug logs with datalens-mcp-call debug_get_logs '{"levels":["error"]}'.

• if a tool call times out or takes longer than 120 seconds: set DATALENS_TIMEOUT=180000 (or higher) in your environment before running the command. this is common for large pages or slow internet.

• if you want a quick preview before a full scrape: use maxRecords: 10 in scrape_start and scrape_export_to_file or scrape_result to inspect the first few rows. verify the field mapping is correct, then increase maxRecords for the full run.

• if the user asks for spreadsheet output: use format: "csv" or format: "xlsx" in scrape_export_to_file. default is JSON.

• if you need to interrupt or slow down a scrape: use scrape_pause to temporarily halt, scrape_resume to continue, or scrape_stop to kill the job. paused jobs can be resumed; stopped jobs cannot.

• never call scrape_start without a jobDraft from scrape_analyze_columns. fabricating a config will produce incorrect results or errors.

• never skip scrape_analyze_columns and jump straight to scrape_start. the analyze step is mandatory to build the config.

output contract

• scrape_detect_tables: returns JSON array of table candidates. each candidate has rootSelector (string), itemSelector (string), documentInfoPath (string), and optional metadata. use these three strings in all downstream calls.

• scrape_analyze_columns: returns JSON object with scraperConfig (object) and jobDraft (object). the jobDraft is a complete, serializable job descriptor that must be passed verbatim to scrape_start. do not edit or fabricate jobDraft values.

• scrape_start: returns JSON object with jobId (string UUID), status (enum: "QUEUED"), and optional createdAt (timestamp). store the jobId for all subsequent operations.

• scrape_status: returns JSON object with jobId (string), status (enum: "QUEUED", "PREPARING", "RUNNING", "COMPLETED", "FAILED", "STOPPED"), scrapedCount (integer, rows collected so far), totalCount (integer, estimated or null), progress (float 0.0-1.0, optional), error (string, present only on failure), and completedAt (timestamp, present only when status is terminal).

• scrape_export_to_file: returns JSON object with filePath (string, absolute path to saved file), format (string, the format used), recordCount (integer, rows in the export), and fileSize (integer, bytes). the file is created at the specified outputDir with a timestamped name.

• scrape_result: returns JSON object with jobId (string), records (array of row objects), count (integer, rows in this response), cursor (string or null, pagination token for next fetch), and hasMore (boolean, true if more rows exist).

• scrape_export: returns JSON object with format (string), content (string, base64-encoded file data), recordCount (integer), and encodedSize (integer, bytes after base64 encoding).

• all error responses: return JSON object with error (string, human-readable message) and optional code (string, machine-readable error code e.g. "EXTENSION_NOT_ACTIVE", "INVALID_SELECTOR", "TIMEOUT").

outcome signal

• scrape_detect_tables succeeds: you receive a list of table candidates with valid selectors. pick the one that visually matches your target data in the webpage.

• scrape_analyze_columns succeeds: you see a jobDraft and a list of detected fields (e.g. "title", "author", "date") that match what you expect to scrape. if fields are missing or misnamed, re-run with a more specific prompt or check the page layout.

• scrape_start succeeds: you receive a jobId. no error or timeout means the job is queued and the extension is listening.

• scrape_status shows COMPLETED: the scrapedCount field matches or exceeds your expected row count (or is close to totalCount). no error field present.

• scrape_export_to_file succeeds: the file exists at the returned filePath. you can open it in your editor, spreadsheet app, or text viewer and verify rows are present and formatted correctly.

• scrape_result returns data with records array: the array is non-empty and each record has the expected fields from step 4. if hasMore is true and cursor is non-null, call scrape_result again with that cursor to fetch the next page.

• scrape_pause, scrape_resume, scrape_stop succeed: the returned status field changes to "PAUSED" (for pause), "RUNNING" (for resume), or "STOPPED" (for stop). no error field.

• debug_get_logs returns entries: check for "error" and "warn" level entries to diagnose extension connectivity, selector validity, or timeout issues.

• user can see and use the data: final exported file is readable, row count is non-zero, field names and values are intact and sensible, and the user can load it into a spreadsheet, database, or downstream tool without errors.

credits: original skill authored by clawhub. enriched to follow implexa standards with explicit decision points, edge case handling, and detailed output contracts.