Private Document AI with OpenVINO

intent

turn local PDFs, invoices, screenshots, and diagrams into structured data or executable code scaffolds using PaddleOCR-VL or PaddleOCR-VL-1.5 on OpenVINO. run everything locally on Intel CPU/GPU hardware with no cloud API calls. use this when you need to extract tables, key-value pairs, and entities from documents; classify documents by type; or convert UI mockups and architecture diagrams into HTML, React, or Jupyter notebook boilerplate. the skill is built for demos and production workflows where document privacy and local inference are non-negotiable.

inputs

required parameters

• file: absolute path to PDF, PNG, JPG, or screenshot file
• mode: one of parse, to-data, or to-code
• out: absolute path to output artifact directory

optional parameters

• extract: comma-separated list of extraction targets (e.g. tables,entities,kv_pairs) for to-data mode
• target: code generation target for to-code mode; one of react, html-css, json-schema, jupyter-notebook
• title: human-readable title for generated artifacts (used in notebooks and code previews)

environment variables and external connections

• PADDLEOCR_VL_OPENVINO_MODEL_DIR: path to PaddleOCR-VL model assets (optional; defaults to {baseDir}/models/paddleocr-vl-openvino/)
• PADDLEOCR_VL_LAYOUT_MODEL_DIR: path to layout detection models (optional)
• PADDLEOCR_VL_VLM_MODEL_DIR: path to vision-language model files (optional)
• python 3.8+, pip, virtual environment recommended
• OpenVINO runtime (auto-installed via requirements.txt)
• PaddleOCR-VL binaries (install from reviewed local wheel only after user approval)

local setup

• run python "{baseDir}/scripts/check_env.py" to verify python, pip, OpenVINO, and model asset paths before any document processing
• run python -m pip install -r "{baseDir}/requirements.txt" in a dedicated virtual environment
• model auto-download is disabled by default; user must explicitly approve any model fetching

procedure

1. environment check (input: none; output: env_status.txt)

   • execute python "{baseDir}/scripts/check_env.py"
   • verify python version, pip, OpenVINO installation
   • check for model asset directories and report missing paths
   • if any check fails, halt and report missing dependency

2. input validation (input: file, mode, out; output: validation result)

   • confirm file exists and is readable (PDF, PNG, JPG, or screenshot)
   • confirm mode is one of parse, to-data, or to-code
   • confirm out directory path is writable or can be created
   • if mode is to-data, confirm extract parameter contains valid targets
   • if mode is to-code, confirm target is one of react, html-css, json-schema, jupyter-notebook
   • if validation fails, write error.json with failure details and halt

3. document parse (input: file; output: parsed.json, parsed.md, result_report.html)

   • invoke python "{baseDir}/scripts/run_skill.py" with --mode parse --file <file> --out <out>
   • run PaddleOCR-VL document understanding on the local file
   • extract layout blocks, text, tables, figures, and reading order
   • preserve page numbers, block type labels, and confidence scores
   • mark low-confidence regions explicitly in output
   • do not silently drop tables, figures, formulas, or key-value regions
   • write canonical structured output to parsed.json (JSON format with all blocks and metadata)
   • write human-readable markdown summary to parsed.md
   • generate interactive HTML report to result_report.html for visual review
   • save effective runtime config to effective_config.json
   • save execution report to run_report.json (timing, block counts, warnings)

4. mode branch: to-data extraction (input: parsed.json, extract parameter; output: JSON files under task_output/)

   • if mode is to-data, invoke downstream extraction transform
   • parse extract parameter; default to tables,entities,kv_pairs if not specified
   • for tables: identify table blocks in parsed.json, extract row/column structure, output table_index.json with page anchors
   • for entities: apply named-entity recognition on text blocks, classify entities (person, org, amount, date, etc.), output entities.json
   • for kv_pairs: locate key-value patterns (labels followed by values), output kv_pairs.json
   • optionally apply document classification (invoice, receipt, form, report, etc.) and output classification.json
   • normalize extracted fields (dates, amounts, phone numbers) to standard formats, output normalized.json
   • generate structured_record.json combining all extracted data into a single flattened document record
   • write traceability.json linking each extracted field back to its source block ID, page number, and confidence
   • if extraction produces no results for a requested target, note it in traceability.json with reason

5. mode branch: to-code generation (input: parsed.json, target parameter; output: code/notebook under task_output/)

   • if mode is to-code, invoke downstream code generation transform
   • confirm target is one of react, html-css, json-schema, jupyter-notebook
   • analyze document structure from parsed.json to identify UI components (buttons, fields, text areas, checkboxes, dropdowns)
   • for react: generate component hierarchy, map form fields to state, output component_map.json, app.jsx, index.html, styles.css
   • for html-css: generate static HTML with inline styles, output index.html, styles.css
   • for json-schema: extract form structure and generate JSON Schema, output schema.json and field_schema.json
   • for jupyter-notebook: convert diagrams or forms into executable notebook boilerplate, output notebook.ipynb and notebook_plan.json (with explanatory cells and setup instructions)
   • include ui_blueprint.json showing field names, types, constraints, and relationships
   • include notes.md with assumptions and manual review warnings
   • write traceability.json linking generated code elements back to source document blocks
   • mark all outputs as drafts; do not generate production-ready code
   • generate code_preview.html showing syntax-highlighted code side-by-side with source document preview

6. artifact organization (input: all generated files; output: {out} directory structure)

   • create output directory at {out} if it does not exist
   • write top-level files: effective_config.json, run_report.json, parsed.json, parsed.md, result_report.html
   • create task_output/ subdirectory
   • write mode-specific artifacts into task_output/ (e.g. structured_record.json, notebook.ipynb, component_map.json)
   • if mode is to-code, also write code_preview.html at top level
   • use consistent JSON formatting (2-space indent)
   • do not overwrite existing files; use sequential naming (e.g. notebook_1.ipynb, notebook_2.ipynb) if needed

7. smoke test (optional, diagnostic only) (input: none; output: smoke test report)

   • if user requests verification before production use, run python "{baseDir}/scripts/smoke_test.py"
   • confirms PaddleOCR-VL integration and model loading
   • does not process real documents; uses synthetic test data only

decision points

• if user provides file but no mode: default to parse and proceed to step 3. recommend explicit mode selection in future calls.

• if mode is parse: execute steps 1-3 and 6 only; skip extraction and code generation. output parsed.json, parsed.md, result_report.html.

• if mode is to-data but extract parameter is missing: default to tables,entities,kv_pairs. if the document contains no tables, do not fail; note in traceability.json that no table blocks were found.

• if mode is to-code but target parameter is missing: halt with error message listing valid targets; do not assume a default target.

• if file does not exist or is not readable: write error.json with file-not-found message and halt before step 1. do not attempt to create or download the file.

• if model assets are not found (e.g. PADDLEOCR_VL_OPENVINO_MODEL_DIR does not exist and models are not in default location): halt with clear message listing expected paths and recommend python "{baseDir}/scripts/check_env.py". do not auto-download models without explicit user approval via a separate approval workflow or flag.

• if document parse fails (e.g. corrupted PDF, unsupported image format, model inference error): write error.json with stage name, error message, and stderr excerpt. do not claim partial outputs. recommend retrying with a different file or running smoke_test.py to diagnose environment issues.

• if extraction produces empty result set (e.g. document is a blank page or image with no text): write structured_record.json with empty arrays and note in run_report.json that no extractable content was found. do not fail; mark as complete with zero results.

• if code generation encounters unsupported diagram type (e.g. flowchart in to-code mode targeting react): write note in notes.md and ui_blueprint.json that the diagram could not be fully converted. provide partial boilerplate. mark as incomplete in traceability.json.

• if output directory path is not writable: attempt to create parent directories. if creation fails, write error message to stderr and halt before step 6. do not fall back to temp directory without user consent.

• if user requests model auto-download: check for explicit flag or environment variable (e.g. ALLOW_MODEL_DOWNLOAD=1). if not set, halt with message explaining the flag and security implications. do not download without approval.

• if runtime encounters network timeout or rate limit (e.g. from a reference lookup): log timeout in run_report.json, continue with available data, and mark affected fields in traceability.json as incomplete. do not retry or hang.

• if OpenVINO model inference times out (e.g. document is very large): log timeout, write parsed.json with partial results up to timeout point, and note in run_report.json that processing was interrupted. do not claim full document coverage.

output contract

default output location: ./artifacts/<document_stem>/ where <document_stem> is the filename without extension

top-level files (always present)

• effective_config.json: JSON object with all runtime parameters and environment paths used in this run
• run_report.json: execution report including stage timings, block/table counts, warnings, and final status (success, partial, failure)
• parsed.json: canonical structured document parse with all blocks, layout info, page numbers, and confidence scores (schema: see {baseDir}/references/schema.md)
• parsed.md: human-readable markdown summary of extracted content
• result_report.html: interactive HTML report with document preview, extracted tables, and navigable block inspector

mode-specific outputs under task_output/ subdirectory

for to-data mode:

• entities.json: array of named entities with type labels and source block references
• kv_pairs.json: array of key-value pairs with page numbers and confidence scores
• table_index.json: array of table metadata (page, position, row count, column count) with content preview
• classification.json: document type classification and confidence (optional, if classification is requested)
• normalized.json: extracted fields normalized to standard formats (ISO dates, standardized phone/address/currency formats)
• structured_record.json: single flattened JSON object combining all extracted fields, suitable for database insert or API submission
• traceability.json: mapping of each field in structured_record.json to source block ID, page number, and confidence score

for to-code mode:

• component_map.json: hierarchical component tree for UI generation (fields, labels, groups, relationships)
• ui_blueprint.json: field schema including names, types (text, number, date, select, checkbox), constraints, and validation rules
• field_schema.json: detailed field metadata (label, placeholder, required, options for selects)
• notes.md: assumptions made during code generation, manual review checklist, and warnings
• target-specific artifacts:
  • react: app.jsx, index.html, styles.css, package.json sketch
  • html-css: index.html (with inline or linked styles), styles.css
  • json-schema: schema.json (full JSON Schema draft 7)
  • jupyter-notebook: notebook.ipynb (executable Jupyter notebook with setup cells, imports, and explanatory markdown), notebook_plan.json (cell structure and metadata)
• traceability.json: mapping of generated code elements (e.g. component names, field IDs) back to source document blocks
• code_preview.html: syntax-highlighted code snippets displayed alongside document preview for visual review

error output (on failure)

• error.json: JSON object with keys: stage (string: which step failed), error_message (string: concise description), stderr_excerpt (string: up to 500 chars of stderr), timestamp (ISO 8601)

all outputs use JSON with 2-space indentation and UTF-8 encoding

outcome signal

the skill worked when:

1. parse mode: parsed.json exists, contains a top-level blocks array with at least one entry (or zero entries if document is blank), run_report.json shows status success or partial, and result_report.html is viewable in a browser showing document preview and extracted content.

2. to-data mode: task_output/structured_record.json exists and is valid JSON with at least one field populated (or empty object if document is blank), traceability.json links each field back to a source block, run_report.json shows status success, and user can open the HTML report to verify extracted data matches visual inspection of the document.

3. to-code mode: task_output/notebook.ipynb or app.jsx (depending on target) exists and is valid code (checked via json.loads() for notebooks, basic syntax validation for JSX), code_preview.html renders the generated code side-by-side with document preview, notes.md contains a review checklist, and user confirms the generated scaffold aligns with the source document layout.

4. error case: error.json exists, contains stage name and error message, and run_report.json shows status failure; user can see which step failed and take corrective action (e.g. check file format, verify model paths, run diagnostics).

5. artifact cleanup: if processing sensitive documents, user deletes the entire {out} directory after review to remove local copies.

all outputs are timestamped in run_report.json and traceability files for audit and debugging.