UAP / Declassified Release Analyzer

intent

this skill turns a folder of declassified UAP/UFO documents into a structured analytic report with inventory, text extraction, entity surfacing, redaction pattern analysis, and cross-document patterns. use it when the user references a directory of declassified government PDFs (war.gov/UFO PURSUE, FBI Vault, NARA boxes, AARO releases, etc.) and wants any kind of summary, inventory, pattern analysis, or structured writeup. the skill is tuned to FOIA government document quirks (scanned PDFs, mixed agency sources, redaction markers, filename conventions) but generalizes to any tranche of declassified materials.

inputs

release directory (required)

• user provides a local path to a folder containing PDF/PNG/JPG files
• script walks the directory tree recursively
• env var: none required; path is a command-line argument

agency vocabulary (bundled reference)

• file: references/agency_vocab.md
• maps filename prefixes to agencies (FBI, DOW, NASA, DOS, NARA, OTHER)
• used by inventory.py and analyze.py to classify files
• if a new tranche has unknown prefixes, you extend this vocab before re-running

FOIA exemption codes (bundled reference)

• file: references/foia_codes.md
• lists FOIA exemptions (b)(1), (b)(3), (b)(6) and classification banners SECRET//NOFORN, REL TO USA, CUI, FOUO
• used by analyze.py to count redaction markers per file

war.gov scraping notes (bundled reference, conditional)

• file: references/war_gov_quirks.md
• only needed if user wants to pull from war.gov/UFO/ source page rather than local files
• documents pagination, download button behavior, inline viewer limitations, egress allowlist requirements

Python dependencies (bundled with skill)

• pdfplumber for text extraction (handles encrypted PDFs gracefully)
• pypdf for page counting (falls back gracefully on unreadable PDFs)
• pandas for CSV generation
• all installed in skill's virtualenv; no user setup required

external connections: none

• this skill operates on local files only
• no API keys, OAuth tokens, or network calls needed
• if user wants to download from war.gov, that happens outside this skill (see references)

procedure

run scripts in this exact order. each script writes intermediate artifacts that the next step consumes. all scripts are idempotent and incremental: re-running on the same folder skips work already done.

step 1 , inventory

• input: <release_root> folder path
• run: python3 ~/.openclaw/workspace/skills/uap-release-analyzer/scripts/inventory.py <release_root>
• output: <release_root>/inventory.csv with one row per file
• output columns: filename, file_path, agency_classification, page_count, file_size_bytes, text_extraction_status
• output: inventory.csv is sorted by agency, then filename
• the script handles encrypted PDFs, weird filenames with spaces or hyphens, and PDF files that pypdf can't parse (logs the error, records page_count as 0)
• do not write inventory by hand; the script is the source of truth

step 2 , text extraction

• input: <release_root> folder path, optional [start] and [end] integer arguments for chunking
• run: python3 ~/.openclaw/workspace/skills/uap-release-analyzer/scripts/extract_text.py <release_root> (or with [start] [end] slice for chunked processing)
• output: <release_root>/text/ directory with one .txt file per PDF (named same as source PDF, extension .txt)
• output: a .txt file with 0 characters means the PDF had no text layer (scanned image PDFs) , this is expected and fine
• output: a .txt file with N > 0 characters means text extraction succeeded
• the script skips any PDF that already has a non-empty .txt file (idempotent)
• chunking: if your sandbox has a per-call timeout, use [start] [end] slice arguments (e.g., extract_text.py <release_root> 0 25 then extract_text.py <release_root> 25 50 etc.) to process in batches. each batch completes quickly and you stay in-turn
• do not background this step; run it in the foreground so you see progress and can chain to the next step
• expected outcome: 60-80% of files yield non-empty text, 20-40% are scanned and produce empty .txt files

step 3 , analytics

• input: <release_root>/text/ directory and <release_root>/inventory.csv
• run: python3 ~/.openclaw/workspace/skills/uap-release-analyzer/scripts/analyze.py <release_root>
• output: <release_root>/analytics/ directory containing:
  • top_terms.csv (term, frequency, percent_of_corpus) , sorted by frequency descending
  • terms_by_agency.csv (agency, term, frequency) , UAP phenomenon vocabulary by agency source
  • entities.json (locations, agency_names, person_names, phenomena_keywords) , structured entity extraction
  • per_file_digest.csv (filename, extracted_chars, entity_count, redaction_marker_count, top_terms_in_file)
  • cross_doc.json (names_in_5plus_files, location_agency_heatmap, year_clusters)
• the script is fast even on 800K+ characters of extracted text; no timeout risk
• entity extraction uses keyword lists and regex, not full NLP NER (see caveat in step 4)

step 4 , report

• input: <release_root>/inventory.csv and <release_root>/analytics/ directory
• run: python3 ~/.openclaw/workspace/skills/uap-release-analyzer/scripts/build_report.py <release_root>
• output: <release_root>/REPORT.md with 11 sections in fixed order (see output contract)
• the report is human-readable in 10 minutes; includes data summaries, narrative patterns, caveats, and file paths
• the report names which files are most redacted, which locations dominate, and which entities span multiple documents

convenience entrypoint

• run: python3 ~/.openclaw/workspace/skills/uap-release-analyzer/scripts/run_all.py <release_root>
• output: runs steps 1-4 in sequence with the same path
• use this when the user says "analyze the release at <path>" with no other context

decision points

if the user provides a path: use it as <release_root>. confirm the path exists and contains files before starting.

if the user does not provide a path: ask for the folder path. do not guess or assume a default location.

if the release is very large (>1000 files or >2GB): ask if the user wants to run the full pipeline or start with inventory only (step 1) to preview scope before committing compute time.

if extract_text.py would timeout on a single call: use [start] [end] chunking instead of backgrounding the job. chunk size of 25-30 files per call stays well under a 45-second timeout. run chunks sequentially in-turn; the script is idempotent so each chunk is independent.

if a file has 0-character extracted text (scanned PDF): this is expected. do not flag it as an error. note it in the "What's missing" section of the report. offer OCR (Tesseract) as a follow-up only if the user asks or if scanned files are >40% of the corpus.

if inventory.py encounters an encrypted PDF: the script logs the error and records page_count as 0. this is acceptable; continue to the next file. note in the report if encrypted files are >5% of the tranche.

if analyze.py finds >3% of files classified as OTHER: signal the user that the agency vocabulary likely needs extending. surface the vocab-extension workflow: ask the user to check the filenames of the OTHER files, add new prefix rules to references/agency_vocab.md, update PREFIX_RULES in scripts/inventory.py and scripts/analyze.py, then re-run inventory and analytics.

if the user asks a narrow question ("how many files?", "which file is most redacted?", "what locations appear?") without requesting a full report: answer by running only the relevant script or reading existing artifacts directly (e.g., read inventory.csv for file counts, read analytics/per_file_digest.csv for redaction markers). do not run the full pipeline if not needed.

if the user has multiple releases in the parent folder (e.g., release_01/ and release_02/): surface the cross-release comparison as the obvious next question. prompt: "want me to compare release_01 and release_02 to surface what's new?"

if the user wants to download from war.gov/UFO/ instead of analyzing local files: refer them to references/war_gov_quirks.md for scraping guidance. note that war.gov is typically not on workspace egress allowlists; they may need browser-driven downloads or a one-time allowlist request. do not implement a scraper within this skill , that is out of scope.

output contract

primary output: REPORT.md

• location: <release_root>/REPORT.md
• format: markdown with 11 sections in this exact order
• success: file exists, is readable, and contains all 11 section headings even if a section is empty

report structure (11 sections in order)

1. inventory , counts, total size, page counts, by agency
2. what's actually in the release , narrative summary of major content buckets
3. where the activity is concentrated , top locations with counts
4. phenomena terminology , UAP/craft/orb/disc/etc. with frequency counts
5. agency cross-references , agencies mentioned in extracted text with frequency
6. year clusters , when the materials originate from (year mentions with counts)
7. redactions , top FOIA exemption markers and classification banners; list of most-redacted files
8. notable individual files , 2-3 files worth the user reading directly (based on redaction level, entity density, or patterns)
9. cross-document patterns , names in 5+ files, location-agency heatmap, repeated entities
10. what's missing / caveats , OCR gaps, files referenced but not downloaded, entity extraction limits (keyword + regex, not NER), year mentions ≠ incident dates
11. files in this analysis , file paths to inventory.csv and analytics/* for user inspection

intermediate outputs

• inventory.csv: columns (filename, file_path, agency_classification, page_count, file_size_bytes, text_extraction_status), sorted by agency then filename
• text/ directory: one .txt file per PDF, named <original_filename>.txt
• analytics/top_terms.csv: (term, frequency, percent_of_corpus), descending by frequency
• analytics/terms_by_agency.csv: (agency, term, frequency)
• analytics/entities.json: structured object with keys locations, agency_names, person_names, phenomena_keywords, each a list with counts
• analytics/per_file_digest.csv: (filename, extracted_chars, entity_count, redaction_marker_count, top_terms_in_file)
• analytics/cross_doc.json: structured object with keys names_in_5plus_files, location_agency_heatmap, year_clusters

data quality notes

• entity extraction uses keyword lists and regex, not full NLP NER. coverage is ~70-80% for common UAP terms and agency names; less for obscure locations or acronyms.
• year mentions (e.g., "1952") are extracted but may not represent incident dates; document this in the report.
• redaction counts are based on text markers (e.g., [REDACTED], (b)(1)) in extracted text. scanned files with redactions as image pixels will not be counted. note this in "what's missing".
• if <60% of the corpus extracts to non-empty text, OCR is a prerequisite for deeper analysis.

outcome signal

the user knows the skill worked when:

1. inventory.csv exists and is readable , the user can open it, see N rows of files, and confirm page counts and agency classifications look plausible.

2. text/ directory is populated , at least 50% of input PDFs have a corresponding .txt file with >0 characters. files with 0-character .txt are flagged in the report as "scanned, no text layer".

3. REPORT.md is written and human-readable , the user can open it in 10 minutes, understand what's in the release, which files are most redacted, what locations and phenomena dominate, and what gaps exist.

4. analytics/ directory contains structured data , the user can inspect top_terms.csv, entities.json, and cross_doc.json for deeper analysis or custom queries.

5. user gets a concrete answer to their original question , if they asked "how many files", they see N and agency breakdown. if they asked "what's most redacted", they see a ranked list. if they asked "build me a report", they get REPORT.md ready to read or share.

6. progress was visible during execution , the user saw [N/total] output lines as scripts ran, so they weren't flying blind on a large tranche.

7. caveats are explicit , the "what's missing" section names OCR gaps, scanned files, and entity extraction limits. the user knows what they're reading and what they're not.