UAP Release Analyzer

SKILL.md

---
name: uap-release-analyzer
version: 1.0.1
description: Use when analyzing folders/tranches of declassified UAP/UFO/FOIA PDFs such as war.gov/UFO PURSUE, FBI Vault, NARA boxes, or AARO releases. Runs inventory, text extraction, entity/redaction analytics, and builds a standardized 11-section REPORT.md with caveats.
author: Chase Wang; OpenClaw port by Hermes Agent
license: MIT
---
# UAP / Declassified Release Analyzer

> OpenClaw port note: this is an OpenClaw-compatible skill adapted from Chase Wang's MIT-licensed `ckpxgfnksd-max/uap-release-analyzer`. Use the bundled scripts from the installed skill directory, e.g. `python3 ~/.openclaw/workspace/skills/uap-release-analyzer/scripts/run_all.py <release_root>`. If your OpenClaw skills directory is customized, locate the installed `uap-release-analyzer` directory and run the same `scripts/run_all.py` entrypoint there.

This skill turns a folder of declassified UAP/UFO documents into a structured analytic report. It was built from a real workflow against the May 2026 war.gov/UFO/ "PURSUE" tranche (162 files, 4,000+ pages, mixed FBI/DOW/NASA/DOS/NARA sources), so it's tuned to the quirks of that universe — but it generalizes to any tranche of FOIA-released government PDFs.

## When to use

Trigger on prompts like "analyze the UFO files I just downloaded", "build me a report on this UAP release", "what's in `~/Downloads/release_01/`?", "compare release 1 and release 2", "find redaction patterns in these FBI files", "summarize this AARO PDF", or whenever the user references a directory of declassified documents and wants any kind of summary, inventory, or pattern surfacing. Also trigger if the user just dumps a path and asks "what's interesting in here?" — this skill is the right tool.

## Why a skill

The work has a fixed shape that repeats across every new tranche:

1. Inventory — what files came down, sizes, page counts, which agency.
2. Text extraction — pull text where there is a text layer; flag the (often majority) of files that are scanned and need OCR.
3. Entity surfacing — locations, agencies, phenomena vocabulary, named people.
4. Redaction pattern analysis — which FOIA exemptions show up where, which files are most redacted.
5. Cross-document patterns — year clusters, agency × location heatmap, names that appear in 5+ files.
6. A standardized report the user can read in ten minutes.

Doing this freshly every time wastes effort and produces inconsistent outputs. The bundled scripts make every tranche analyzable the same way.

## The standard workflow

Run scripts in this order. Each writes intermediate artifacts that the next step consumes. They are **idempotent and incremental** — re-running on the same folder skips work that's already done.

```
release_root/
  release_NN/                 # the actual PDFs/PNGs/JPGs (input)
  text/                       # extracted text per PDF (created)
  inventory.csv               # one row per file (created)
  analytics/                  # aggregated outputs (created)
    top_terms.csv
    terms_by_agency.csv
    entities.json
    per_file_digest.csv
    cross_doc.json
  REPORT.md                   # human-readable analytic writeup (created)
```

**Step 1 — Inventory.** Run `scripts/inventory.py <release_root>`. This walks the release directory, classifies each file by filename prefix (see `references/agency_vocab.md`), reads PDF page counts, and writes `inventory.csv`. Don't write inventory by hand — the script handles encrypted PDFs, weird filenames with spaces or em-dashes, and files that pypdf can't open.

**Step 2 — Text extraction.** Run `scripts/extract_text.py <release_root> [start] [end]`. Extracts text via pdfplumber, writing one `.txt` per PDF into `text/`. Skips files that already have a non-empty `.txt`. Many FBI / NARA / older photo-PDFs have **no text layer** — those will produce 0-char files; that's expected and fine, the analytics treat them as "scanned, OCR needed". The optional `[start] [end]` slice arguments let you process in chunks if your sandbox has a per-call timeout (the war.gov FBI sections are 200+ pages each — extract them in batches of ~25 if running in a 45-second-call environment).

**Run scripts in the foreground of your turn**, not via background-and-end-turn patterns. The pipeline is fast enough (a few minutes from cold) that you can stay in-turn. If a single `extract_text.py` call would actually time out, prefer the `[start] [end]` chunking pattern over backgrounding — chunked calls each finish quickly, the script is idempotent, and progress is visible.

**Step 3 — Analytics.** Run `scripts/analyze.py <release_root>`. Reads the extracted text + inventory, then writes the contents of `analytics/`. This is fast even on 800K+ characters of text.

**Step 4 — Report.** Run `scripts/build_report.py <release_root>`. Reads inventory + analytics and writes a `REPORT.md` with the sections listed under "Report structure" below.

When the user just says "analyze the release at `<path>`", run all four in sequence with that path. When they ask a narrower question ("how many files?", "which file is most redacted?"), call only the relevant script or read the existing artifacts directly.

## Report structure

Always use this exact section order in `REPORT.md` so reports across tranches stay comparable. If a section has no data for this tranche, leave a one-line "no data" note — don't omit the heading.

```
# <Release name> — Raw Analytics
**Source:** ... · **Cleared for release:** ...
**Files in this analysis:** N of M (note any gaps)

## 1. Inventory                    — counts, total size, page counts, by agency
## 2. What's actually in the release  — narrative summary of the major buckets
## 3. Where the activity is concentrated  — top locations
## 4. Phenomena terminology         — UAP/craft/orb/disc/etc. with counts
## 5. Agency cross-references       — agencies named in text
## 6. Year clusters                 — when is this material from
## 7. Redactions                    — top markers + most-redacted files
## 8. Notable individual files
## 9. Cross-document patterns
## 10. What's missing / caveats     — OCR gaps, files we couldn't pull, etc.
## 11. Files in this analysis       — paths to inventory.csv / analytics/*
```

The "What's missing" section matters — it's what makes the report honest. Always call out files we couldn't OCR, files referenced on a source page but not downloaded, and heuristic limits of the entity extraction.

## Agency classification

Files are classified by filename prefix. The full vocabulary is in `references/agency_vocab.md`. The high-confidence prefixes from the war.gov universe:

- `65_hs1*`, `fbi-photo-*`, `usper-*`, `serial*`, `2024-04-30-*` → FBI
- `dow-uap*`, `western_us_event*` → DOW (Department of War)
- `nasa-uap*` → NASA
- `dos-uap*`, `059uap*` → DOS (State)
- `18_*`, `38_*`, `59_*`, `255*`, `331_*`, `341_*`, `342_*` → NARA (record-group prefixes)
- otherwise → OTHER (flag for the user; might be a new bucket worth adding to the vocab)

If you encounter a tranche with prefixes not in the vocab, add them to `references/agency_vocab.md` (the table) and `scripts/inventory.py` + `scripts/analyze.py` (`PREFIX_RULES`) rather than scattering inline filename checks across scripts. A useful threshold: if `OTHER` exceeds ~3% of files in any tranche, that's a signal the vocab needs extending, not the data being weird.

When bootstrapping a brand-new tranche (e.g., the user has just downloaded `release_02/` and asks "what's the fastest way to a written report?"), surface this vocab-extension workflow in your reply alongside `run_all.py`. Otherwise the user will discover the OTHER bucket only after the fact.

## FOIA / classification markers

`references/foia_codes.md` lists the FOIA exemptions and classification stamps to look for. Most of the meaningful redaction signal in modern tranches comes from `(b)(1)` (national security), `(b)(3)` (statutory), `(b)(6)` (personal privacy), and the classification banners `SECRET//NOFORN`, `REL TO USA`, `CUI`, `FOUO`. The analyzer counts these by file so the report can name the most-redacted documents.

## Pulling files from war.gov/UFO/ (PURSUE releases)

If the user wants to pull from the source page rather than analyzing files they already have, see `references/war_gov_quirks.md`. It documents the things that bit us on the first run: the page renders 10 records per page across paginated DOM, the "Download" button is hooked through `<a download>`-style behavior so URLs are only available after the modal opens, ~28 of 162 records are served via inline viewer (no clean URL), and `www.war.gov` is typically not on a workspace egress allowlist (you'll need browser-driven downloads or a one-time allowlist). Don't reinvent the scrape — check the reference first.

## Working with the user

- **Start narrow.** Ask for the path if they didn't give one. Don't guess.
- **Show progress.** Tranches are big (the May 2026 release was 2.5 GB / 4,000 pages); print `[N/total]` lines as you go so the user isn't flying blind.
- **Don't run OCR by default.** Tesseract on 4,000 scanned pages takes hours. Note the gap in the "What's missing" section and offer OCR as a follow-up.
- **Surface cross-tranche links** if the user has more than one release in the parent folder — a sibling `release_02/` makes "what's new vs. release_01?" the obvious next question.
- **Honest caveats.** Entity extraction here is keyword-list + regex, not full NER. Year mentions ≠ incident dates. Say so in the report.

## Bundled scripts

- `scripts/inventory.py <release_root>` — build inventory.csv
- `scripts/extract_text.py <release_root> [start] [end]` — extract text in optional chunks
- `scripts/analyze.py <release_root>` — write analytics/
- `scripts/build_report.py <release_root>` — write REPORT.md
- `scripts/run_all.py <release_root>` — convenience: run the four in order

## Bundled references

- `references/agency_vocab.md` — filename-prefix → agency rules
- `references/foia_codes.md` — FOIA exemptions and classification stamps
- `references/war_gov_quirks.md` — how war.gov/UFO/ is structured + scraping notes
- `references/openclaw_usage.md` — OpenClaw/ClawHub install and usage notes

Read references on demand. Don't preload them into context unless the user's question is in their domain.