clean-text-toolkit

v0.3.0

local toolkit for the grind: read text, find structured bits, clean it up, redact secrets, pass it downstream. python 3 standard library only. no pandas, no nltk, no pip, no remote calls.

companion to clean-csv-toolkit: that handles tabular data, this handles unstructured text.

intent

use clean-text-toolkit when you need to extract patterns from plain text (urls, emails, phones, ips, credit cards, dates, hashtags, color codes, money amounts), redact personally identifiable info (email, phone, ssn, credit cards, aws keys, jwts, uuids), normalize messy encoding and whitespace (bom, crlf, smart quotes, unicode normalization), or perform line-level operations (dedupe, sort, count, shuffle, head, tail). also handles word-frequency stats, text diffs in three modes (unified, side-by-side, html), template substitution without jinja2, url-safe slug generation, and markdown conversion to plain text or minimal html. run locally on your machine with zero external dependencies or network calls.

inputs

• python3 binary (required, verified by scripts/check_deps.sh)
• input text file(s): any encoding (defaults utf-8, falls back to utf-8-sig, cp1252, latin-1)
• optional output file paths (txt, json, jsonl, tsv, html depending on script)
• optional configuration files:
  • --stopwords PATH: newline-delimited word list for wordcount.py
  • --rules PATH: json file of find-and-replace rules for replace.py
  • json object or inline --set key=value for template.py
• no external apis, no authentication required, no network calls

procedure

1. extract structured patterns from text (extract.py)

inputs: text file, pattern kind (url, email, phone, ipv4, ipv6, hashtag, mention, hex-color, money, iso-date), optional flags (--unique, --sort, --with-line)

1. run python3 scripts/extract.py <input-file> --kind <kind> (stdout) or --output <file> (write to file)
2. if --with-line: prefix each match with its source line number
3. if --unique: dedupe matches before output
4. if --sort: alphabetically sort matches
5. output format: one match per line (txt) or json array (--output file.json) or jsonl (--output file.jsonl)
6. check exit code (0 = matches found, 1 = zero matches, 2 = bad args)

outputs: extracted items to stdout or file

2. normalize messy text (normalize.py)

inputs: text file, chain of transforms in command-line order

1. run python3 scripts/normalize.py <input-file> <output-file> [transforms...]
2. apply transforms in listed order:
   • --trim: strip leading/trailing whitespace from each line
   • --collapse-spaces: condense runs of spaces to single space
   • --strip-blank: drop empty lines
   • --to-unix: convert crlf and cr to lf
   • --to-crlf: convert lf and cr to crlf
   • --dehyphenate: rejoin hyphenated line-breaks (ocr/pdf artifact)
   • --unsmart: convert smart quotes and em-dashes to ascii equivalents
   • --strip-bom: remove byte-order mark
   • --strip-zwsp: remove zero-width spaces and joiners
   • --tabs-to-spaces N: replace tabs with N spaces
   • --spaces-to-tabs N: replace N spaces with tabs
   • --lower / --upper / --title: case conversion
   • --normalize-unicode NFC|NFD|NFKC|NFKD: unicode normalization form
3. write normalized text to output file
4. check exit code (0 = success, 1 = empty input, 2 = bad args)

outputs: normalized text file

3. redact pii patterns (redact.py)

inputs: text file, pii kinds (email, phone, ipv4, ipv6, url, credit-card with luhn validation, ssn-us, uuid, hex-token 32+ hex chars, aws-access-key AKIA*, jwt with eyJ header), optional flags (--keep-counts, --preserve-length, --token-template)

1. run python3 scripts/redact.py <input-file> <output-file> [--kinds email,phone,...]
2. default kinds = all supported patterns
3. scan file for each pii pattern using regex + validation (credit-card validated against luhn checksum)
4. replace each match with placeholder: default [REDACTED_{kind}_{i}] or custom --token-template
5. if --keep-counts: identical pii values get identical placeholders (deterministic)
6. if --preserve-length: pad or truncate placeholder to match original length
7. write redacted text to output file
8. check exit code (0 = redactions made, 1 = zero redactions, 2 = bad args)

outputs: redacted text file with pii replaced

4. line-level utilities (lines.py)

inputs: text file, operation (count, dedupe, sort, shuffle, head, tail), optional flags (--case-insensitive, --keep first|last, --numeric, --reverse, --seed for shuffle, -n for head/tail count)

1. run python3 scripts/lines.py <input-file> --op <operation> [flags] (stdout) or --output <file> (write to file)
2. for count: return total line count (streams, no buffering)
3. for head -n N: output first N lines (streams)
4. for tail -n N: output last N lines (streams)
5. for dedupe: drop duplicate lines (buffered in memory, case-insensitive if flag set, --keep first|last chooses which copy to keep)
6. for sort: sort lines alphabetically (buffered, --numeric sorts numerically, --reverse reverses order, --case-insensitive for case-insensitive sort)
7. for shuffle: randomize line order (buffered, --seed N for deterministic output)
8. write output to stdout or file
9. check exit code (0 = operation succeeded, 1 = empty input, 2 = bad args)

outputs: processed lines to stdout or file

5. word and character statistics (wordcount.py)

inputs: text file, optional flags (--top N for most frequent words, --stopwords file, --min-length N, --ignore-case, --regex PATTERN for word boundaries, --json for machine-readable output)

1. run python3 scripts/wordcount.py <input-file> [flags]
2. scan file and count words, characters, lines, sentences
3. default regex [A-Za-z']+ extracts words, override with --regex
4. if --stopwords: load file (one word per line) and exclude from top-word results
5. if --top N: output N most frequent words with counts
6. if --json: output stats as json object (word_count, char_count, line_count, sentence_count, top_words array)
7. write to stdout
8. check exit code (0 = success, 1 = empty input, 2 = bad args)

outputs: human-readable stats or json to stdout

6. text diff in three modes (diff_text.py)

inputs: before file, after file, mode (unified default, side, html), optional flags (--ignore-case, --ignore-whitespace, --context N)

1. run python3 scripts/diff_text.py <before-file> <after-file> --mode <mode> [flags]
2. if --mode unified (default): output unified diff (compatible with patch tools)
3. if --mode side: output two-column side-by-side diff with context
4. if --mode html: output full html file with red/green coloring (write to --output file.html)
5. if --ignore-case: perform case-insensitive comparison
6. if --ignore-whitespace: treat all whitespace as equal
7. if --context N: include N lines of unchanged context around each change
8. write output to stdout or html file
9. check exit code (0 = files identical or diff generated, 1 = files differ, 2 = bad args)

outputs: diff to stdout or html file

7. template substitution (template.py)

inputs: template file with placeholders, json object or inline --set key=value overrides

1. run python3 scripts/template.py <template-file> [--set key=value...] [--json-file path]
2. parse template file for three placeholder syntaxes:
   • mustache: {{name}}
   • dollar: ${name}
   • percent: %(name)s
3. apply filters (pipe syntax): {{name|upper}}, {{url|urlencode}}, etc. supported filters: upper, lower, title, strip, capitalize, reverse, len, escape-html, escape-json, urlencode
4. support default values: {{name ?Unknown}}
5. substitute placeholders with values from json object or --set args
6. if --strict: exit 1 if any placeholder remains unresolved after substitution
7. write rendered output to stdout
8. check exit code (0 = all placeholders resolved or non-strict, 1 = unresolved placeholders in strict mode, 2 = bad args)

outputs: rendered text to stdout

no jinja2, no eval, hand-rolled regex tokenizer only

8. url-safe slug generation (slug.py)

inputs: text string or file, optional flags (--separator char, --max-length N, --no-lower, --ascii for unicode transliteration, --keep-dots for filenames, --dedupe)

1. run python3 scripts/slug.py --text "string" (single string mode, stdout) or python3 scripts/slug.py <input-file> --output <output-file> (batch mode)
2. remove punctuation and convert to lowercase (unless --no-lower)
3. if --ascii: transliterate unicode to ascii via nfkd decomposition
4. replace runs of non-alphanumeric with separator (default -)
5. if --keep-dots: preserve dots in output (useful for filenames)
6. if --dedupe: drop duplicate separators
7. truncate to --max-length if set
8. output slug to stdout (single mode) or write one slug per line (batch mode)
9. check exit code (0 = success, 1 = input slugifies to empty string, 2 = bad args)

outputs: slug(s) to stdout or file

9. markdown processing (markdown.py)

inputs: markdown file, mode (text, html, extract), optional flags (--link-style anchor|url|both for text mode)

1. run python3 scripts/markdown.py <input-file> --mode <mode> [flags]
2. if --mode text: strip markdown to plain text (--link-style controls [text](url) rendering: anchor = text, url = url, both = text (url))
3. if --mode html: render minimal html approximation (headings, paragraphs, lists, blockquotes, fenced code, links, images, bold/italic/code)
4. if --mode extract: output structured items (headings, links, images, code blocks, list items) as json (--output file.json), jsonl (--output file.jsonl), or tsv (--output file.tsv)
5. write output to stdout or file
6. check exit code (0 = success, 1 = no extractable items in extract mode, 2 = bad args)

outputs: plain text, html file, or json/jsonl/tsv to stdout or file

10. find and replace (replace.py)

inputs: text file, find pattern(s), replace value(s), optional flags (--dry-run, --max N, --word for word boundaries in literal mode, --rules json file)

1. run python3 scripts/replace.py <input-file> <output-file> --find pattern --replace value [--find ... --replace ...]
2. alternative: --rules path.json with array of {find, replace, mode, ...} objects
3. for each find/replace pair: apply in order within same pass
4. mode options: regex (default, supports capture groups \1, \2), literal (exact string match), word (literal with word boundaries)
5. if --dry-run: preview matches with line:col and context, don't write
6. if --max N: cap replacements per rule
7. write output file
8. check exit code (0 = replacements made, 1 = zero replacements, 2 = bad args)

outputs: text file with replacements applied

decision points

api key / remote auth not needed: all operations are local and stateless. no decision branching required.

output format choice: extract.py, lines.py, wordcount.py, markdown.py support multiple output formats (txt, json, jsonl, tsv, html). caller specifies via --output file.ext or defaults to stdout.

if input file is empty: most scripts exit 1 (no matches, no lines, no words). normalize.py and template.py exit 0 even on empty input (they produce empty output, which is valid).

if no matches for extraction or redaction: exit code 1, so scripts chain cleanly in shell conditionals. extract.py --kind email app.log && echo "found emails" will not echo if zero emails exist.

if redaction produces zero changes: exit 1, signaling caller that no pii was found. useful for ci pipelines that require proof redaction happened.

if replace.py produces zero replacements: exit 1. use in ci to fail if a pattern doesn't match expected input.

if pii validation fails (e.g., luhn checksum for credit cards): pattern is skipped, not redacted. false negatives are preferred over false positives.

if placeholder resolves to empty string (e.g., template.py with missing required key in non-strict mode): output empty string. in --strict mode, exit 1 instead.

if file encoding detection fails: attempt utf-8, utf-8-sig, cp1252, latin-1 in order. if all fail, script exits 2 (bad args / unreadable file).

if output path is unsafe (contains shell metacharacters, directory traversal, etc.): exit 2 and refuse to write. validated by strict allowlist regex that rejects ;, |, &, >, <, $, backtick, and relative paths outside cwd.

if stopwords file does not exist (wordcount.py): exit 2 and report missing file.

if rules json is malformed (replace.py): exit 2 and report parse error.

output contract

• extract.py: one match per line (default txt mode), or json array, or jsonl, or one per line with line number prefix. exit 0 if matches found, 1 if zero, 2 if bad args.
• normalize.py: text file in utf-8. exit 0 if success, 1 if empty input, 2 if bad args.
• redact.py: text file in utf-8 with pii replaced by placeholders. exit 0 if redactions made, 1 if zero, 2 if bad args.
• lines.py: processed lines in txt format to stdout or file. exit 0 if success, 1 if empty input, 2 if bad args. dedupe and sort return lines in stable order; shuffle is randomized (unless --seed provided); head and tail stream without buffering.
• wordcount.py: human-readable stats to stdout, or json object if --json. columns: word count, char count, line count, sentence count, optionally top N words with frequencies. exit 0 if success, 1 if empty input, 2 if bad args.
• diff_text.py: unified diff (default) or side-by-side (--mode side) or html file (--mode html). exit 0 if files identical or diff generated, 1 if files differ, 2 if bad args.
• template.py: rendered text to stdout. exit 0 if all placeholders resolved (or non-strict mode), 1 if unresolved in strict mode, 2 if bad args.
• slug.py: slug(s) to stdout (single mode) or file (batch mode). exit 0 if success, 1 if slug is empty string, 2 if bad args.
• markdown.py: plain text, html file, or json/jsonl/tsv to stdout or file depending on mode. exit 0 if success, 1 if no items extracted, 2 if bad args.
• replace.py: text file with replacements applied in single pass. exit 0 if replacements made, 1 if zero, 2 if bad args. --dry-run outputs preview only.

all outputs are utf-8. all file paths are validated against safe-path policy (no shell metacharacters, no directory traversal).

outcome signal

• extract.py: see matches printed to console or written to file. if zero matches, exit code 1 signals no data found (not an error, just empty result set).
• normalize.py: inspect output file for expected formatting (unix line endings, removed bom, collapsed spaces, etc.). compare before/after with diff or diff_text.py if unsure.
• redact.py: scan output file for placeholder tokens like [REDACTED_email_1] in place of email addresses. if no redactions occur, exit code 1 warns that no pii matched (verify pattern assumptions).
• lines.py: visually inspect output (count should match wc -l, dedupe should show fewer lines, sort should be alphabetical, shuffle should be randomized). exit 0 confirms operation succeeded.
• wordcount.py: review stats on console (total word/char/line/sentence counts) and top words. compare --json output with other tools (grep word frequency) to spot-check accuracy.
• diff_text.py: unified diff shows + (added), - (removed), context lines. side-by-side shows two columns with pipes marking diffs. html file opens in browser with red/green coloring. exit 0 = files identical, exit 1 = files differ.
• template.py: rendered output on console shows all placeholders substituted with values. check for unresolved {{name}} or ${name} patterns in output (indicate missing keys). in --strict mode, missing keys cause exit 1 (fail fast).
• slug.py: output is lowercase, hyphenated, url-safe string. test with curl or paste into browser address bar if in doubt.
• markdown.py: in text mode, inspect output for removed markdown syntax (no #, *, [link](url)). in extract mode, json output lists headings, links, images as objects. compare with markdown source visually.
• replace.py: in --dry-run mode, preview shows line:col of each match and context. in normal mode, diff the output against input to verify replacements. exit 0 signals changes were made; exit 1 signals no matches (pattern mismatch or wrong file).

edge cases and known limits

• rate limits / timeouts: not applicable (no remote calls). local i/o is synchronous and single-threaded.
• auth expiry: not applicable (no external services).
• empty input: most scripts return exit 1 (zero results is not a failure, just empty). use exit code in conditionals for ci pipelines.
• very large files: extract, lines (head/tail), wordcount streaming operations do not load full file into memory. sort, dedupe, shuffle, normalize buffer in memory but handle multi-gigabyte files on modern laptops (memory grows with number of unique lines, not file size). html diff loads both files (worst case 2x file size).
• pii pattern false positives: credit-card uses luhn validation to suppress random 16-digit sequences. hex-token and similar patterns intentionally over-match; review redaction counts before sharing output. phone pattern avoids collision with ips, dates, credit-card-with-spaces via explicit format matching (intl +, parenthesized, or 3-3-4 dashed).
• phone number false negatives: pattern supports +<digits>, (XXX) XXX-XXXX, XXX-XXX-XXXX, XXX XXX XXXX but misses exotic local formats. tune with custom regex in replace.py if needed.
• unicode normalization: four forms (nfc, nfd, nfkc, nfkd) produce different outputs. nfc is standard for most text; nfkc for compatibility. test before committing to pipeline.
• markdown extraction incomplete: patterns extract common elements (headings, links, code blocks, lists) but not every markdown extension (tables, footnotes, etc.). extract mode outputs json; inspect output for covered cases.
• template placeholder ambiguity: if text contains literal {{ without intent to substitute, escape as {{ or rewrite template to avoid collision. strict mode (--strict flag) forces all placeholders to resolve.
• path safety: rejects ;, |, &, >, <, $, backtick, and relative paths (e.g., ../../../etc/passwd). all output paths must be within current working directory or absolute paths.

quick reference

# extract all emails, dedupe, sort
python3 scripts/extract.py app.log --kind email --unique --sort

# clean up ocr garbage
python3 scripts/normalize.py scanned.txt clean.txt --strip-bom --to-unix --dehyphenate --collapse-spaces

# redact everything, keep counts deterministic
python3 scripts/redact.py transcript.txt safe.txt --keep-counts

# drop duplicate lines, case-insensitive
python3 scripts/lines.py users.txt --op dedupe --case-insensitive --output unique.txt

# top 20 words, ignore case, load stopwords
python3 scripts/wordcount.py essay.txt --top 20 --ignore-case --stopwords stop.txt

# html diff for sharing
python3 scripts/diff_text.py before.txt after.txt --mode html --output diff.html

# template with mustache syntax
python3 scripts/template.py msg.txt --set name=Alice --set role=Engineer

# generate url-safe slug from filename
python3 scripts/slug.py --text "Hello World (2024)" --keep-dots

# strip markdown to plain text
python3 scripts/markdown.py readme.md --mode text

# find and replace with regex capture groups
python3 scripts/replace.py log.txt clean.log --find '(\d{4})-(\d{2})-(\d{2})' --replace '\3/\2/\1'

exit codes

code	meaning
0	success, matches found, operation completed, files identical
1	zero results, empty input, files differ, unresolved placeholders (strict mode)
2	bad arguments, unsafe path, missing input, unknown kind, bad regex, parse error, unsupported output extension

chaining works: normalize.py && redact.py && wordcount.py halts on first failure (exit 2) or continues on empty result (exit 1) depending on your shell set -e policy.

safety properties

• pure python 3 standard library. zero third-party dependencies. no pip, no npm, no external binaries except python3 itself.
• no subprocess calls, no shell invocation, no eval, no jinja2.
• all file paths validated by strict allowlist regex. rejects shell metacharacters (;, |, &, >, <, $, backtick, ', "), directory traversal (..), and relative paths outside cwd.
• reads only input paths caller provides. writes only to output paths caller provides.
• utf-8 default with fallback to utf-8-sig, cp1252, latin-1 on read. all writes are utf-8.
• deterministic output where it matters: shuffle --seed N is reproducible, extract and wordcount emit results in same order per input, sort is stable.

performance benchmarks

• lines.py --op dedupe on 100,000 short lines (500 distinct): ~60ms.
• lines.py --op sort on 100,000 lines: ~100ms.
• extract.py scans file in single streaming pass; memory constant regardless of file size (streams one line at a time).
• wordcount.py streams line-by-line for char/line/sentence counts; buffers only unique words for frequency ranking.
• redact.py single-pass scan with regex matching; memory grows with number of unique pii values (for --keep-counts dedup table).

pairs well with

• clean-csv-toolkit: same author, same stdlib-only design, handles tabular data.
• openclaw-prompt-shield: chain extract.py --kind email,url output into prompt-shield to scrub user input before llm calls.

license

MIT

original author: gopendrasharma89-tech