clean-log-toolkit

v0.1.1

local toolkit for reading log files, detecting format, finding actual problems, and producing summaries you can paste into tickets. built on python 3 standard library only. no awk/sed/jq wrappers, no pip installs, no remote calls.

intent

this skill handles the repetitive work of log inspection: auto-detect format (apache-common, apache-combined, nginx-access, syslog, json-line, or custom regex), parse lines into structured rows (csv/tsv/jsonl), aggregate errors by level and time bucket, normalize repeated errors via fingerprint hashing to surface root causes, and grep logs with time-window and level filters. use this when you need to turn an unstructured log dump into actionable summary data for a ticket or incident response without leaving the command line.

inputs

• input_logfile (required): path to a local log file. must be readable and under 1 GB (no hard limit enforced, but memory use scales linearly with file size). encoding auto-detected: tries utf-8-sig, then cp1252, then latin-1.
• --format (optional, parse.py only): one of apache-common, apache-combined, nginx-access, syslog, json-line. if omitted, auto-detects by sniffing first 50 lines. falls back to generic timestamp + level + message extraction if no match.
• --regex PATTERN (optional, parse.py only): python regex with named groups ((?P<name>...)) to define custom format. named groups must include at least one of: ts (timestamp), level (log level), message (message text). overrides --format.
• --fields FIELD1,FIELD2,... (optional, parse.py only): comma-separated list of output columns. only applies with --format json-line. defaults to all fields found.
• --bucket {minute,hour,day} (optional, errors.py only): time granularity for error timeline aggregation. defaults to hour.
• --level LEVEL1,LEVEL2,... (optional, errors.py and grep.py): comma-separated log levels to include. case-insensitive. recognized levels: trace, debug, info, notice, warn, warning, error, err, fatal, critical, crit, emerg, emergency. defaults to warn,error,fatal for errors.py; no default for grep.py (matches all levels).
• --top N (optional, errors.py only): return top N most frequent error fingerprints. defaults to 10.
• --output FILE (optional, errors.py only): write report to file. extension determines format: .json, .md (markdown), .csv. if omitted, prints one-screen summary to stdout.
• --pattern REGEX (optional, grep.py only): include lines matching this python regex. combined with --not-pattern in same pass (both filters apply).
• --not-pattern REGEX (optional, grep.py only): exclude lines matching this regex.
• --since TIMESTAMP (optional, grep.py only): include lines on or after this timestamp. accepts iso 8601 (with or without tz/microseconds), space-separated date-time, slash-separated date-time, apache-style (10/May/2026:10:00:00 +0000), syslog-style (May 10 10:00:00, year assumed current), or date-only (2026-05-10, 2026/05/10, 10/05/2026).
• --until TIMESTAMP (optional, grep.py only): include lines before or at this timestamp. same format as --since.
• -B N (optional, grep.py only): print N lines before each match.
• -A N (optional, grep.py only): print N lines after each match.
• -C N (optional, grep.py only): shorthand for -B N -A N.
• --invert (optional, grep.py only): invert match logic; keep lines that do not match --pattern.
• --with-line (optional, grep.py only): prepend line number to each output line.

external connections: none. all operations are local file i/o.

procedure

parse.py: auto-detect and structure logs

1. validate input path against allowlist regex (rejects shell metacharacters, symlink traversal). fail with exit code 2 if invalid.
2. validate output path similarly. fail with exit code 2 if unsafe or parent directory does not exist.
3. determine output format from extension: .csv, .tsv, .jsonl. fail with exit code 2 if unsupported.
4. if --regex is provided, compile it as a python regex. fail with exit code 2 if invalid regex or no named groups.
5. if --regex is not provided and --format is given, validate against supported formats. fail with exit code 2 if unknown.
6. if neither --regex nor --format is given, open input file and sniff first 50 lines to auto-detect format (test against apache-common, apache-combined, nginx-access, syslog, json-line patterns in that order). if no match, use fallback generic parser.
7. open input file with utf-8 encoding. on decode error, retry with utf-8-sig, then cp1252, then latin-1. log encoding choice to stderr.
8. iterate over lines:
   • parse timestamp (extract_timestamp helper tries iso 8601 variants, apache, syslog formats in order).
   • parse log level (extract_level helper recognizes warn/warning, error/err, fatal/critical/crit/emerg/emergency, case-insensitive, folds to canonical names).
   • extract remaining fields per format (e.g., http_status, remote_addr for apache-combined).
   • if --fields is set (json-line only), filter to named columns.
   • if any field is missing, use empty string placeholder.
   • accumulate row as dict.
9. write rows to output file in requested format (csv, tsv, or jsonl). csv/tsv use , or \t as delimiter, include header row.
10. close file and report row count to stdout. exit 0 if one or more rows written, exit 1 if zero rows, exit 2 on error.

errors.py: aggregate and fingerprint errors

1. parse input log file (reuse parse.py logic or read as raw lines if already structured).
2. filter to lines with timestamp and level that matches --level (default warn,error,fatal).
3. bucket each log line by timestamp according to --bucket (minute/hour/day). truncate timestamp to bucket granularity.
4. normalize message via fingerprint: replace integers, UUIDs (8-4-4-4-12 hex pattern), hex tokens (0x[0-9a-f]+), file:line pairs (path/to/file.ext:1234), and embedded iso 8601 timestamps with placeholders (<N>, <UUID>, <HEX>, <FILE:LINE>, <TS>).
5. count occurrences of each (level, bucket, fingerprint) triple.
6. rank fingerprints by total count (across all time buckets for that level). select top N (default 10).
7. for each top fingerprint, collect one sample original message and all time buckets where it occurred.
8. if --output is omitted, print one-screen summary (level, top 3 fingerprints, min/max timestamp). exit 0 if errors found, exit 1 if none.
9. if --output FILE is set:
   • if .json: write dict with structure {"level": [...], "bucket": "...", "fingerprints": [{"fingerprint": "...", "count": N, "sample": "...", "buckets": {timestamp: count, ...}}, ...], "generated_at": "..."}.
   • if .md: write markdown table with columns fingerprint, count, sample, buckets. include header "Error Report" and generation timestamp.
   • if .csv: write csv with columns level, bucket, fingerprint, count, sample. include header.
10. close file and report summary to stdout. exit 0 if errors found, exit 1 if none, exit 2 on invalid args.

grep.py: log-aware filtering

1. validate input path. fail with exit code 2 if unsafe.
2. if --output FILE is set, validate output path similarly.
3. compile --pattern regex if provided. fail with exit code 2 if invalid.
4. compile --not-pattern regex if provided. fail with exit code 2 if invalid.
5. parse --since timestamp using same logic as parse.py. fail with exit code 2 if unparseable.
6. parse --until timestamp similarly. fail with exit code 2 if unparseable.
7. open input file with utf-8 encoding. on decode error, retry with utf-8-sig, then cp1252, then latin-1.
8. buffer input lines (store in memory with indices for -B/-A context). iterate:
   • extract timestamp (skip lines with no recognizable timestamp if --since or --until is set).
   • extract level (skip if --level is set and level not in set).
   • test against --pattern (skip if not provided or doesn't match). test against --not-pattern (skip if provided and matches).
   • test against time window: skip if --since is set and timestamp < since, or --until is set and timestamp > until.
   • if all filters pass, mark line as match.
9. for each match, collect context: B lines before (or fewer if at start), A lines after (or fewer if at end). output match with context to stdout or --output FILE.
10. if --with-line is set, prepend 1-indexed line number to each output line.
11. report match count to stderr. exit 0 if one or more matches, exit 1 if zero matches, exit 2 on error.

decision points

• format auto-detection (parse.py): if --regex is provided, use custom regex and skip format detection. else if --format is provided, use named format (validate it exists). else sniff first 50 lines against apache-common, apache-combined, nginx-access, syslog, json-line in that order; use first match. if no match, fall back to generic timestamp + level + message extractor.
• encoding fallback (all scripts): if utf-8 decode fails on first line, retry with utf-8-sig (skips BOM), then cp1252 (windows-1252), then latin-1 (iso 8859-1). if all fail, exit 2. once first successful decode is found, stick with that encoding for entire file.
• timestamp parsing (parse.py, grep.py): try iso 8601 variants (with/without tz, with/without microseconds, zulu), space-separated, slash-separated, apache-style, syslog-style in order. use first match. if none match and timestamp is required (e.g., for grep with --since), treat line as missing timestamp and skip (or for parse.py, use empty string).
• level filtering (errors.py, grep.py): if --level is not provided in errors.py, default to warn,error,fatal. if --level is not provided in grep.py, no filtering (match all levels). comparison is case-insensitive and folds aliases (warn/warning, error/err, fatal/critical/crit/emerg/emergency).
• fingerprint collision (errors.py): if two semantically different errors differ only in numbers, uuids, or hashes, they are collapsed into one fingerprint with a combined count. this is a best-effort heuristic. if this matters for your analysis, use --top with a larger N and inspect samples manually, or re-run with --regex to extract a more precise discriminator.
• empty results (all scripts): parse.py exits 1 if zero rows extracted. errors.py exits 1 if zero matching log entries found. grep.py exits 1 if zero matches found. this allows chaining in bash: parse.py ... && errors.py ... stops on zero rows.
• time zone handling (grep.py): if --since or --until lacks timezone and matches a format that should have one (iso 8601 with tz, apache-style with tz), assume utc. for syslog-style (month day hh:mm:ss, no year or tz), assume current year and local timezone. if ambiguous (e.g., 2026-05-10 with no tz), assume utc.
• rate limits / performance: no remote api calls, so no rate limit handling needed. local file i/o only. if file exceeds available memory, script may fail with out-of-memory error (no graceful degradation). grep.py buffers entire input in memory for context lines; large files (>1 GB) may be slow.
• auth / security: no authentication required. file path validation uses strict allowlist regex (^[a-z0-9_./:-]+$ equivalent) to reject shell metacharacters, symlink traversal (../, ..), and suspicious patterns. all file i/o uses standard python open(), no subprocess calls.

output contract

parse.py output

• format: csv (default), tsv, or jsonl. csv/tsv include header row on line 1.
• columns: determined by format detected or --fields specified. minimum: timestamp, level, message. apache-combined adds remote_addr, http_method, request_path, http_status, response_bytes, referer, user_agent. nginx-access similar. json-line retains all fields.
• missing fields: represented as empty string ("" in csv/tsv, null in jsonl).
• encoding: always utf-8, no BOM.
• line ending: unix (lf), not crlf.
• location: path specified by second positional argument or --output FILE.
• on success: file written, line count printed to stdout (e.g., "parsed 1234 lines"), exit 0.
• on zero rows: file still created (with header only for csv/tsv), stdout reports "parsed 0 lines", exit 1.
• on error: file not created or left in partial state; error message to stderr, exit 2.

errors.py output

• format: text summary to stdout (if no --output), or json/markdown/csv file (if --output specified).
• stdout summary (no --output): human-readable one-screen output. example: "ERROR: 523 total, top 3 fingerprints: [1] Connection timeout... (234x) [2] Disk full... (145x) [3] Auth failed... (92x). Time range: 2026-05-10T08:00:00Z to 2026-05-10T15:30:00Z".
• json output (--output report.json): dict with keys levels (list of level strings found), bucket (granularity), fingerprints (list of dicts with fingerprint, count, sample, buckets sub-dict), time_range (dict with min, max), generated_at (iso 8601 timestamp).
• markdown output (--output report.md): formatted as markdown table. example: "# Error Report\n\n|Fingerprint|Count|Sample|Time Buckets|\n|---|---|---|---|\n|Connection timeout...|234|Connection timeout to 10.0.0.5...|{2026-05-10T10:00:00Z: 5, ...}|\n...".
• csv output (--output report.csv): header level,bucket,fingerprint,count,sample. one row per (level, bucket, fingerprint, count) tuple. sample is truncated to ~200 chars to fit.
• encoding: utf-8.
• on success: file written (or stdout printed), exit 0.
• on zero matching entries: stdout reports "no matching log entries found", exit 1.
• on error: stderr message, exit 2.

grep.py output

• format: plain text, one line per match. context lines (from -B/-A/-C) interleaved with matches.
• line numbering: if --with-line, format is "NNNN: ".
• location: stdout (default) or file specified by --output FILE.
• encoding: utf-8.
• on success: all matching lines printed, exit 0.
• on zero matches: no output, exit 1.
• on error: stderr message, exit 2.

outcome signal

• parse.py success: output file exists and is readable, contains at least one row (header + data), byte size > 0. stdout shows "parsed N lines" where N >= 1. exit code is 0.
• parse.py zero-row case: output file exists with header only (csv/tsv) or empty (jsonl). stdout shows "parsed 0 lines". exit code is 1 (not a failure, but signals no data to downstream tools).
• errors.py success: stdout contains human-readable summary with top fingerprints listed. if --output file is set, file exists and contains structured report (json/markdown/csv). exit code is 0.
• errors.py zero-error case: stdout reports "no matching log entries found". no output file written (or empty file if specified). exit code is 1.
• grep.py success: stdout or output file contains at least one matching line. exit code is 0. stderr may show "Found N matches".
• grep.py zero-match case: no output to stdout or file. exit code is 1. stderr may show "No matches found".
• any script error: stderr contains error message (invalid args, bad regex, unsafe path, unsupported format, unreadable file, encoding failure). exit code is 2. output file is not written or partially written.