clean-log-toolkit

v0.2.0

intent

clean-log-toolkit reads a local log file, auto-detects the format (apache-common, apache-combined, nginx-access, syslog, json-line, or custom regex), and outputs structured data (CSV/TSV/JSONL) or error summaries. use it when you need to parse unknown log formats, aggregate errors by level and time bucket, grep with log-aware filters (timestamp + level + pattern), or tail a live log file with rotation detection. built on Python 3 standard library only, no external dependencies or remote calls.

inputs

log file path (required)

• local file, readable by the calling user. assumed UTF-8; falls back through utf-8-sig, cp1252, latin-1 if needed.
• edge case: empty files produce exit code 1 (zero rows / zero matches).
• edge case: very large files (>1 GB) are read line-by-line to avoid memory spike.

output file path (required for parse.py, errors.py; optional for grep.py and follow.py)

• must be a valid filesystem path without shell metacharacters. validated against strict allowlist regex.
• extension determines output format: .csv, .tsv, .jsonl, .md, .json. if not provided, output goes to stdout.

--format (optional, parse.py only)

• one of: apache-common, apache-combined, nginx-access, syslog, json-line. if omitted, auto-detected by sniffing first 50 lines. if auto-detection fails, falls back to generic timestamp + level + message extraction.

--regex PATTERN (optional, parse.py only)

• custom regex with named groups (e.g., (?P<ts>\S+), (?P<level>\S+), (?P<message>.*)). takes precedence over --format. if provided, auto-detection is skipped.

--fields (optional, parse.py only)

• comma-separated list of named group names to include in output (e.g., ts,level,msg). if omitted, all groups are included.

--level (optional, errors.py, grep.py, follow.py)

• comma-separated list of log levels to include (case-insensitive). canonical names: TRACE, DEBUG, INFO, NOTICE, WARN, ERROR, FATAL. default: WARN,ERROR,FATAL. if a line has no detected level, it is skipped.

--bucket (optional, errors.py only)

• one of: minute, hour, day. groups the error timeline into buckets. default: hour.

--top (optional, errors.py only)

• integer, number of top-N error fingerprints to report. default: 10.

--pattern REGEX (optional, grep.py, follow.py)

• only include lines matching this regex (case-sensitive). applied after level filter.

--not-pattern REGEX (optional, grep.py, follow.py)

• exclude lines matching this regex (case-sensitive). applied after --pattern.

--since TIMESTAMP (optional, grep.py, follow.py)

• include only lines on or after this timestamp. accepts iso 8601 (2026-05-10T10:00:00Z), space-separated (2026-05-10 10:00:00), apache-style (10/May/2026:10:00:00 +0000), syslog-style (May 10 10:00:00), or date-only (2026-05-09, 2026/05/09, 09/05/2026). if --since is date-only, it starts at 00:00:00 of that day.

--until TIMESTAMP (optional, grep.py, follow.py)

• include only lines before or on this timestamp. same formats as --since. if --until is date-only, it includes through 23:59:59 of that day.

-B / -A / -C (optional, grep.py only)

• context lines. -B N prints N lines before each match, -A N prints N lines after, -C N prints N before + N after. default: 0 (no context).

--with-line (optional, grep.py only)

• prepend line number to each output line.

--invert (optional, grep.py only)

• invert the match: include lines that do NOT match --pattern. if --not-pattern is also provided, that still applies.

--max-events N (optional, follow.py only)

• exit cleanly after N matched events. useful for CI pipelines.

--timeout SECONDS (optional, follow.py only)

• exit on inactivity. if no new lines arrive for SECONDS, the process exits with code 0. useful for log rotation or end-of-stream detection.

--json (optional, follow.py only)

• emit one JSON envelope per matched line, with extracted timestamp and level. default: plain text.

--output PATH (optional, errors.py only)

• write report to file instead of stdout. format inferred from extension: .md, .json, .csv.

python3 binary

• must be available in PATH. version 3.7+. verify with python3 --version.

procedure

parse.py: parse a log file into structured rows

1. input: log file path, optionally --format, --regex, --fields, output path
2. validate: check input path is readable, output path is safe (no shell metacharacters).
3. sniff format: if --regex is provided, skip sniffing and use custom pattern. otherwise, if --format is provided, use it directly. otherwise, read first 50 lines and try to match apache-common, apache-combined, nginx-access, syslog, json-line patterns in that order.
4. extract rows: iterate through all lines in the log file. for each line, apply the detected or custom regex and extract named groups into a dict. skip lines that do not match the pattern.
5. filter fields: if --fields is provided, keep only those columns. otherwise, include all matched named groups.
6. serialize: write all rows to output file (or stdout if no output path) in the format determined by output extension (.csv, .tsv, .jsonl, or json-line if extension is .jsonl).
7. output: success returns exit code 0 if >=1 row extracted. zero rows extracted returns exit code 1. invalid args / bad regex / unsafe path returns exit code 2.
8. output location: output file (e.g., structured.csv) or stdout.

errors.py: aggregate errors by level, time bucket, and fingerprint

1. input: log file path, optionally --level, --bucket, --top, --output
2. validate: check input path is readable, output path is safe (if provided).
3. parse all lines: iterate through the log file. for each line, extract timestamp and level using the same heuristic as grep.py (does not need auto-format detection; generic extraction is sufficient).
4. filter by level: if --level is provided, keep only lines with those levels. default: WARN,ERROR,FATAL.
5. bucketize: group each line by its timestamp bucketed into the requested granularity (minute, hour, or day). within each bucket, track count of lines.
6. fingerprint normalization: for each message, replace numbers (including floats, negatives), uuids (8-4-4-4-12 hex), hex tokens (runs of hex digits), file:line pairs (e.g., foo.py:123), and embedded timestamps with placeholders. two messages that differ only in these values are collapsed into one fingerprint group.
7. rank: sort all fingerprints by frequency (descending). keep top --top (default 10).
8. format report: if --output is provided, write in the format of the extension (.md for markdown, .json for structured json, .csv for timeline). if no --output, print a one-screen summary to stdout.
9. output: success returns exit code 0 if >=1 matching log entry found. zero matches returns exit code 1. invalid args / unsafe path returns exit code 2.
10. output location: output file (e.g., report.md) or stdout.

grep.py: grep with log-aware filters

1. input: log file path, optionally --pattern, --not-pattern, --level, --since, --until, -B/-A/-C, --with-line, --invert, output path
2. validate: check input path is readable. validate regex patterns compile. validate --since and --until parse as valid timestamps.
3. parse timestamps: extract timestamp from each line using the same generic heuristic as errors.py. if timestamp cannot be parsed, skip the line (no match).
4. filter by level: if --level is provided, keep only lines with those levels. if --level is omitted, no level filtering.
5. filter by time window: if --since is provided, skip lines with timestamp < --since. if --until is provided, skip lines with timestamp > --until.
6. filter by pattern: if --pattern is provided, keep only lines where the full line matches the pattern (case-sensitive). if --invert is set, invert this filter (keep lines that do NOT match).
7. filter by not-pattern: if --not-pattern is provided, skip lines matching this pattern (case-sensitive).
8. add context: if -B/-A/-C is provided, for each matching line, include N surrounding lines before / after (or both). context lines are printed even if they do not match --pattern.
9. format output: if --with-line is set, prepend line number to each line. output to stdout or file (if output path provided).
10. output: success returns exit code 0 if >=1 match found. zero matches returns exit code 1. invalid args / bad regex / bad timestamp / unsafe path returns exit code 2.
11. output location: stdout or output file.

follow.py: tail -F equivalent with log-aware filtering

1. input: log file path, optionally --pattern, --not-pattern, --level, --since, --max-events, --timeout, --json
2. validate: check input path is readable. validate regex patterns compile.
3. open file: open the log file for reading. record the inode and file size.
4. tail from end (or --since): if --since is not provided, seek to end of file and start reading new lines. if --since is provided, scan from beginning and skip lines with timestamp < --since, then resume tailing.
5. loop: block on new data (read loop). for each new line, apply filters (--level, --pattern, --not-pattern) as in grep.py.
6. emit matched lines: for each match, either print plain text or emit a json envelope (if --json is set) with fields: timestamp, level, message, line_number.
7. detect rotation: after each read, check if the inode has changed or file size has decreased (truncation). if so, close the file, reopen it, and resume tailing.
8. exit conditions: exit cleanly (code 0) when --max-events is reached, or --timeout seconds elapse without new data, or EOF is reached (unlikely in tail mode).
9. output: all matches printed to stdout or file.

check_deps.sh: verify python3 is available

1. input: none
2. check: run which python3 or python3 --version. if successful, exit 0. if not found, print diagnostic and exit 2.

decision points

Auto-format detection vs. explicit --format vs. custom --regex

• if --regex is provided, skip sniffing and use the custom pattern (parse.py).
• else if --format is provided, use it directly without sniffing.
• else sniff the first 50 lines and try apache-common, apache-combined, nginx-access, syslog, json-line in order. use the first match.
• if sniffing finds no match, fall back to generic timestamp + level + message extraction.

Empty or malformed input

• if the input file is empty or contains zero lines, return exit code 1 (no rows / no matches).
• if a line is malformed and does not match the detected format, skip it (do not halt).

Level detection

• if a line has no detected level keyword (TRACE, DEBUG, INFO, NOTICE, WARN, WARNING, ERROR, ERR, FATAL, CRITICAL, CRIT, EMERG, EMERGENCY), treat it as level-less and skip it in grep.py, errors.py, and follow.py unless --level is omitted.
• if --level is omitted, include all lines regardless of level.

Timestamp parsing in grep.py and follow.py

• if a line cannot be parsed for a timestamp, skip it (no match in grep.py; do not tail it in follow.py).
• if --since or --until is a date-only value (e.g., 2026-05-09), expand it to a time range: --since becomes 2026-05-09T00:00:00, --until becomes 2026-05-09T23:59:59.

Fingerprint collisions in errors.py

• two semantically different errors that differ only in numbers / uuids / hashes will be collapsed into the same fingerprint. if this is a problem, increase --top to inspect more groups or manually grep for specific patterns.

Log rotation in follow.py

• if inode changes or file size decreases (truncation), the file has rotated. close the current handle and reopen the file by path.
• if --max-events or --timeout is set, exit cleanly before attempting to reopen.

Character encoding

• all files are assumed UTF-8. if a read fails, fall back through utf-8-sig, cp1252, latin-1 in order. if all three fail, skip the line.
• all output is written as UTF-8.

Context lines in grep.py

• context lines are printed in order (before matches, then the match, then after matches). if two matches overlap in their context windows, do not duplicate lines.
• context lines that precede the first match or follow the last match in a file are not printed (only context between and within matches).

Path safety

• all input and output paths are validated against a strict allowlist regex that rejects shell metacharacters (backticks, $, |, &, ;, <, >, etc.). unsafe paths cause exit code 2.

output contract

parse.py output

• CSV: comma-separated, RFC 4180 compliant. header row with named group names.
• TSV: tab-separated. header row with named group names.
• JSONL: one JSON object per line, keys are named group names, values are strings.
• on zero rows: exit code 1 (no output file written).
• on success: exit code 0, output file created with >=1 row.

errors.py output

• markdown report (.md): human-readable summary with timestamp, section for each top-N fingerprint, count, and example lines.
• json report (.json): structured object with keys: total_entries, total_buckets, bucket_unit (minute/hour/day), top_errors (array of {fingerprint, count, examples, bucket_timeline}).
• csv report (.csv): timeline only. columns: bucket, count, fingerprint.
• stdout summary (no --output): one-screen text dump with top-N fingerprints and counts.
• on zero matches: exit code 1 (no output file written).
• on success: exit code 0, output file created or summary printed.

grep.py output

• plain text, one matched line per output line.
• if --with-line is set, prepend line number (1-indexed) before each line.
• if -B/-A/-C is set, context lines are printed in order without modification.
• on zero matches: exit code 1 (no output).
• on success: exit code 0, one or more lines printed to stdout or file.

follow.py output

• plain text: one matched line per output line as it arrives.
• if --json is set: one json envelope per matched line, with keys: timestamp, level, message, line_number.
• on --timeout or --max-events reached: exit code 0.
• on error (invalid regex, bad path): exit code 2.

error cases

• invalid arguments (unrecognized flag, bad regex, unparseable timestamp): exit code 2, print diagnostic to stderr.
• unsafe file path (shell metacharacters detected): exit code 2, print diagnostic to stderr.
• input file not found or not readable: exit code 2, print diagnostic to stderr.
• zero rows / zero matches found: exit code 1.
• success: exit code 0.

outcome signal

• parse.py: the output file (CSV/TSV/JSONL) is created and contains one header row plus N data rows. you can open it in a spreadsheet or pipe it to another tool (clean-csv-toolkit, etc.).
• errors.py: the report file (Markdown/JSON/CSV) is created and contains >=1 error group summary. you can paste the Markdown into a ticket, or parse the JSON for downstream tooling, or load the CSV into a spreadsheet.
• grep.py: matched lines are printed to stdout or file. each line includes the original log text, context lines if requested, and line number if --with-line is set.
• follow.py: new matching lines appear in stdout or file in real-time as the log is written. if --json is set, you see structured envelopes with parsed timestamp and level. when --max-events is reached or --timeout fires, the process exits cleanly (exit code 0).
• check_deps.sh: if successful, python3 path and version are printed. if failed, a diagnostic is printed and exit code is