systematic-debugging

intent

systematic debugging eliminates random fix-and-check cycles that waste time and introduce new bugs. root cause investigation always comes before any fix attempt. this skill enforces a disciplined four-phase process (investigate, analyze, hypothesize, implement) that forces you to understand what broke and why before touching code. use it for any technical issue: test failures, production bugs, unexpected behavior, performance problems, build failures, integration issues. it's especially critical under time pressure, when fixes seem obvious, or when previous attempts failed.

inputs

context needed:

• error messages, stack traces, log output
• reproduction steps or failing test case
• git history and recent changes (commits, dependencies, config)
• system architecture documentation (component boundaries, data flow)
• access to affected systems for testing and instrumentation
• understanding of your own codebase layout and conventions

external connections:

• git (local repo access, recent commit history)
• test framework (pytest, jest, mocha, etc.)
• logging/monitoring system if available (datadog, cloudwatch, etc.)
• version control platform (github, gitlab, etc.) for diff/history
• issue tracker if tracking the bug (jira, linear, github issues)

edge cases to anticipate:

• issue is timing-dependent or intermittent (race conditions, timeouts)
• issue only reproduces in specific environments (staging vs. prod, specific OS)
• issue involves external services that are flaky or unavailable
• multi-component system with complex data flow between layers
• insufficient logging/instrumentation to trace the problem
• previous developer deleted context or didn't document assumptions

procedure

phase 1: root cause investigation

step 1: read error messages completely

• input: error message, stack trace, warning output
• do not skim or skip past any line
• note exact line numbers, file paths, error codes
• output: annotated error message with key details highlighted

step 2: reproduce the issue consistently

• input: issue description, suspected trigger conditions
• execute exact reproduction steps multiple times
• verify it happens every time or identify conditions when it does/doesn't occur
• if not reproducible, gather more data (logs, metrics, user reports) instead of guessing
• output: documented reproduction steps with frequency (always, intermittent, one-time)

step 3: check recent changes

• input: git history, recent commits, dependency updates, config changes
• run git diff and git log to find what changed
• review new dependencies, config file changes, environment differences
• correlate timing of changes with issue first appearance
• output: list of suspect changes with timestamps

step 4: gather diagnostic evidence in multi-component systems

• input: system architecture, component boundaries, data flow diagram
• when system has multiple layers/services (workflow → build → signing, api → service → database), instrument each boundary before analyzing
• for each component boundary, add logging that shows: data entering component, data exiting component, environment/config state, errors at that layer
• run once to collect evidence showing which exact component fails
• output: evidence log showing data flow through each layer with success/failure markers

step 5: trace data flow backward from symptom to source

• input: failing component identified, error output, code call stack
• use root-cause-tracing.md technique if error is deep in call stack
• ask: where does the bad value originate? what called this with bad value? keep tracing up the call chain
• output: traced chain from symptom back to original source with bad data

phase 2: pattern analysis

step 6: find working examples in same codebase

• input: broken code, codebase, similar working implementations
• locate similar code that works correctly
• compare structure, variable names, logic flow
• output: side-by-side comparison of working vs. broken code

step 7: compare against reference implementation

• input: official pattern, documentation, library examples
• read the reference completely, don't skim
• understand every line, all assumptions, all dependencies
• output: annotated reference with notes on what your code missed

step 8: identify all differences between working and broken

• input: working example, broken code, reference implementation
• list every difference, no matter how small
• don't assume differences don't matter
• categorize: logic differences, config differences, environment differences, dependency differences
• output: exhaustive difference list with severity assessment

step 9: understand all dependencies

• input: broken component, codebase documentation
• identify all other components this needs
• map required config, environment variables, external services
• verify assumptions about state and initialization
• output: dependency map with required state

phase 3: hypothesis and testing

step 10: form single explicit hypothesis

• input: root cause investigation evidence, pattern analysis
• state clearly in writing: "i think X is the root cause because Y"
• be specific (not "something is wrong" but "IDENTITY env var is not propagated from workflow to build step")
• output: written hypothesis statement

step 11: test hypothesis with minimal change

• input: hypothesis, affected code, test framework
• make the smallest possible change to test the theory
• change one variable at a time
• run the test that previously failed
• output: test result (pass or fail)

step 12: verify result before proceeding

• input: test result, original failure condition
• if test passes: move to phase 4
• if test fails: document why this hypothesis was wrong, return to step 10 (form new hypothesis)
• do not add more fixes on top of a failed fix
• output: confirmation of hypothesis validity or statement of new hypothesis

phase 4: implementation

step 13: create failing test case

• input: reproduction steps, test framework
• write the simplest possible test that reproduces the issue
• use automated test if possible, one-off test script if no framework available
• must have a failing test before attempting any fix
• reference superpowers:test-driven-development for proper test structure
• output: failing test in version control

step 14: implement single fix addressing root cause

• input: identified root cause, failing test
• implement one change that addresses the root cause
• one change at a time, no bundled fixes
• resist "while i'm here" improvements and refactoring
• output: code change with git commit message explaining why

step 15: verify fix works completely

• input: modified code, test suite
• run the failing test, confirm it passes
• run full test suite, confirm no other tests broke
• verify original issue actually resolved in real system
• output: green test results, confirmed issue resolution

step 16: count failed fix attempts

• input: history of attempted fixes, current result
• if this is fix attempt 1 or 2 and it didn't work: return to phase 1 with new information
• if this is fix attempt 3 or more and still failing: move to step 17
• output: decision to continue debugging or escalate

step 17: question architecture (only after 3+ failed fixes)

• input: failed fixes, system design, component interactions
• evaluate if pattern is fundamentally sound or stuck due to inertia
• identify if fixes are creating new symptoms in different places
• identify if all fixes require massive refactoring to implement
• discuss architectural concerns with your human partner before attempting fix #4
• output: architectural assessment and decision to refactor vs. continue patching

decision points

if issue is not reproducible:

• do not guess at fix
• gather more data: logs, metrics, user reports, system state
• return to phase 1 with better evidence

if issue is in multi-component system:

• add instrumentation at each boundary before analyzing
• trace data flow through each layer to find breaking point
• don't assume "it's probably the API" without evidence

if error is deep in call stack:

• use backward tracing technique (root-cause-tracing.md)
• don't fix the symptom at the bottom of stack
• trace back to find original bad input

if you catch yourself proposing fixes without completing phase 1:

• stop immediately
• common rationalization: "issue seems simple" or "i'm in a hurry"
• return to root cause investigation
• systematic debugging is faster than thrashing

if you've already tried 2 fixes and neither worked:

• don't attempt fix #3 without returning to phase 1
• the issue may have changed or you may have misunderstood root cause
• new information from failed attempts must be re-analyzed

if 3 or more fixes have failed:

• stop fixing
• this signals architectural problem, not hypothesis problem
• discuss with human partner before attempting more fixes
• may require refactoring instead of patching

if investigation reveals issue is environmental or timing-dependent:

• document exactly what you investigated
• you've completed the process correctly
• implement appropriate handling: retry logic, timeouts, error messages
• add monitoring/logging for future investigation
• note: 95% of "no root cause found" cases are incomplete investigation

if your human partner uses redirect language:

• "is that not happening?" → you assumed without verifying, return to phase 1
• "will it show us...?" → you need more evidence gathering
• "stop guessing" → you're proposing fixes without understanding
• "ultra-think this" → question fundamentals, not just symptoms
• "we're stuck?" (frustrated) → your approach isn't working, restart from phase 1

output contract

successful debugging produces:

• documented root cause: written explanation of what was broken and why, with evidence
• failing test case: automated test that reproduces the issue (or test script), stored in version control
• minimal fix: single code change addressing root cause (not symptom), with git commit explaining the change
• passing tests: original failing test now passes, all other tests still pass
• verified resolution: confirmation that original issue is actually fixed in real system (not just in test)
• optional architectural notes: if 3+ fixes failed, document architectural concerns and refactoring discussion

if any of these are missing, the debugging process is incomplete.

outcome signal

you know systematic debugging worked when:

• you can clearly explain what was broken and why (you understand root cause, not just symptom)
• the failing test passes and stays passing
• all other tests continue passing (no new breakage)
• the original issue is actually resolved (not masked)
• the fix is minimal and doesn't require massive refactoring
• you didn't need more than 2-3 focused fix attempts
• you can point to the exact change (git commit) that fixed it
• team can reproduce and understand your fix without asking questions

anti-signals (you're doing it wrong):

• you have a theory but haven't written a failing test
• you've tried more than 3 fixes and issue still occurs
• you fixed the symptom but don't understand root cause
• you added multiple changes at once and don't know which one fixed it
• you skipped phases 1-3 and jumped to implementing
• you're proposing a fix without completing root cause investigation
• you're under time pressure and saying "just try this quick thing"