Systematic Debugging

intent

use this skill whenever you hit a bug, test failure, unexpected behavior, performance problem, build failure, or integration issue. the core principle is non-negotiable: always find root cause before attempting fixes. symptom fixes are failure. this skill is especially critical under time pressure, when a "quick fix" seems obvious, after multiple failed fix attempts, or when you don't fully understand the issue. don't skip it because the problem seems simple, because you're in a hurry, or because someone wants it fixed now. systematic debugging is faster than guess-and-check thrashing.

inputs

no external connections required for the debugging process itself. however, you will need:

• access to error messages, stack traces, and logs from the failing system
• git history (recent commits, diffs) if investigating code changes
• the actual codebase, config files, and environment setup
• ability to reproduce the issue or gather diagnostic data from logs
• reference implementations or documentation for patterns you're comparing against
• access to supporting techniques: root-cause-tracing.md, defense-in-depth.md, condition-based-waiting.md (same directory)

optional but useful:

• monitoring dashboards or observability tools (DataDog, New Relic, CloudWatch)
• version control logs and CI/CD pipeline outputs
• access to your human partner for architectural questions (needed only if 3+ fixes fail)

procedure

you must complete each phase sequentially. do not skip ahead or jump to implementation.

phase 1: root cause investigation

stop. do not attempt any fix until you complete this phase.

1. read error messages completely

   • input: error output, stack traces, logs
   • activity: don't skip past errors or warnings. they often contain the exact solution. read stack traces completely. note line numbers, file paths, error codes.
   • output: documented error context with line numbers and file paths

2. reproduce the issue consistently

   • input: bug report, issue description, system state
   • activity: can you trigger it reliably? what are the exact steps? does it happen every time? if not reproducible, gather more data instead of guessing.
   • output: repeatable steps OR evidence from logs showing the failure pattern
   • edge case: if issue is flaky/intermittent, don't skip. log it and repeat until you see pattern in timing, state, or concurrency.

3. check recent changes

   • input: git log, commit history, deployment records, config changes
   • activity: what changed that could cause this? review git diff, recent commits, new dependencies, config changes, environmental differences.
   • output: list of recent changes that could impact the failing system

4. gather diagnostic evidence (multi-component systems)

   • input: system architecture, component boundaries, current logs
   • activity: for each component boundary, log what data enters and exits. verify environment/config propagation. check state at each layer. run once to gather evidence showing where it breaks, then analyze to identify the failing component, then investigate that component.
   • example (multi-layer system):

     # Layer 1: Workflow
     echo "=== Secrets available in workflow: ==="
     echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
     
     # Layer 2: Build script
     echo "=== Env vars in build script: ==="
     env | grep IDENTITY || echo "IDENTITY not in environment"
     
     # Layer 3: Signing script
     echo "=== Keychain state: ==="
     security list-keychains
     security find-identity -v
     
     # Layer 4: Actual signing
     codesign --sign "$IDENTITY" --verbose=4 "$APP"
     

   • output: diagnostic logs showing which layer fails (e.g., secrets to workflow ok, workflow to build broken)

5. trace data flow backward

   • input: error location, current values, call stack
   • activity: where does the bad value originate? what called this with bad value? keep tracing up until you find the source. fix at source, not at symptom. see root-cause-tracing.md for complete technique.
   • output: documented chain from symptom back to original trigger

phase 2: pattern analysis

do not implement anything yet. find the pattern first.

1. find working examples

   • input: broken code, codebase
   • activity: locate similar working code in the same codebase. what works that's similar to what's broken?
   • output: reference implementation that exhibits desired behavior

2. compare against references

   • input: reference implementation, pattern documentation, broken code
   • activity: if implementing a pattern, read the reference completely. don't skim. read every line. understand the pattern fully before applying.
   • output: full understanding of the reference pattern

3. identify differences

   • input: working example, broken code
   • activity: list every difference between working and broken, however small. don't assume "that can't matter."
   • output: documented list of differences (config, state, dependencies, assumptions)

4. understand dependencies

   • input: broken code, system documentation
   • activity: what other components does this need? what settings, config, environment? what assumptions does it make?
   • output: dependency map (components, configs, environment vars, assumptions)

phase 3: hypothesis and testing

scientific method. one hypothesis, one test, one variable at a time.

1. form single hypothesis

   • input: phase 1 evidence, phase 2 pattern analysis
   • activity: state clearly: "i think X is the root cause because Y." write it down. be specific, not vague.
   • output: written hypothesis (one sentence, specific)

2. test minimally

   • input: hypothesis, failing system
   • activity: make the smallest possible change to test the hypothesis. one variable at a time. don't fix multiple things at once.
   • output: test result (pass/fail)

3. evaluate result

   • input: test result
   • activity: did it work? yes, go to phase 4. didn't work? form a new hypothesis. do not add more fixes on top.
   • output: decision (proceed to phase 4 OR loop back with new hypothesis)

4. when knowledge is missing

   • input: unclear component or pattern
   • activity: say "i don't understand X." don't pretend to know. ask for help. research more.
   • output: research results or partner feedback

phase 4: implementation

fix the root cause. not the symptom. not adjacent issues.

1. create failing test case

   • input: reproduction steps, test framework
   • activity: write the simplest possible automated test that fails. if no framework, write a one-off test script. you must have a failing test before fixing. use superpowers:test-driven-development for proper test structure.
   • output: failing test that reproduces the issue
   • edge case: if issue is production-only, write a test that simulates production conditions (e.g., concurrent requests, timing, state)

2. implement single fix

   • input: root cause from phase 1, test case from step 1
   • activity: address the root cause identified in phase 1. one change at a time. no "while i'm here" improvements. no bundled refactoring.
   • output: code changes addressing root cause

3. verify fix works

   • input: fixed code, test case, original system
   • activity: test passes now? no other tests broken? issue actually resolved? use superpowers:verification-before-completion before claiming success.
   • output: passing tests, verified issue resolution, no regression

4. if fix doesn't work

   • input: failing test, attempted fix
   • activity: stop. count how many fixes you've tried. if less than 3, return to phase 1 and re-analyze with new information. if 3 or more, jump to step 5.
   • output: decision (phase 1 restart OR architectural review)

5. if 3+ fixes failed: question architecture

   • input: failed fixes, architectural patterns
   • activity: watch for this pattern: each fix reveals new shared state/coupling/problem in a different place. fixes require "massive refactoring" to implement. each fix creates new symptoms elsewhere. if you see this, stop. discuss with your human partner. is this pattern fundamentally sound? are you sticking with it through sheer inertia? should you refactor architecture vs. continue fixing symptoms? this is not a failed hypothesis. this is a wrong architecture.
   • output: architectural discussion decision (refactor OR continue with new understanding)

decision points

when to stop and restart phase 1:

• if you catch yourself thinking "quick fix for now, investigate later," stop and go back to phase 1
• if you're proposing multiple fixes without investigation, stop and go back to phase 1
• if you're on fix attempt 2 or 3 and each one reveals a new problem in a different place, stop. question architecture instead of trying fix 4.

when you see these signals from your human partner, you're doing it wrong:

• "is that not happening?" signals you assumed without verifying. return to phase 1.
• "will it show us...?" signals you should have added diagnostic instrumentation. return to phase 1.
• "stop guessing" signals you're proposing fixes without understanding. return to phase 1.
• "ultra-think this" signals you need to question fundamentals, not symptoms. return to phase 1.
• "we're stuck?" (frustrated) signals your approach isn't working. return to phase 1.

single fix vs. multiple fix attempts:

• if less than 3 fixes have failed, return to phase 1 with new information
• if 3 or more fixes have failed, skip phase 1 and question the architecture (phase 4, step 5)

flaky/intermittent issues:

• if issue doesn't reproduce consistently, don't guess. gather more diagnostic data and look for timing, state, or concurrency patterns
• if investigation reveals the issue is truly environmental, timing-dependent, or external, document what you investigated and implement appropriate handling (retry, timeout, error message). then add monitoring/logging.

when external dependencies are involved:

• if you depend on third-party APIs or services, check rate limits, auth expiry, network timeouts, and empty result sets as part of phase 1 investigation
• if these are constraints, implement defensive handling rather than guessing at timing

output contract

phase 1 completion: documented error context, reproduction steps, recent changes, diagnostic evidence showing which component fails, data flow trace from symptom to origin.

phase 2 completion: reference implementation identified, differences documented, dependency map created.

phase 3 completion: written hypothesis, test result, decision (proceed or loop).

phase 4 completion: failing test case passes, no regressions, issue verified as resolved. if architectural review required, documented discussion and decision.

success means: root cause identified and fixed, not symptom patched. test coverage added. no new issues introduced.

outcome signal

you know this skill worked when:

• the failing test now passes
• the original issue no longer reproduces
• no other tests or functionality broke
• you can explain exactly why the issue happened (not "i'm not sure but the fix works")
• if 3+ fixes failed, you've had an architectural discussion instead of continuing to patch symptoms

you know you skipped the process when:

• you proposed a fix without completing phase 1
• you tried multiple fixes without going back to phase 1 to re-analyze
• you say "i don't fully understand but this might work"
• you're on fix attempt 4+ without questioning architecture
• your human partner asks "is that not happening?" or "stop guessing"