indexion-refactor

SKILL.md

indexion refactor — Codebase Refactoring

Detect and eliminate duplication at three levels — textual, structural, and conceptual —
using indexion's analysis commands, then verify SoT is enforced.

When to Use

After adding a new abstraction (type, module, API layer)

After introducing a new file format or I/O boundary

When a fix required touching 3+ files for the same reason

When a "guard" or "skip" was added to work around a structural problem

When opendir, ENOENT, or similar filesystem errors appear from unexpected paths

When extracting shared code across packages

When cleaning up after a refactor (removing trivial wrapper functions)

Periodic SoT health check on a codebase

Three Levels of Duplication

Level
What it is
Tool
Example

Textual
Copy-pasted code blocks, identical functions
plan refactor
is_whitespace copied across 5 modules

Structural
Same logic structure with different names
plan solid, plan unwrap
cross-package extraction candidates, trivial wrappers

Conceptual
Same domain concept implemented independently
explore + manual analysis
Three modules each deciding "is this file an archive?"

Textual duplication is easy to find and fix. Conceptual duplication is the hardest and
most dangerous — it produces no copy-paste matches but means changing one concept requires
updating every scattered implementation.

Workflow

Phase 1: Clear textual duplication (plan refactor)

Start with high-confidence matches and work down.

# Step 1: Find 90%+ duplicates (high confidence)
indexion plan refactor --threshold=0.9 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  cmd/indexion/

indexion plan refactor --threshold=0.9 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  src/

Read the output in three sections:

Section
What it finds
Action

Similar Files
Files with high overall similarity
Investigate for structural consolidation

Duplicate Code Blocks
Line-level identical code between files
Extract to @common or shared module

Function-Level Duplicates
Structurally similar functions (TF-IDF on bodies)
Unify into single SoT function

Same-file duplicates (functions within one file at 90%+) are the highest-value
targets — easiest to fix, clearest wins. Example: get_global_data_dir and
get_global_cache_dir share 95% structure, extracted into resolve_os_dir.

# Step 2: Use grep to trace references before consolidating
indexion grep "TypeIdent:TfidfEmbeddingProvider" src/
indexion grep --semantic=name:is_whitespace src/

# Step 3: Fix, then re-run to confirm duplicates are gone
indexion plan refactor --threshold=0.9 --include='*.mbt' ...

# Step 4: Lower threshold and iterate
indexion plan refactor --threshold=0.85 --include='*.mbt' ...

plan refactor options:

Option
Default
Description

--threshold=FLOAT
0.7
Minimum similarity threshold

--strategy=NAME
hybrid
Similarity: hybrid, tfidf, bm25, jsd, ncd

--fdr=FLOAT
0
FDR correction (0=disabled)

--style=STYLE
raw
Output: raw, structured

--format=FORMAT
md
Output: md, json, text, github-issue

--name=NAME
--
Project name (for structured style)

--include=PATTERN
--
Include pattern (repeatable)

--exclude=PATTERN
--
Exclude pattern (repeatable)

-o, --output=FILE
stdout
Output file path

--specs-dir=DIR
kgfs
KGF specs directory

What remains after cleanup (stop signals):

Platform stubs (native.mbt / stub.mbt) — intentional platform branching

Type method similarity (to_string on different types) — different types, same pattern

CLI command boilerplate (command() functions) — @argparse API pattern, not duplication

Semantic-but-different functions (is_disqualifying_keyword vs is_skip_token) — different purpose

Phase 2: Extract cross-package shared code (plan solid)

After cleaning within each directory, find code that should be shared across packages.

# Find overlap between two packages
indexion plan solid --from=src/a,src/b

# Specify extraction target
indexion plan solid --from=src/a,src/b --to=src/common

# Use tree edit distance for precise function-level matching
indexion plan solid --from=src/a,src/b --strategy=apted

# Higher threshold for stricter matching
indexion plan solid --from=src/a,src/b --threshold=0.95

# Filter files
indexion plan solid --from=src/a,src/b --include='*.mbt' --exclude='*_test.mbt'

plan solid differs from plan refactor:

plan refactor
plan solid

Scope
Internal duplication within directories
Cross-directory overlap

Goal
Consolidate within a codebase
Extract shared code into a new package

Input
<path>
--from=dirA,dirB

plan solid options:

Option
Default
Description

--from=DIRS
(required)
Source directories (comma-separated or repeatable)

--to=DIR
--
Target directory for extraction

--rules=FILE
--
Rules file (.solidrc)

--rule=RULE
--
Inline rule (repeatable)

--threshold=FLOAT
0.9
Minimum similarity threshold

--strategy=NAME
tfidf
Similarity: tfidf, apted, tsed

--include=PATTERN
--
Include pattern (repeatable)

--exclude=PATTERN
--
Exclude pattern (repeatable)

--format=FORMAT
md
Output: md, json, github-issue

-o, --output=FILE
stdout
Output file path

--specs-dir=DIR
kgfs
KGF specs directory

Workflow:

Run plan refactor on each directory individually first to clean internal duplication

Run plan solid --from=dirA,dirB to find cross-directory extraction candidates

Extract shared code following the plan's recommendations

Use indexion grep "TypeIdent:SharedType" to verify all references are updated

Phase 3: Remove unnecessary wrappers (plan unwrap)

After consolidation, clean up trivial delegation functions that add indirection without value.

# Step 1: Quick check
indexion grep --semantic=proxy src/

# Step 2: Detailed report
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Step 3: Preview changes (safe — no files modified)
indexion plan unwrap --dry-run --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Step 4: Apply fixes
indexion plan unwrap --fix --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Step 5: Run tests
moon test --target native

What gets detected: Functions whose body is a single function call with
all arguments forwarded as simple identifiers — no control flow, no transforms.

// Detected (default) — trivial delegation
fn matches_pattern(text : String, pat : String) -> Bool {
  @glob.glob_match(text, pat)
}

// Excluded by default (use --all to include)
fn length(self : MyList) -> Int {
  self.items.length()    // self-delegation (encapsulation)
}
fn emit(value : String) -> Action {
  Emit(value)            // bare constructor
}

plan unwrap modes:

Mode
Flag
Description

Report
(default)
List wrappers found

Preview
--dry-run
Show all edits without modifying files

Fix
--fix
Apply edits to files

plan unwrap options:

Option
Default
Description

--dry-run
--
Preview edits

--fix
--
Apply edits

--all
--
Include self-delegation and bare constructor wrappers

--include-self
--
Include self.field.method patterns

--include-bare
--
Include bare constructor wrappers

--include=PATTERN
--
Include pattern (repeatable)

--exclude=PATTERN
--
Exclude pattern (repeatable)

--format=FORMAT
md
Output: md, json, text

-o, --output=FILE
stdout
Output file path

--specs-dir=DIR
kgfs
KGF specs directory

Review before removing:

Platform wrappers (FFI, @osenv_path) are abstraction layers, not accidental indirection

Public API wrappers used by external packages — removing them is a breaking change

Always --dry-run first

Phase 4: Detect concept-level duplication (explore + analysis)

This is the hardest level. Textual and structural tools won't find it because the
code is different — but the concept is the same.

# Find which files share vocabulary (= work in the same concept domain)
indexion explore --threshold=0.4 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  src/ cmd/

Files at 40-60% similarity without structural duplication are concept neighbors —
they use the same terms because they deal with the same domain.

For each high-similarity pair, ask: "What concept do they share, and who owns it?"

# Inspect shared vocabulary with tree structure comparison
indexion explore file_a.mbt file_b.mbt --threshold=0 --strategy=apted

Common patterns of concept leakage:

Symptom
Concept leaked
Fix

Both files call is_X(spec) then Y::from_spec(spec)
"Determine if X and configure Y"
Extract try_do_X(path, spec) into the module that owns X

Both files @fs.read_file_to_string(path) when content is already loaded
"Read file content"
Pass content as argument, don't re-read

Both files parent_dir(path) then @fs.read_dir(dir)
"List sibling files"
Centralize directory walking into pipeline

Multiple if is_virtual_path(x) { skip } guards
"Real vs virtual path"
Make the type system prevent virtual paths from reaching here

Both files buf.write_string("\n"); buf.write_string(x)
"Join text entries"
Extract join_text_entries() into the owning module

Phase 5: Consolidate into SoT

The module that defines the concept should be the only one that implements the logic.

Rules:

One concept, one module, one function. If "extract text from archive" appears in vfs.mbt, discover.mbt, and args.mbt, it belongs in vfs.mbt only.

Callers receive results, not ingredients. Don't export is_archive_spec + ArchiveSpec::from_spec + expand_archive separately. Export try_extract_archive_text(path, spec) -> String?.

Guards are symptoms, not fixes. if is_virtual_path(x) { skip } means virtual paths shouldn't reach here at all. Fix the source, not the sink.

Re-reading from disk what's already in memory is a concept leak. If SupportedFile.content holds the text, no downstream code should call @fs.read_file_to_string(file.path).

Phase 6: Verify

# Confirm textual duplication is gone
indexion plan refactor --threshold=0.9 \
  --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' \
  src/ cmd/indexion/

# Confirm concept similarity is reduced
indexion explore file_a.mbt file_b.mbt --threshold=0

# Confirm wrappers are cleaned up
indexion plan unwrap --include='*.mbt' --exclude='*_wbtest.mbt' \
  --exclude='*moon.pkg*' --exclude='*pkg.generated*' src/

# Run tests
moon test --target native

After SoT consolidation:

Textual similarity between the concept owner and its callers drops

Callers become shorter (one API call instead of multi-step logic)

The concept owner may grow, but it's the single place to change

Phase 7: Prove non-recurrence with tests

Write a test that structurally prevents the old pattern from recurring:

test "SoT: SupportedFile.path is always a real filesystem path" {
  // Create an archive, run load_supported_file_info
  // Assert: no path contains "!/"
  // Assert: every path passes @fs.path_exists
}

The test doesn't check behavior — it checks the SoT invariant.

Red Flags

"I need to add a guard here"

If you're adding if is_special_case(x) { skip } to a function that shouldn't receive
special cases, the problem is upstream. The function's caller should never pass that value.

"It works but prints errors to stderr"

Stderr messages from C runtime (opendir: No such file or directory) mean invalid data
reached a system call. catch absorbs the error, but perror() already printed.
The only fix is preventing invalid data from reaching the call.

"I'll fix it in each command separately"

If the same fix is needed in explore, search, grep, reconcile, plan documentation...
the fix belongs in the shared pipeline, not in each command.

"The similarity is just shared vocabulary, not real duplication"

40-60% TF-IDF similarity between modules that aren't supposed to share concepts is a
warning. The vocabulary match IS the signal.

Quick Reference: Which Command When

Question
Command

"What files are similar?"
explore --format=list

"What exactly is duplicated?"
plan refactor --threshold=0.9

"What code overlaps between packages A and B?"
plan solid --from=A,B

"Which functions are trivial wrappers?"
plan unwrap or grep --semantic=proxy

"What concept do these files share?"
explore file_a file_b --threshold=0 --strategy=apted

"Has the duplication been fixed?"
Re-run plan refactor with same threshold