grepai-storage-gob

SKILL.md

GrepAI Storage with GOB

This skill covers using GOB (Go Binary) as the storage backend for GrepAI, the default and simplest option.

When to Use This Skill

Single developer projects

Small to medium codebases

Simple setup without external dependencies

Local development environments

What is GOB Storage?

GOB is Go's native binary serialization format. GrepAI uses it to store:

Vector embeddings

File metadata

Chunk information

Everything is stored in a single local file.
Advantages

Benefit
Description

🚀 Simple
No external services needed

⚡ Fast setup
Works immediately

📁 Portable
Single file, easy to backup

💰 Free
No infrastructure costs

🔒 Private
Data stays local

Limitations

Limitation
Description

📏 Scalability
Not ideal for very large codebases

👤 Single user
No concurrent access

🔄 No sharing
Can't share index across machines

💾 Memory
Loads into RAM for searches

Configuration

Default Configuration

GOB is the default backend. Minimal config:

# .grepai/config.yaml
store:
  backend: gob

Explicit Configuration

store:
  backend: gob
  # Index stored in .grepai/index.gob (automatic)

Storage Location

GOB storage creates files in your project's .grepai/ directory:

.grepai/
├── config.yaml    # Configuration
├── index.gob      # Vector embeddings
└── symbols.gob    # Symbol index for trace

File Sizes

Approximate .grepai/index.gob sizes:

Codebase
Files
Chunks
Index Size

Small
100
500
~5 MB

Medium
1,000
5,000
~50 MB

Large
10,000
50,000
~500 MB

Operations

Creating the Index

# Initialize project
grepai init

# Start indexing (creates index.gob)
grepai watch

Checking Index Status

grepai status

# Output:
# Index: .grepai/index.gob
# Files: 245
# Chunks: 1,234
# Size: 12.5 MB
# Last updated: 2025-01-28 10:30:00

Backing Up the Index

# Simple file copy
cp .grepai/index.gob .grepai/index.gob.backup

Clearing the Index

# Delete and re-index
rm .grepai/index.gob
grepai watch

Moving to a New Machine

# Copy entire .grepai directory
cp -r .grepai /path/to/new/location/

# Note: Only works if using same embedding model

Performance Considerations

Memory Usage

GOB loads the entire index into RAM for searches:

Index Size
RAM Usage

10 MB
~20 MB

50 MB
~100 MB

500 MB
~1 GB

Search Speed

GOB provides fast searches for typical codebases:

Codebase Size
Search Time

Small (100 files)
<50ms

Medium (1K files)
<200ms

Large (10K files)
<1s

When to Upgrade

Consider PostgreSQL or Qdrant when:

Index exceeds 1 GB

Need concurrent access

Want to share index across team

Codebase has 50K+ files

.gitignore Configuration

Add .grepai/ to your .gitignore:

# GrepAI (machine-specific index)
.grepai/

Why: The index is machine-specific because:

Contains binary embeddings

Tied to the embedding model used

Each machine should generate its own

Sharing Index (Not Recommended)

While you can copy the index file, it's not recommended because:

Must use identical embedding model

File paths are absolute

Different machines may have different code versions

Better approach: Each developer runs their own grepai watch.

Migrating to Other Backends

To PostgreSQL

Update config:

store:
  backend: postgres
  postgres:
    dsn: postgres://user:pass@localhost:5432/grepai

Re-index:

rm .grepai/index.gob
grepai watch

To Qdrant

Update config:

store:
  backend: qdrant
  qdrant:
    endpoint: localhost
    port: 6334

Re-index:

rm .grepai/index.gob
grepai watch

Common Issues

❌ Problem: Index file too large
✅ Solution: Add more ignore patterns or migrate to PostgreSQL/Qdrant

❌ Problem: Slow searches on large codebase
✅ Solution: Migrate to Qdrant for better performance

❌ Problem: Corrupted index
✅ Solution: Delete and re-index:

rm .grepai/index.gob .grepai/symbols.gob
grepai watch

❌ Problem: "Index not found" error
✅ Solution: Run grepai watch to create the index

Best Practices

Use for small/medium projects: Up to ~10K files

Add to .gitignore: Don't commit the index

Backup before major changes: Copy index.gob before experiments

Re-index after model changes: If you change embedding models

Monitor file size: Migrate if index exceeds 1GB

Output Format

GOB storage status:

✅ GOB Storage Configured

   Backend: GOB (local file)
   Index: .grepai/index.gob
   Size: 12.5 MB

   Contents:
   - Files: 245
   - Chunks: 1,234
   - Vectors: 1,234 × 768 dimensions

   Performance:
   - Search latency: <100ms
   - Memory usage: ~25 MB