Memory guardian for OpenClaw — three-layer anti-amnesia with capacity management. v6.0: MEMORY.md soft cap (5000 chars) + USER.md cap (1500 chars) prevent pr...
---
name: memory-engine
description: "Memory guardian for OpenClaw — three-layer anti-amnesia with capacity management. v6.0: MEMORY.md soft cap (5000 chars) + USER.md cap (1500 chars) prevent prompt explosion. Auto-snapshot before compaction/GC. Integrity monitoring restores from snapshot if MEMORY.md is wiped. Works alongside OpenClaw native memorySearch (vector+hybrid) when available, falls back to built-in FTS5 when not. Focuses on what native lacks: auto-write, session extraction, watcher daemon, GC, health checks, backup, capacity management. Layer 1 (system): cron every 1h + watcher detects resets <30s + integrity checks. Layer 2 (platform): memory-flush + session-memory hook. Layer 3 (agent): write/search/maintain/boot scripts. Use when: recalling past conversations, writing memory entries, checking memory health, maintaining search index. Triggers: 'remember', 'recall', 'what did we discuss', 'memory status', 'search memory', 'reindex'."
---
# Memory Engine 🧠⚡
**Persistent memory that survives session restarts — with three layers of protection against amnesia.**
## The Problem
AI agents wake up blank every session. Existing solutions either:
- Need embedding APIs (cost money, need keys)
- Only search (useless if nothing was written)
- Rely on the AI "remembering" to save (it won't)
Memory Engine solves all three.
## v5.0 — Memory Guardian Architecture
**v5.0 redefines Memory Engine's role**: instead of a "full-stack memory system", it's now a **memory guardian** — handling everything OpenClaw native memory *doesn't* do.
| Feature | Native memorySearch | Memory Engine |
|---------|-------------------|---------------|
| Semantic vector search | ✅ (embedding API) | ❌ (deferred to native) |
| BM25 keyword search | ✅ (hybrid mode) | ✅ (FTS5 fallback) |
| Auto-write daily logs | ❌ | ✅ `memory-write.js` |
| Session reset detection | ❌ | ✅ `memory-watcher.sh` (<30s) |
| Active session extraction | ❌ | ✅ `--active` mode |
| Health check + GC | ❌ | ✅ `memory-maintain.js` |
| Session resume context | ❌ | ✅ `memory-resume.js` |
| GitHub backup/restore | ❌ | ✅ `memory-backup.sh` |
| Zero-dep writing | ❌ (needs embedding API) | ✅ (pure Node.js) |
### Upgrade from v3/v4 (zero data loss)
```bash
node scripts/memory-migrate.js # preview changes
node scripts/memory-migrate.js --apply # apply (backs up config first)
node scripts/memory-migrate.js --rollback --apply # revert if needed
```
The migrate tool:
1. Enables native `memorySearch` with hybrid search + temporal decay
2. Enables `session-memory` hook (saves session on /new and /reset)
3. Updates cron to 1h if still at 6h
4. Preserves ALL existing data — your memory files, index, daily logs untouched
5. FTS5 search remains as fallback if native isn't configured
**No breaking changes**: all v3/v4 commands still work. `memory-search.js` auto-detects native mode and adapts. `memory-boot.js` skips FTS5 indexing when native is active (saves time).
## Three-Layer Anti-Amnesia Architecture
```
Layer 1: SYSTEM (runs without AI) Layer 2: PLATFORM (OpenClaw built-in)
┌─────────────────────────┐ ┌──────────────────────────┐
│ cron job (every 1h) │ │ memory-flush │
│ • rebuild search index │ │ • auto-triggers before │
│ • health check │ │ context compaction │
│ • auto-create daily log │ │ • forces AI to write │
│ if missing │ │ memory/YYYY-MM-DD.md │
│ • active session extract│ │ • user-invisible │
│ • ensure watcher alive │ │ │
│ │ │ ⚙️ Built into OpenClaw, │
│ 🔴 watcher daemon │ │ just needs config │
│ • polls every 30s │ └──────────────────────────┘
│ • detects session reset │
│ • extracts memory <30s │
│ • auto-started by cron │
│ │
│ ⚙️ Pure shell/node, │
│ no AI involvement │
└─────────────────────────┘
Layer 3: AGENT (AI calls these)
┌──────────────────────────────────────────────────┐
│ memory-write.js → append daily log / MEMORY.md│
│ memory-search.js → FTS5 search (native fallback)│
│ memory-index.js → build/update FTS5 index │
│ memory-maintain.js → stats / rebuild / prune │
│ memory-migrate.js → upgrade/rollback helper NEW│
│ │
│ 📋 Governed by AGENTS.md rules: │
│ Session start → health check + read MEMORY.md │
│ During conversation → log key events │
│ Session end → summarize + promote to MEMORY.md │
└──────────────────────────────────────────────────┘
```
**Why three layers?** Any single layer can fail:
- AI forgets to write → cron + memory-flush still capture data
- Cron stops → AI + memory-flush still work
- Memory-flush doesn't trigger (short session) → AI + cron still work
All three failing simultaneously = near zero probability.
## Setup (2 minutes)
### Quick Setup (v5.0 — recommended for OpenClaw ≥ 2026.4)
```bash
# 1. Install cron (watcher + maintenance)
(crontab -l 2>/dev/null; echo "0 * * * * $(pwd)/scripts/memory-cron.sh") | crontab -
# 2. Auto-configure native search + hooks
node scripts/memory-migrate.js --apply
# 3. Restart gateway
openclaw gateway restart
```
That's it! Native memorySearch handles search, we handle everything else.
### Classic Setup (FTS5 mode — for OpenClaw < 2026.4 or no embedding API)
```bash
# 1. Install FTS5 dependency
npm install -g better-sqlite3
# 2. Install cron job
(crontab -l 2>/dev/null; echo "0 * * * * $(pwd)/scripts/memory-cron.sh") | crontab -
```
### Step 3: Enable memory-flush (add to openclaw.json)
```json
{
"agents": {
"defaults": {
"compaction": {
"memoryFlush": {
"enabled": true,
"softThresholdTokens": 4000,
"prompt": "Pre-compaction memory flush. Store durable memories in memory/YYYY-MM-DD.md (create memory/ if needed). APPEND only, do not overwrite. Include: key decisions, task outcomes, user preferences, project status changes. Format each entry as: - HH:MM [tag] description. If nothing worth storing, reply with NO_REPLY."
},
"reserveTokensFloor": 20000,
"postIndexSync": "async"
},
"heartbeat": {
"every": "30m"
}
}
}
}
```
> **⚠️ Important**: Do NOT instruct memory-flush to write MEMORY.md or run exec commands.
> During flush, OpenClaw restricts writes to `memory/YYYY-MM-DD.md` only — MEMORY.md, SOUL.md, AGENTS.md are read-only.
> Exec/shell commands may also be restricted. The cron job handles index rebuilding independently.
### Step 4: Add rules to AGENTS.md
```markdown
## Session Startup (MANDATORY)
1. Run: node skills/memory-engine/scripts/memory-boot.js
(single command: health check + index update + MEMORY.md content)
2. Use memory-search.js for specific recall (NOT full file reads)
## During Conversation (MANDATORY)
- Key decisions: node skills/memory-engine/scripts/memory-write.js --today "decision" --tag decision
- Task completion: node skills/memory-engine/scripts/memory-write.js --today "done" --tag done
## Session End (MANDATORY)
1. node skills/memory-engine/scripts/memory-write.js --today "Session summary"
2. node skills/memory-engine/scripts/memory-write.js --core "durable fact" --section category
3. node skills/memory-engine/scripts/memory-index.js
```
## Commands
### Migrate (upgrade/rollback) — NEW in v5.0
```bash
node scripts/memory-migrate.js # preview: what would change
node scripts/memory-migrate.js --apply # apply changes (backs up config)
node scripts/memory-migrate.js --rollback --apply # revert to FTS5-only mode
node scripts/memory-migrate.js --status # show current mode (JSON)
```
Enables native memorySearch, session-memory hook, optimal cron frequency.
Config backup created automatically before changes. Zero data loss guaranteed.
### Boot (session startup)
```bash
node scripts/memory-boot.js # health + index + MEMORY.md output
```
Replaces 3 separate calls (`--status` + `memory-index.js` + `read MEMORY.md`) with one command.
Auto-recovers from corrupted database. v5.0: auto-detects native mode, skips FTS5 when native active.
### Write (most important!)
```bash
# Daily log (auto-creates memory/YYYY-MM-DD.md)
node scripts/memory-write.js --today "Deployed v2.0 to production"
node scripts/memory-write.js --today "User prefers dark mode" --tag preference
# Long-term memory (appends to MEMORY.md, capacity-managed — 5000 char soft cap)
node scripts/memory-write.js --core "Stack: Next.js + PostgreSQL + Redis"
node scripts/memory-write.js --core "Deploy via Docker Compose" --section infrastructure
# User profile (appends to USER.md — 1500 char soft cap)
node scripts/memory-write.js --user "Prefers concise answers, dislikes verbose explanations"
# Snapshot MEMORY.md (before risky operations)
node scripts/memory-write.js --snapshot
# Health check (now includes capacity % for MEMORY.md and USER.md)
node scripts/memory-write.js --status
```
Health output:
```json
{
"hasTodayLog": true,
"hasCoreMemory": true,
"coreMemory": { "chars": 2651, "cap": 5000, "usage": "53%" },
"userProfile": { "chars": 380, "cap": 1500, "usage": "25%" },
"snapshots": 3,
"gapCount": 2,
"healthScore": 71,
"warnings": ["2 gaps in last 14 days"]
}
```
### Resume (zero-latency session recovery) — Enhanced in v4.0
```bash
node scripts/memory-resume.js # generate recovery summary (<2000 tokens)
node scripts/memory-resume.js --max-chars 1000 # shorter for tight budgets
```
Use in AGENTS.md as the first session startup step. Outputs:
- Last session's final topic & status
- Recent 3 days' key entries (priority-sorted by tag)
- Active MEMORY.md sections summary
- **NEW v4.0**: Auto-detects unextracted reset sessions and extracts them before resuming
(eliminates the "resume misses recent work" gap)
### Search (0 tokens, ~300 token results)
```bash
node scripts/memory-search.js "deployment plan" # top 3, 200 chars
node scripts/memory-search.js "API重构" --json --max 5 # CJK support
node scripts/memory-search.js "query" --max 1 --max-chars 100 # ultra-minimal
# Date filtering (auto-detects "4月2日" / "April 2" in queries)
node scripts/memory-search.js "4月2日完成了什么" # auto date filter
node scripts/memory-search.js "tasks" --date 2026-04-02 # exact date
node scripts/memory-search.js "progress" --recent 7 # last 7 days
node scripts/memory-search.js "bugs" --after 2026-04-01 --before 2026-04-08
# Last N entries (no query needed, time-ordered) — NEW in v3.0
node scripts/memory-search.js --last 5 # last 5 entries across all logs
node scripts/memory-search.js --last 10 --today # today's last 10
node scripts/memory-search.js --last 3 --tag done # last 3 [done] entries
node scripts/memory-search.js --last 5 --tag decision # last 5 decisions
```
### Index
```bash
node scripts/memory-index.js # incremental (skips unchanged)
node scripts/memory-index.js --force # full rebuild
```
### Maintain
```bash
node scripts/memory-maintain.js # show stats
node scripts/memory-maintain.js --reindex # force rebuild
node scripts/memory-maintain.js --prune-days 90 # trim old entries
# GC: detect & remove stale MEMORY.md entries — NEW in v3.0
node scripts/memory-maintain.js --gc # preview stale entries
node scripts/memory-maintain.js --gc --apply # remove them
```
GC detects: old version references (>14d), completed TODOs, entries >60d old, duplicates.
### Consolidate (capacity management) — NEW in v6.0
```bash
node scripts/memory-maintain.js --consolidate # analyze merge candidates
node scripts/memory-maintain.js --consolidate --apply # remove duplicates (snapshots first)
```
When MEMORY.md exceeds 70% of the 5000-char soft cap, use this to find and merge duplicate entries.
Auto-snapshots MEMORY.md before any destructive operation.
For deeper consolidation (merging semantically similar entries), manually edit MEMORY.md.
### Auto-Extract (session transcript mining) — Enhanced in v4.0
```bash
node scripts/memory-auto-extract.js # extract from active session (full scan)
node scripts/memory-auto-extract.js <path.jsonl> # extract from specific file
node scripts/memory-auto-extract.js --scan # scan all unprocessed reset sessions
node scripts/memory-auto-extract.js --active # incremental extract from active sessions (NEW v4.0)
node scripts/memory-auto-extract.js --dry-run # preview without writing
```
Extracts: git commits, file writes, deployments, PM2 restarts, user instructions,
test results, API responses, config changes, version releases, bug fixes (NEW v4.0).
`--active` mode reads only new bytes since last extraction (offset-based), safe for frequent runs.
Integrated into cron: auto-scans reset sessions + active sessions every 1h.
### Watcher (real-time session reset detection) — NEW in v4.0
```bash
bash scripts/memory-watcher.sh # start daemon (polls every 30s)
bash scripts/memory-watcher.sh --once # single check, no loop
```
- Detects new `.reset.` files within 30 seconds
- Immediately extracts memory + rebuilds index
- Auto-started by cron (ensures watcher is always running)
- PID managed via `.memory/watcher.pid`
- Solves the #1 failure mode: session reset → instant amnesia
### Compact (compress old logs)
```bash
node scripts/memory-compact.js --stats # show candidates
node scripts/memory-compact.js --older-than 30 --dry-run # preview
node scripts/memory-compact.js --older-than 30 # execute (originals → memory/archive/)
```
Compaction extracts headings + key bullets (✅/🔴/重要), saves originals to `memory/archive/`. Typical savings: 60-70%.
## How Search Works
1. **Chunking**: ~300 chars per chunk, 60 char overlap, split at headings
2. **Indexing**: SQLite FTS5 full-text index + file metadata
3. **Search**: BM25 + phrase matching for English; TF-density bigram for CJK
4. **Date filtering**: Auto-detects dates in queries (4月2日, April 2, 2026-04-02)
5. **Ranking**: TF_density × coverage × core_boost(1.5×) × temporal_decay(30-day half-life)
5. **Output**: truncated snippets with `file#line` citations
## Token Budget
| Operation | Tokens |
|-----------|--------|
| Search (SQLite) | 0 |
| Results (3 × 200 chars) | ~300 |
| Read full daily file | ~2000 |
| Read 4 daily files | ~8000 |
| **Savings** | **~95%** |
## Backup & Disaster Recovery
### Auto-backup (recommended)
Memory-cron automatically pushes to GitHub every 6 hours (if git remote is configured):
```bash
# One-time setup:
cd ~/.openclaw/workspace
git init && git branch -M main
git remote add origin https://<token>@github.com/<user>/openclaw-workspace.git
bash skills/memory-engine/scripts/memory-backup.sh # first push
# After this, cron handles it automatically every 6h
```
### Restore after reinstall
```bash
# After fresh OpenClaw install:
bash memory-restore.sh https://<token>@github.com/<user>/openclaw-workspace.git
# Restores: memories, config, skills, crontab, rebuilds search index
```
### What's backed up
- `MEMORY.md` + `memory/*.md` (all memories — irreplaceable)
- `SOUL.md`, `USER.md`, `IDENTITY.md` (personality)
- `AGENTS.md`, `TOOLS.md`, `HEARTBEAT.md` (behavior rules)
- `skills/` (all installed skills)
- `openclaw.json` → backup copy
- `crontab` → backup copy
### What's NOT backed up (rebuilt automatically)
- `.memory/index.sqlite` (rebuilt from md files by restore script)
- `.memory/cron.log` (transient)
## Requirements
### v5.0 Native mode (recommended)
- Node.js 18+
- OpenClaw ≥ 2026.4 (for native memorySearch + session-memory hook)
- `better-sqlite3` **optional** (only needed for FTS5 fallback)
- No API keys required (OpenClaw auto-detects embedding providers)
### Classic FTS5 mode
- Node.js 18+
- `better-sqlite3` (for search/index)
- Write + health check: zero dependencies
- No API keys, no cloud, no Docker
don't have the plugin yet? install it then click "run inline in claude" again.