cairn

local-first retrieval grounding for an llm. you curate what gets indexed (no auto-crawl), cairn add brings it in, and either you or a model running over mcp can query the result. one sqlite file, zero telemetry, no accounts.

intent

cairn builds a hybrid search index over codebases, design docs, audit notes, web pages, pdfs, and raw text you deliberately ingest. use this when you need retrieval-augmented generation (rag) with full control over what's indexed, zero network egress except on your terms, and queryable knowledge graphs extracted from code (ast-based tree-sitter) and markdown (llm-extracted entities and edges). five query surfaces cover chunk search, entity graphs, composed retrieval, shortest-path finding between concepts, and tag-filtered discovery. deploy embedded (in-process, ~785 mb models) or pointed at a local ollama daemon. safe for mcp-connected agents because model invocation is disabled by default and ingestion accepts allowlists and size caps.

inputs

required

• target path or url , local directory (code kind), file path (file or pdf kind), or explicit url for web ingestion. tree-sitter-supported languages auto-parse on ingest; unsupported formats fall back to chunking + embedding only.
• sqlite database , defaults to ~/.cairn/cairn.db. created on first use. one file per logical index; separate instances for sensitive / shared / test contexts.

optional inputs

• label (string) , human-readable name for the source. defaults to basename of path or domain of url. used in search results to disambiguate hits across multiple indexed sources.
• kind (enum: code, file, pdf, web) , ingest type. auto-detected for local paths (dir = code, .pdf = pdf, others = file). explicit for urls (web).
• include / exclude globs (array of strings) , filter file traversal during addCode. e.g., include: ['**/*.rs'] restricts to rust; exclude: ['**/node_modules/**'] skips dependencies.
• tag filter (string) , restrict search/graph queries to entities with a specific tag. tags are emitted by the llm during doc extraction (e.g., attack, invariant, mev). discover in-use tags via the tags tool.
• embedding model (path or ollama model name) , defaults to nomic-embed-text on ollama or auto-cached gguf on embedded runtime. switching models requires re-indexing.
• chat model (env: CAIRN_CHAT_MODEL or param) , llm for doc-extraction concept/edge/tag inference. defaults to qwen3-0.6b q8 (sub-1b, local, cheap). only used during ingest if markdown/pdf/web content is present.

environment variables

name	default	impact
CAIRN_RUNTIME	ollama	switch backend: ollama (daemon @ 127.0.0.1:11434) or embedded (in-process node-llama-cpp, ~785 mb gguf auto-download on first use).
CAIRN_OFFLINE	unset	when 1 or true, blocks web ingestion (cairn add <url>) and blocks hugging face gguf auto-download (embedded runtime only). pre-cache models to ~/.cairn/models/ first. localhost ollama is not blocked.
CAIRN_ALLOWED_ROOTS	unset (any path allowed)	comma-separated absolute paths. when set, cairn add rejects any local file/code/pdf kind outside these roots. trailing slashes normalized. defense-in-depth for mcp-connected agents.
CAIRN_MAX_INGEST_FILES	10000	pre-flight check: abort ingest if directory walk exceeds this count. bypassable via cli --force (mcp does not expose force).
CAIRN_MAX_INGEST_BYTES	524288000 (500 mb)	pre-flight check: abort if total bytes in source tree exceed this. bypassable via cli --force (mcp does not expose force).
CAIRN_CPU_ONLY	unset	force cpu-only inference on embedded runtime (no cuda/metal).
CAIRN_DEBUG_DOC	unset	log per-document extraction counts during ingest (concept/edge/tag yields). noise for large ingests but useful for tuning.

external connections

• localhost ollama (default) , expects nomic-embed-text model pre-pulled. set CAIRN_RUNTIME=ollama and ensure ollama is running at http://127.0.0.1:11434. embedding requests made per-chunk; respects ollama timeout (network error causes ingest to fail and roll back).
• hugging face cdn (embedded runtime only) , auto-downloads nomic-embed-text and qwen3-0.6b-q8 gguf files on first use unless CAIRN_OFFLINE=1. downloads go to ~/.cairn/models/. sha256s published in docs/setup.md for air-gapped verification. zero downloads if models are pre-cached.
• local filesystem , read-only on query, read-write on ingest. respects CAIRN_ALLOWED_ROOTS allowlist if set.

procedure

ingest workflow

step 1: initialize cairn instance

• input: optional dbPath (string, defaults to ~/.cairn/cairn.db), optional runtime ('ollama' or 'embedded'), optional environment variables already set
• action: construct new Cairn({ dbPath, runtime }) in library or invoke cairn-mcp for mcp mode. on first run, creates sqlite schema (version 2). on embedded runtime, auto-downloads models to ~/.cairn/models/ unless CAIRN_OFFLINE=1 (fails fast if offline but models not pre-cached).
• output: cairn instance ready to accept ingest and query calls. logs single startup warning if CAIRN_ALLOWED_ROOTS is unset and you're using mcp (operator notification that path allowlisting is available but not configured).

step 2: validate ingest target against safety gates

• input: target path/url, kind, source label
• action: if target is local path (code/file/pdf kind), check against CAIRN_ALLOWED_ROOTS allowlist if set. if allowlist is configured and target is outside any root, abort with clear error message naming the rejected path and listing allowed roots. if CAIRN_OFFLINE=1 and kind is web, abort with message that web ingestion is blocked. continue to step 3 if all gates pass.
• output: bool (gate pass/fail). if fail, error message with remediation instructions.

step 3: pre-flight size checks

• input: target path (local only), CAIRN_MAX_INGEST_FILES, CAIRN_MAX_INGEST_BYTES, optional force flag (cli only, mcp does not support)
• action: walk target directory (if code kind) and collect file count + total byte size. if count exceeds CAIRN_MAX_INGEST_FILES and force is not set, abort with message listing the count and the limit. if bytes exceed CAIRN_MAX_INGEST_BYTES and force is not set, abort with message listing the byte size and the limit. force=true bypasses both checks (cli only; mcp ingest ops always respect limits).
• output: bool (pass/fail). if fail, error message with byte/file counts and remediation (raise limits, target a subset, use --force on cli).

step 4: chunk the source

• input: target path/url, kind (code/file/pdf/web), language (inferred for code, null otherwise)
• action: for code kind, walk directory respecting include/exclude globs, parse each supported language via tree-sitter (rust, python, javascript, go, solidity, etc.; unsupported formats chunked as raw text), extract code chunks with line boundaries and ast context. for file kind, read raw bytes and chunk by semantic breaks (paragraphs, newlines, size heuristics). for pdf, extract text per page or per section if metadata available, then chunk. for web, fetch url (fails if CAIRN_OFFLINE=1), strip html, chunk text. all chunks tagged with source id, file path, line range (if code), and raw content.
• output: array of chunk objects: { id, sourceId, filePath, lineStart, lineEnd, kind, content, language }. one sqlite transaction queues all chunks for embedding and fts5 indexing.

step 5: embed chunks (call embedding runtime)

• input: chunk array from step 4, embedding model name/path
• action: batch chunk contents into requests to embedding runtime (ollama nomic-embed-text or embedded node-llama-cpp). requests made sequentially or batched depending on runtime (ollama batches in its own thread, embedded batches in process). each chunk gets a 768-dim float32 embedding. on network timeout (ollama), retry up to 2 times with exponential backoff, then fail and roll back transaction. on embedded runtime, cpu-only flag respected; first run downloads model unless CAIRN_OFFLINE=1.
• output: array of embeddings (one per chunk), shape (count, 768). stored in chunks_vec table (sqlite-vec fts + vector similarity).

step 6: extract entities and edges from markdown/pdf/web (llm doc-extraction, optional)

• input: chunks with kind=pdf/web/file (code chunks are ast-parsed in step 4, not llm-extracted), chat model name, CAIRN_DEBUG_DOC flag
• action: for each document (file or web page), concatenate its chunks up to token limit (~4k tokens), send to chat model (qwen3-0.6b-q8 via ollama or embedded) with structured prompt requesting concepts, edges, and tags in json-schema format. llm emits json with shape: { concepts: [{ name, kind, description }], edges: [{ from, to, relation }], tags: [{ entity, tag }] }. json is schema-validated (ollama format or llama.cpp grammar enforces validity). if CAIRN_DEBUG_DOC=1, log per-doc yields (concept count, edge count). on llm error or invalid json despite schema, skip document and log warning (continue with next).
• output: concept/edge/tag records bound to source id. no re-extraction on refresh (hash-gated: if doc content hash unchanged, concepts/edges/tags from prior index reused).

step 7: extract entities and edges from code (ast-based, tree-sitter)

• input: code chunks from step 4 with language set and ast parse trees
• action: walk ast per language (rust: functions, structs, traits, impl blocks; python: functions, classes, methods; javascript: functions, classes; go: functions, structs, methods; solidity: contracts, functions, events). emit entity records (name, kind, scope, file, line range). walk ast for calls, dependency references, imports; emit edge records (from_entity, to_entity, relation='calls'|'depends_on'|'imports'|'implements'). build per-source parse graph. on refresh, drop and rebuild parse edges for that source only (doc-extracted concepts/edges/tags remain unless doc content hash changed).
• output: entity records: { id, sourceId, name, kind, scope, filePath, lineStart, lineEnd }. edge records: { fromEntityId, toEntityId, relation, confidence }. all written to sqlite in the ingest transaction.

step 8: embed entities (call embedding runtime)

• input: entity name + description, embedding model
• action: for each entity, embed its name and description concatenated (or name only if description is empty). batch and call embedding runtime same as step 5. store in entities_vec table.
• output: array of entity embeddings (768-dim).

step 9: index full-text search (fts5)

• input: chunk and entity records with content and names
• action: insert all chunks and entities into fts5 virtual tables (chunks_fts, entities_fts). fts5 tokenizes and indexes; no additional work needed in code.
• output: fts5 indices ready for prefix/phrase search.

step 10: commit transaction and finalize

• input: all tables populated (chunks, chunks_vec, chunks_fts, entities, entities_vec, entities_fts, edges, entity_tags, source_links)
• action: commit sqlite transaction. update sources table with source metadata (label, kind, uri, file count, ingest timestamp, content hash for doc-extraction gating). if a source with same uri and label already exists, soft-delete the old version (mark deleted=1, keep for rollback) and insert new version (or update if refresh). fk cascades and triggers auto-clean stale entities/edges/tags from the old source.
• output: source id (integer), ingestion success signal. return to user.

query workflows

search (hybrid chunk search)

• input: query text, k (result count, default 5), optional kind/source/tag filters
• action: (a) fts5 search on chunks_fts for prefix/phrase match, collect top-k fts scores. (b) embed query text (same model as chunks), vector search chunks_vec for cosine sim top-k. (c) reciprocal rank fusion (rrf) merge fts + vec rankings. (d) if filters set (kind/source/tag), post-filter merged results by those criteria. (e) return top-k merged results with content, file path, line range, source label, confidence score.
• output: array of chunks: { id, content, filePath, lineStart, lineEnd, sourceLabel, score }.

graph (entity-level retrieval)

• input: query text xor entity_id, k (default 5), optional tag filter
• action: if query set, embed query, vector search entities_vec for top-k entity matches (no fts on entities; vec only). if entity_id set, skip search and use that entity directly. fetch entity by id and recurse: (a) all entities connected by 1-hop edges (in/out degree). (b) if k > 1, fetch k-1 adjacent edges (highest confidence first). (c) if tag filter set, keep only entities with that tag. return entity details + immediate neighbors.
• output: entity record with adjacent edge graph: { id, name, kind, filePath, description, edges: [{ toEntity, relation, confidence }] }.

ask (search + entity context, composed)

• input: query text, k (chunk results, default 5), kind/source/tag filters, maxEntitiesPerHit (default 3), maxEdgesPerEntity (default 2)
• action: run search (step above, get top-k chunks). for each chunk, find all entities mentioned in that chunk's file (by line range overlap or file match). fetch each entity, then fetch maxEdgesPerEntity edges from each entity (1-hop). return chunks + per-chunk entity context. expensive but one round trip.
• output: array of chunks with entity context: { chunk: {...}, entities: [{ id, name, kind, edges: [...] }] }.

path (shortest path via bfs)

• input: from entity (id or name), to entity (id or name), maxDepth (default 10), directed (default false)
• action: resolve entity names to ids (search if name passed). bfs traversal: layer 0 is from-entity. layer 1 fetches all adjacent entities (edges pointing in/out, or in-only if directed=true). layer N fetches all adjacent to layer N-1 entities, excluding visited nodes. one sql query per layer (not per node). stop when to-entity found or maxDepth exceeded. return shortest path edges.
• output: array of entity id path: [id1, id2, ..., idN], plus array of edges connecting them. if no path found, empty array.

tags (discover tag vocabulary)

• input: none (or optional kind filter)
• action: group by tag in entity_tags table, count entities per tag, optionally filter by entity kind. return sorted by count descending.
• output: array of tags with counts: [{ tag, count }, ...].

mcp server mode

startup

• input: environment variables set (runtime, offline, allowed-roots, size caps, debug flags)
• action: invoke cairn-mcp binary. initialize cairn instance. expose stdio jsonrpc 2.0 interface with tools: search, list, graph, ask, path, tags, add, refresh. log startup warning if CAIRN_ALLOWED_ROOTS unset.
• output: listening on stdin/stdout for mcp client (claude desktop, opcode, etc.). each tool call validated: add and refresh checked against allowed-roots and size caps; mutating ops remove, link, unlink, reindex not exposed (cli only).

cli mode

example invocations

• cairn add ./src --label my-project , ingest code directory.
• cairn add https://example.com/docs --label docs-web , ingest web page (blocked if CAIRN_OFFLINE=1).
• cairn search "how does chunking work" -k 5 , hybrid search, return top-5 chunks.
• cairn graph "fee invariant" --tag invariant , find entities tagged invariant matching query.
• cairn ask "what mitigates pool squatting" --tag attack , search + entity context, filtered by attack tag.
• cairn path 1:engine.rs:swap 1:math.rs:calc_swap_fee , shortest path between two named entities.
• cairn tags , list all tags and counts.
• cairn refresh my-project , re-index by label (or use source id or uri).
• cairn list --kind code , list indexed sources