Ganglion , Operator's Manual

intent

ganglion is a domain-specific execution engine for bittensor subnet mining. it orchestrates autonomous mining agents searching for optimal model configurations via a pipeline framework. use it to scaffold projects, run mining pipelines (full or staged), query cross-run knowledge, mutate components at runtime, and coordinate multi-bot mining via shared knowledge. the engine exposes three interfaces: cli (local mode), http bridge api (remote mode), and mcp server (for external agents). ganglion is search infrastructure. it handles orchestration, dependency resolution, retry logic, and knowledge injection. you define what good looks like in config.py.

inputs

required binaries:

• python >= 3.11
• ganglion (installed via pip install ganglion)

required environment variables:

• LLM_PROVIDER_API_KEY: api key for the llm runtime (e.g., openai sk-xxx, anthropic key, etc.). required for pipeline execution.

optional environment variables for mode detection:

• GANGLION_PROJECT: if set, forces local mode. path to a ganglion project directory.
• GANGLION_URL: if set, forces remote mode. http endpoint of a running ganglion http bridge (e.g., http://127.0.0.1:8899).

project directory structure: the target project must contain:

• config.py: defines subnet_config (SubnetConfig) and pipeline (PipelineDef). see references/configuration.md for schema.
• tools/: directory of tool definitions (optional but recommended).
• agents/: directory of agent definitions (optional but recommended).
• skill/: directory of skill definitions (optional).

external connections (for mcp integration):

• mcp client mode: external mcp servers (stdio or sse transport). configured in config.py or dynamically via /v1/mcp/servers endpoint.
• mcp server mode: ganglion exposes itself as an mcp server. clients connect via stdio (claude desktop, claude code) or sse (http-based clients).

api authentication (multi-role mcp serving only):

• bearer tokens per role, passed in Authorization: Bearer <token> header for sse transport. defined in roles.json if using ganglion mcp-serve --roles ./roles.json.

procedure

1. mode detection

input: environment state (GANGLION_PROJECT, GANGLION_URL, current working directory)

action: determine execution mode.

if [ -n "$GANGLION_PROJECT" ] || [ -z "$GANGLION_URL" ]; then
  MODE="local"
else
  MODE="remote"
fi

output: MODE is either "local" or "remote". all subsequent commands depend on this choice.

2. scaffold a new project (local mode only)

input: target directory path, subnet identifier (e.g., sn9), netuid (e.g., 9)

action:

ganglion init ./my-subnet --subnet sn9 --netuid 9

this creates:

• config.py with placeholder subnet_config and pipeline definitions
• tools/ directory (empty)
• agents/ directory (empty)
• skill/ directory (empty)

output: new project directory tree ready for editing.

3. edit project configuration

input: config.py file in project root

action: define subnet_config (SubnetConfig instance) and pipeline (PipelineDef instance). include any static mcp client configs here if needed (MCPClientConfig objects).

output: valid config.py that passes validation. test with ganglion pipeline ./my-subnet to catch syntax/dag errors early.

4a. observe state (local mode)

input: project directory, query parameters (optional)

actions (pick one per query):

# pipeline structure
ganglion pipeline ./my-subnet

# registered tools
ganglion tools ./my-subnet

# registered agents
ganglion agents ./my-subnet

# cross-run knowledge (filter by capability and bot_id)
ganglion knowledge ./my-subnet --bot-id alpha --capability training --max-entries 10

# overall status
ganglion status ./my-subnet

# metrics and run history
ganglion metrics ./my-subnet

output: json or table (depending on command) showing current framework state.

4b. observe state (remote mode)

input: GANGLION_URL env var, http client (curl), query parameters (optional)

actions (pick one per query):

# pipeline structure
curl -s "$GANGLION_URL/v1/pipeline" | jq .data

# registered tools
curl -s "$GANGLION_URL/v1/tools" | jq .data

# registered agents
curl -s "$GANGLION_URL/v1/agents" | jq .data

# cross-run knowledge
curl -s "$GANGLION_URL/v1/knowledge?bot_id=alpha&capability=training&max_entries=10" | jq .data

# overall status
curl -s "$GANGLION_URL/v1/status" | jq .data

# metrics and run history
curl -s "$GANGLION_URL/v1/metrics" | jq .data

# check health (no data envelope)
curl -s "$GANGLION_URL/healthz" | jq .

output: json payload (extract with jq .data for non-health endpoints).

5a. start local mode (if GANGLION_PROJECT is not set)

input: project directory path

action:

export GANGLION_PROJECT=./my-subnet
ganglion status $GANGLION_PROJECT

subsequent commands can now omit the project path and reference $GANGLION_PROJECT directly.

output: env var set; ready for local commands.

5b. start remote mode (http bridge)

input: project directory path, bot identifier (string), port (int, default 8899)

action:

ganglion serve ./my-subnet --bot-id alpha --port 8899
export GANGLION_URL=http://127.0.0.1:8899

this starts an http server. the --bot-id value is used to namespace knowledge and usage metrics in multi-bot setups.

output: http server listening on specified port. GANGLION_URL env var set. ready for remote api calls.

6a. execute pipeline (local mode)

input: project directory, optional stage name, optional runtime overrides (json)

actions (pick one):

# run full pipeline (all stages in dependency order)
ganglion run ./my-subnet

# run single stage
ganglion run ./my-subnet --stage plan

# run with runtime config overrides
ganglion run ./my-subnet --overrides '{"target_metric":"accuracy","max_iterations":20}'

the orchestrator:

• validates the pipeline dag (detects cycles, missing dependencies).
• injects relevant prior knowledge into agent prompts.
• executes stages in dependency order, applying stage-specific retry policies.
• records all outcomes and tool calls in run history.
• appends discovered patterns and antipatterns to the knowledge store.

output: run completes with exit code 0 (success) or non-zero (failure). check ganglion status or ganglion metrics for detailed results.

6b. execute pipeline (remote mode)

input: GANGLION_URL, http client (curl), optional json payload with overrides

actions (pick one):

# run full pipeline
curl -s -X POST "$GANGLION_URL/v1/run/pipeline" \
  -H "Content-Type: application/json" \
  -d '{}' | jq .data

# run single stage
curl -s -X POST "$GANGLION_URL/v1/run/stage/plan" \
  -H "Content-Type: application/json" \
  -d '{}' | jq .data

# run with overrides
curl -s -X POST "$GANGLION_URL/v1/run/pipeline" \
  -H "Content-Type: application/json" \
  -d '{"target_metric":"accuracy","max_iterations":20}' | jq .data

output: json payload containing run id, status, and initial results. long-running executions return immediately with a run id; poll /v1/runs/<run_id> to monitor progress.

7a. mutate framework at runtime (local mode - limited)

input: project directory

action: editing config.py directly, then reloading. local mode does not support dynamic mutations without process restart. for iterative mutation workflows, switch to remote mode.

output: changes take effect on next ganglion run.

7b. mutate framework at runtime (remote mode - full)

input: GANGLION_URL, http client (curl), mutation payload (json)

mutations available:

register a new tool:

curl -s -X POST "$GANGLION_URL/v1/tools" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my_tool",
    "code": "<python code>",
    "category": "training",
    "description": "optional description"
  }' | jq .data

register a new agent:

curl -s -X POST "$GANGLION_URL/v1/agents" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "MyAgent",
    "model": "gpt-4",
    "system_prompt": "you are a mining agent"
  }' | jq .data

patch pipeline (add, remove, or update stages):

curl -s -X PATCH "$GANGLION_URL/v1/pipeline" \
  -H "Content-Type: application/json" \
  -d '{
    "operations": [
      {"op": "add_stage", "stage": {"name": "validate", "agent": "Validator", "depends_on": ["train"]}},
      {"op": "update_stage", "name": "plan", "updates": {"agent": "NewAgent"}},
      {"op": "remove_stage", "name": "old_stage"}
    ]
  }' | jq .data

update an agent prompt:

curl -s -X POST "$GANGLION_URL/v1/prompts" \
  -H "Content-Type: application/json" \
  -d '{
    "agent": "Planner",
    "system_prompt": "new prompt text"
  }' | jq .data

swap a retry policy for a stage:

curl -s -X POST "$GANGLION_URL/v1/policies" \
  -H "Content-Type: application/json" \
  -d '{
    "stage": "plan",
    "policy": "SN50_PRESET"
  }' | jq .data

all mutations are validated, audited, and reversible.

output: json payload with updated framework state. check audit log via /v1/audit.

8. query and inject knowledge

input: project directory or GANGLION_URL, capability filter (string, optional), bot_id filter (string, optional)

action (local):

ganglion knowledge ./my-subnet --capability training --bot-id alpha --max-entries 20

action (remote):

curl -s "$GANGLION_URL/v1/knowledge?capability=training&bot_id=alpha&max_entries=20" | jq .data

the knowledge store holds:

• patterns: configurations, hyperparameters, and techniques that improved the target metric
• antipatterns: what failed and should be avoided
• per-bot namespacing: each --bot-id maintains its own knowledge pool in multi-bot setups

knowledge is automatically injected into agent prompts during pipeline execution. query it to understand what the system has learned.

output: array of knowledge entries with timestamps, bot_id, capability, and content.

9. rollback a mutation

input: GANGLION_URL, rollback target (latest or audit log index)

actions:

# undo the last mutation
curl -s -X POST "$GANGLION_URL/v1/rollback/last" | jq .data

# undo a specific mutation by audit log index
curl -s -X POST "$GANGLION_URL/v1/rollback/0" | jq .data

# undo all mutations (rollback to index 0)
curl -s -X POST "$GANGLION_URL/v1/rollback/0" | jq .data

every mutation records its reverse operation in the audit log. rollback is idempotent.

output: framework state reverted to target point. audit log updated with rollback entry.

10a. mcp client mode: connect external tools

external mcp servers expose tools (e.g., weather, database, compute) to ganglion agents.

static configuration (config.py):

from ganglion.mcp.config import MCPClientConfig

mcp_clients = [
  MCPClientConfig(
    name="weather",
    transport="stdio",
    command=["python", "-m", "weather_server"],
    category="external_tools",
    timeout=30
  ),
  MCPClientConfig(
    name="compute_api",
    transport="sse",
    url="http://compute-server:9000/sse",
    tool_prefix="compute_",
    category="infrastructure"
  )
]

dynamic registration (remote mode):

curl -s -X POST "$GANGLION_URL/v1/mcp/servers" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "weather",
    "transport": "stdio",
    "command": ["python", "-m", "weather_server"],
    "category": "external_tools",
    "timeout": 30
  }' | jq .data

check connected servers:

curl -s "$GANGLION_URL/v1/mcp" | jq .data

reconnect a server:

curl -s -X POST "$GANGLION_URL/v1/mcp/servers/weather/reconnect" | jq .data

disconnect a server:

curl -s -X DELETE "$GANGLION_URL/v1/mcp/servers/weather" | jq .data

tools from mcp servers appear in the tool registry and are available to agents with no additional configuration.

output: tools from external mcp servers registered and ready for use. ganglion tools / /v1/tools includes them.

10b. mcp server mode: expose ganglion tools to external agents

ganglion itself can run as an mcp server so external agents (claude desktop, cursor, openai code interpreter, openclaw) can invoke ganglion tools.

start ganglion mcp server (stdio transport):

ganglion mcp-serve ./my-subnet --transport stdio

use this with claude desktop or claude code. ganglion tools appear in the claude tool palette.

start ganglion mcp server (sse transport):

ganglion mcp-serve ./my-subnet --transport sse --mcp-port 8900

use this with http-based mcp clients (generic clients, cursor, windsurf). endpoints:

• GET http://127.0.0.1:8900/sse , sse stream (tool list + results)
• POST http://127.0.0.1:8900/messages , send tool calls

multi-role mcp serving (access control):

ganglion mcp-serve ./my-subnet --roles ./roles.json

roles.json defines separate mcp server instances with different tool access and auth:

[
  {"name": "admin",    "categories": null,                        "token": "admin-xyz",    "port": 8901},
  {"name": "worker",   "categories": ["observation","execution"], "token": "worker-abc",   "port": 8902},
  {"name": "observer", "categories": ["observation"],             "token": "observer-def", "port": 8903}
]

null categories = all tools. each role gets a separate port (sse) and bearer token. at most one role can use stdio transport.

connect an external mcp client to ganglion's sse server:

set up client config (cursor, windsurf, custom):

• server url: http://127.0.0.1:8900/sse
• messages endpoint: http://127.0.0.1:8900/messages
• transport: sse
• auth header (if using roles): Authorization: Bearer worker-abc

usage tracking (multi-role only):

each call to a ganglion tool is logged to .ganglion/usage.db per bot_id (tool name, success/failure, timestamp, duration). query via:

curl -s "$GANGLION_URL/v1/usage" | jq .data
curl -s "$GANGLION_URL/v1/usage?bot_id=alpha" | jq .data

output: ganglion tools exposed as mcp server. external agents can discover and invoke them.

11. multi-bot workflows

multiple openclaw or agent sessions share a knowledge pool and optionally contribute to shared metrics.

local mode (shared project directory):

# session 1
export GANGLION_PROJECT=./my-subnet
ganglion run ./my-subnet --bot-id alpha

# session 2 (different terminal)
export GANGLION_PROJECT=./my-subnet
ganglion run ./my-subnet --bot-id beta

discoveries by alpha are visible to beta via ganglion knowledge --bot-id beta --capability training.

remote mode (shared project, different ports):

# terminal 1
ganglion serve ./my-subnet --bot-id alpha --port 8899
export GANGLION_URL=http://127.0.0.1:8899

# terminal 2
ganglion serve ./my-subnet --bot-id beta --port 8900
export GANGLION_URL=http://127.0.0.1:8900

each server has its own http listener but shares the underlying project directory and knowledge pool.

output: multiple bots mining in parallel. cross-bot knowledge flows through the shared knowledge store. cooperation emerges from shared memory, not explicit coordination.

decision points

mode selection (step 1):

• if GANGLION_PROJECT env var is set, use local mode (cli commands).
• else if GANGLION_URL is set, use remote mode (http api).
• else default to local mode and require passing project path explicitly to each command.

mcp client config (step 10a):

• if mcp servers are required at startup, define them statically in config.py (MCPClientConfig list).
• if mcp servers may be added/removed at runtime, use remote mode and call /v1/mcp/servers endpoints to manage them dynamically.
• if using sse transport (http-based mcp server), provide url instead of command.
• if using stdio transport (local subprocess), provide command instead of url.

project configuration validation (step 3):

• if ganglion pipeline ./my-subnet reports PipelineValidationError, the dag contains cycles or missing dependencies. fix config.py and re-run validation.
• if ganglion run fails with FileNotFoundError: No config.py, verify the project path and that config.py exists in the root.

runtime mutations during execution (step 7b):

• if attempting to mutate while a pipeline run is in progress, the api returns ConcurrentMutationError. wait for the run to complete, then retry the mutation.

knowledge injection (step 8):

• if --bot-id is not specified in local mode or bot_id query param is not passed in remote mode, knowledge is not filtered by bot. all knowledge is visible.
• if --capability or capability param is not specified, knowledge is not filtered by capability. all knowledge is visible.

external agent access (step 10b):

• if using claude desktop or claude code, start ganglion mcp server with --transport stdio. ganglion tools appear natively in claude's tool palette.
• if using generic mcp clients (cursor, windsurf, custom), start with --transport sse and point the client at http://<host>:<port>/sse.
• if using multi-role setup, extract bearer token from roles.json and pass it in the Authorization: Bearer <token> header.

multi-bot coordination (step 11):

• if bots need to share knowledge, they must point to the same project directory (local mode) or the same ganglion http server (remote mode).
• each bot must have a unique --bot-id value to maintain separate knowledge namespaces and avoid collisions.

output contract

local mode execution (steps 4a, 5a, 6a):

• ganglion commands return json or tables to stdout and exit with status 0 (success) or non-zero (failure).
• errors include a traceback and error code (e.g., PipelineValidationError, FileNotFoundError).
• run results are persisted in .ganglion/runs/ directory in the project.
• knowledge entries are appended to .ganglion/knowledge.db in the project.

remote mode api responses (steps 4b, 5b, 6b, 7b, 8, 9, 10a, 10b):

• all non-health endpoints return {"data": <payload>} envelope on success. extract with jq .data.
• errors return {"detail": {"error": {"code": "ERROR_CODE", "message": "description"}}}.
• health endpoints (/healthz, /readyz) return raw json without envelope.
• http status codes: 200 (success), 400 (validation error), 409 (concurrent mutation), 500 (server error).

pipeline execution (local and remote):

• on success: framework state updated, run recorded in history, knowledge appended, exit code 0.
• on failure: stage failure logged with error details, retry policy applied (if configured), knowledge recorded as antipattern, exit code non-zero.

mutations (step 7b):

• on success: component registered/updated, audit log entry created, framework state updated, rollback data recorded.
• on failure: validation error returned, no state change, no audit entry.

mcp server startup (step 10b):

• on success: mcp server listening on specified port(s), tools enumerable via mcp protocol.
• on failure: port binding error or configuration error, server exits with non-zero code.

knowledge store (step 8):

• .ganglion/knowledge.db persists cross-run patterns and antipatterns.
• each entry has fields: timestamp (iso8601), bot_id (string), capability (string), pattern_type ("pattern" or "antipattern"), content (text), source_run_id (string).

audit log (step 7b, 9):

• .ganglion/audit.log records every mutation with timestamp, operation type, user/bot, payload, and rollback data.
• audit entries are immutable.

usage tracking (step 10b, multi-role only):

• .ganglion/usage.db records tool calls per bot_id: tool name, success/failure, timestamp, duration, bot_id.

outcome signal

successful project init (step 2):

• target directory now contains config.py, tools/, agents/, skill/ subdirectories.
• ganglion pipeline ./my-subnet runs without errors and outputs the pipeline dag.

successful local mode startup (step 5a):

• echo $GANGLION_PROJECT shows the project path.
• ganglion status $GANGLION_PROJECT returns framework state (no errors).

successful remote mode startup (step 5b):

• http server is listening (verify with curl -s "$GANGLION_URL/healthz").
• echo $GANGLION_URL shows the http endpoint.
• curl -s "$GANGLION_URL/v1/status" | jq .data returns framework state (no errors).

successful pipeline run (steps 6a, 6b):

• all stages execute in dependency order.
• run completes with exit code 0 (local) or status 200 (remote).
• ganglion metrics ./my-subnet shows updated run history, or /v1/metrics returns new run entry.
• .ganglion/runs/<run_id>/ directory contains execution logs and results (local mode).

successful mutation (step 7b):

• /v1/audit shows new mutation entry.
• subsequent ganglion tools or /v1/tools includes the new tool.
• /v1/rollback/last can undo the mutation.

successful knowledge query (step 8):

• ganglion knowledge ./my-subnet or curl -s "$GANGLION_URL/v1/knowledge" returns non-empty array of entries.
• entries include bot_id, capability, pattern_type, and content fields.

successful mcp integration (steps 10a, 10b):

• mcp client: curl -s "$GANGLION_URL/v1/mcp" lists connected servers (no errors).
• mcp server (stdio): claude code sees ganglion tools in its tool palette.
• mcp server (sse): external client connects to http://127.0.0.1:<port>/sse and receives tool list.

**successful multi-bot setup (