Game Ops Monitor

real-time game server JVM monitoring via scouter collector. query TPS, online users, JVM heap usage. detect performance alerts. supports batch queries across 7000+ processes. use when checking server health, monitoring throughput, or detecting memory pressure.

---

intent

monitor live game server performance metrics from scouter collector HTTP API. covers TPS (transactions per second), online player count, and JVM heap utilization. designed for ops teams managing large-scale game infrastructure (900+ machines, 7000+ processes). returns human-readable alerts and aggregated reports. run this skill when asked to check server health, compare performance across instances, detect bottlenecks, or generate performance snapshots for ops reviews.

---

inputs

environment variables (required)

• SCOUTER_COLLECTOR_URL: base URL of scouter collector API. format: http://<collector-ip>:6188. example: http://192.168.1.50:6188. no default. skill fails if unset.
• SCOUTER_OBJ_TYPE: object type filter for object queries. default: Java. set to filter by agent type if collector tracks non-JVM processes.

optional configuration (env vars or script constants)

• TPS_CRITICAL: threshold for critical TPS alert. default: 100. values below this trigger 🔴 alert.
• TPS_WARNING: threshold for warning TPS alert. default: 300. values between TPS_CRITICAL and TPS_WARNING trigger ⚠️ alert.
• HEAP_CRITICAL: critical heap utilization percentage. default: 90. values above this trigger 🔴 memory alert.
• HEAP_WARNING: warning heap utilization percentage. default: 75. values between HEAP_WARNING and HEAP_CRITICAL trigger ⚠️ memory alert.

external connection

• scouter collector API: HTTP API at ${SCOUTER_COLLECTOR_URL}/scouter/v1/. endpoints: /objects, /object/{objHash}/counter/{metric}, /status. collector must be running and reachable over HTTP. no authentication required (assumes collector on private/internal network). network timeout should be set to 10s per request; collector may be slow under heavy load.

upstream data source

scouter collector receives real-time metrics from java agents deployed on game servers. agents are loaded via -javaagent:scouter-agent.jar JVM flag. setup is out-of-scope for this skill; see setup section at end.

---

procedure

step 1: validate collector connectivity

inputs: SCOUTER_COLLECTOR_URL

action: send GET request to /status endpoint to verify collector is alive and responding.

response=$(curl -s -m 10 "http://${SCOUTER_COLLECTOR_URL}/scouter/v1/status")

outputs:

• on success: JSON response with "status":"ok". proceed to step 2.
• on failure (timeout, HTTP error, malformed response): log error message and exit. do not continue.

edge case: collector restarts frequently or runs low on memory. if 5 consecutive requests timeout over 1 minute, escalate to ops / collector admin.

---

step 2: fetch all monitored objects (game server instances)

inputs: SCOUTER_COLLECTOR_URL, SCOUTER_OBJ_TYPE

action: query /objects endpoint to list all registered game server processes.

objects=$(curl -s -m 10 "http://${SCOUTER_COLLECTOR_URL}/scouter/v1/objects?type=${SCOUTER_OBJ_TYPE}")

outputs: JSON array of objects. each object contains:

• objHash: hex string identifier for the process (used in subsequent counter queries)
• objName: human-readable server ID (e.g., xy_s11649)
• objType: agent type (e.g., Java)
• address: IP address of host running the process
• state: ACTIVE, INACTIVE, SHUTDOWN, etc.

edge case: empty result set means no game servers registered. may indicate agents not yet started or not registered with collector. log and return empty report.

edge case: result set contains 1000s of objects. consider implementing pagination or caching if queries become slow. current API does not support pagination; all results returned in one response.

---

step 3: validate and extract object hashes

inputs: objects JSON array from step 2

action: iterate over each object. for each, extract objHash field. validate that objHash is a valid hex string (uppercase or lowercase hex digits only). reject any objHash containing non-hex characters; this prevents shell injection if a malicious or compromised collector returns malformed data.

echo "$objects" | jq -r '.[] | .objHash' | while read objHash; do
  if ! echo "$objHash" | grep -qE '^[0-9a-fA-F]+$'; then
    echo "[WARN] objHash validation failed: $objHash ,  skipping"
    continue
  fi
  # proceed to step 4 for this objHash
done

outputs: list of validated hex objHashes, one per line. objNames corresponding to each hash (for reporting).

edge case: objHash validation fails. log warning and skip that object. continue with remaining objects.

---

step 4: fetch performance metrics for each object (parallel or sequential)

inputs: validated objHash, SCOUTER_COLLECTOR_URL

action: for each objHash, fetch 4 counter metrics in parallel (or sequential if parallelism unavailable):

1. TPS: GET /object/{objHash}/counter/tps
2. Active users (online count): GET /object/{objHash}/counter/active
3. Heap used: GET /object/{objHash}/counter/heap/used
4. Heap max: GET /object/{objHash}/counter/heap/max

each request uses 10s timeout.

tps=$(curl -s -m 10 "http://${SCOUTER_COLLECTOR_URL}/scouter/v1/object/${objHash}/counter/tps")
active=$(curl -s -m 10 "http://${SCOUTER_COLLECTOR_URL}/scouter/v1/object/${objHash}/counter/active")
heap_used=$(curl -s -m 10 "http://${SCOUTER_COLLECTOR_URL}/scouter/v1/object/${objHash}/counter/heap/used")
heap_max=$(curl -s -m 10 "http://${SCOUTER_COLLECTOR_URL}/scouter/v1/object/${objHash}/counter/heap/max")

outputs: 4 JSON objects with fields:

• objHash: the queried object hash
• counter: metric name (tps, active, heap/used, heap/max)
• value: numeric value (TPS in micro-ticks, active count as integer, bytes for heap)
• timestamp: Unix milliseconds when metric was sampled

edge case: individual counter request times out. mark metric as UNAVAILABLE for that object; do not fail the entire query. continue with other metrics.

edge case: counter endpoint returns error (e.g., 404, 500). log error. treat metric value as null; do not crash.

edge case: collector is rate-limiting requests (HTTP 429). backoff 5s and retry up to 2 times. if still rate-limited, report degraded performance and skip remaining objects.

---

step 5: transform and normalize metric values

inputs: raw metric values from step 4

action: parse JSON and apply transformations:

• TPS: divide raw value by 1000 and floor to integer. TPS is returned in micro-ticks; 1000 micro-ticks = 1 TPS.
• active: use as-is (already an integer user count).
• heap used and max: convert from bytes to GB (divide by 1024^3) for readability. calculate heap utilization percentage: (heap_used / heap_max) * 100, rounded to 1 decimal place.

tps_real=$(echo "$tps" | jq -r '.value / 1000 | floor')
active_real=$(echo "$active" | jq -r '.value')
heap_used_gb=$(echo "$heap_used" | jq -r '.value / 1024 / 1024 / 1024 | round')
heap_max_gb=$(echo "$heap_max" | jq -r '.value / 1024 / 1024 / 1024 | round')
heap_pct=$(echo "scale=1; ${heap_used} * 100 / ${heap_max}" | bc)

outputs: normalized metrics (TPS as integer, active as integer, heap as GB strings with 2 decimals, heap % as float with 1 decimal).

edge case: heap_used > heap_max (can occur during GC pause). cap percentage at 100% for display; do not error.

edge case: heap_max is 0 or not set. heap percentage becomes undefined. display as -- (not available). do not crash.

---

step 6: apply alert threshold logic and assign status codes

inputs: normalized metrics from step 5, alert thresholds (TPS_CRITICAL, TPS_WARNING, HEAP_CRITICAL, HEAP_WARNING)

action: for each server instance, evaluate alert conditions:

TPS status:

• if tps < TPS_CRITICAL (default 100): status = 🔴 critical
• else if tps < TPS_WARNING (default 300): status = ⚠️ warning
• else: status = ✅ ok

Heap status:

• if heap_pct > HEAP_CRITICAL (default 90): status = 🔴 critical
• else if heap_pct > HEAP_WARNING (default 75): status = ⚠️ warning
• else: status = ✅ ok

Online users status:

• if active < 10 and server state is ACTIVE: status = ⚠️ empty (server running but no players). note: this is informational, not a hard alert.
• else: status = ✅ ok

outputs: status assignments for TPS, heap, and online users for each server.

---

step 7: format output based on query scope

inputs: all metric data and statuses from steps 1-6, original user query

action: detect query intent and select output format:

query scope detection: scan the original user query for keywords:

• if query contains "TPS" or "吞吐" or "throughput": output TPS-focused report (see decision points section for logic).
• if query contains "在线" or "online" or "players": output active user report.
• if query contains "内存" or "heap" or "memory": output heap utilization report.
• if query contains server ID (e.g., xy_s11649): output single-server deep dive.
• if query contains "报表" or "report" or "summary": output full performance snapshot.
• default (no keywords matched): output full health dashboard.

format output as human-readable text with emoji status indicators, aligned columns, sorted by metric (TPS ascending), and section headers.

example output:

========== Game Server Health Dashboard ==========
Time: 2024-04-28 14:00:00 UTC
Total servers: 6,847
Healthy: 6,812 (99.5%)
Alerting: 35 (0.5%)

========== TPS Alerts ==========
🔴 Critical (<100 TPS):
  xy_s20133    89 TPS / 412 online / Heap 67% ✅
  xy_s20132   156 TPS / 891 online / Heap 45% ✅

⚠️ Warning (100-300 TPS):
  xy_s11647  287 TPS / 3,521 online / Heap 55% ✅

✅ OK (>300 TPS): 6,812 servers

========== Memory Alerts ==========
🔴 Critical (Heap >90%):
  xy_s20140  Heap 95% (3.80G/4.00G) / TPS 156
  xy_s20141  Heap 92% (3.68G/4.00G) / TPS 123

outputs: formatted text string ready for display or logging.

---

step 8: handle special workflows

inputs: formatted output from step 7, optional historical data for report generation

action: branch based on query scope (see decision points).

1. single-server query (e.g., "check xy_s11649"): output drill-down for that one server with all metrics, trends if available, and actionable recommendations.

2. performance report (e.g., "generate daily report"): aggregate statistics across all servers:

   • total count, healthy/alerting breakdown
   • TPS distribution histogram (buckets: >1000, 500-1000, 300-500, 100-300, <100)
   • memory alert breakdown
   • top 10 servers by TPS
   • top 10 servers by online users
   • list of servers requiring immediate action (critical alerts)

3. alert-only query (e.g., "which servers have TPS alerts"): output alert section only, omit healthy servers.

outputs: specialized formatted output per workflow.

---

decision points

decision: collector connectivity check (step 1)

• if collector responds with HTTP 200 and {"status":"ok"}: proceed to step 2.
• else if collector responds with HTTP 500 or 503: collector is overloaded or crashed. log error. optionally retry once after 5s backoff. if still failing, abort and return error message: "Collector unreachable. Contact ops team." do not attempt to use stale/cached data.
• else if request times out (no response after 10s): network issue or collector hung. log timeout. abort. return error message: "Collector timeout. Network or service issue."
• else if response is malformed JSON: log parse error. abort.

decision: object list empty (step 2)

• if objects array is empty: log warning. return empty report with message: "No game servers registered with collector. Check agent deployment." do not proceed further.
• if objects array contains objects: proceed to step 3.

decision: query scope detection (step 7)

• if query string contains "TPS" or "吞吐": output TPS-focused report. highlight critical and warning TPS servers. omit OK servers unless requested. include online and heap context.
• if query string contains "在线" or "online": output active user report. sort by player count descending. include TPS and heap context.
• if query string contains "内存" or "heap" or "memory": output heap utilization report. sort by heap % descending. highlight servers over warning threshold.
• if query string contains a server ID (pattern: [a-z]+_s\d+ or similar): output single-server report. fetch and display all metrics, optional recent trends if available.
• if query string contains "报表" or "report" or "summary": output full performance snapshot. include aggregate stats, distributions, top servers, and action items.
• else: output full health dashboard with all metrics and alerts.

decision: output filtering for non-alert queries

• if query requests specific metric (TPS/online/memory): display all servers for that metric, sorted. include servers with OK status (do not filter out).
• if query requests alerts only: display only servers with 🔴 or ⚠️ status. omit ✅ servers.
• else: display all servers (both alert and OK status).

decision: single-server query (not found)

• if user queries a server ID that does not exist in collector: return error message: "Server [ID] not found in collector. Check server name or wait for registration (agents may take 30s after startup to appear)." do not crash.

decision: performance report generation

• if user requests a report and current data is available: generate report from current metrics (step 7 output).
• if user requests daily/weekly report and historical data is not available: log note. generate report from current snapshot only. note: "Historical data not available. Report reflects current state only."

decision: error recovery

• if any step 2-6 encounters transient error (timeout, temp network issue): retry up to 2 times with 5s backoff. if still failing, continue with partial data and mark unavailable metrics as UNAVAILABLE in output.
• if any step 2-6 encounters permanent error (e.g., collector API changed, auth failure): log error and abort. return error message to user.

---

output contract

success case: stdout text format

output is a formatted text string containing:

• header: time, total server count, health summary (healthy/alerting counts)
• alert sections: grouped by metric (TPS, memory) and severity (critical/warning/ok)
• server entries: each line shows objName, metric values, and status emoji
• columns: aligned with padding for readability
• sort order: TPS alerts sorted ascending by TPS value; memory alerts sorted descending by heap %; online