monitoring-operations

SKILL.md

OCI Monitoring and Observability - Expert Knowledge

🏗️ Use OCI Landing Zone Terraform Modules

Don't reinvent the wheel. Use oracle-terraform-modules/landing-zone for observability stack.

Landing Zone solves:

❌ Bad Practice #10: No logging, monitoring, notifications (Landing Zone deploys complete observability)

❌ Bad Practice #7: Limited security services (Landing Zone integrates Cloud Guard, VSS, OSMS)

This skill provides: Metrics, alarms, and troubleshooting for monitoring deployed WITHIN a Landing Zone.

⚠️ OCI CLI/API Knowledge Gap

You don't know OCI CLI commands or OCI API structure.
Your training data has limited and outdated knowledge of:

OCI CLI syntax and parameters (updates monthly)

OCI API endpoints and request/response formats

Monitoring service CLI operations (oci monitoring alarm, oci monitoring metric)

Metric namespaces and MQL (Monitoring Query Language)

Latest Logging and Service Connector features

When OCI operations are needed:

Use exact CLI commands from this skill's references

Do NOT guess metric namespace names

Do NOT assume AWS CloudWatch patterns work in OCI

Load reference files for detailed MQL documentation

What you DO know:

General observability concepts

Alerting and threshold design principles

Log aggregation patterns

This skill bridges the gap by providing current OCI-specific monitoring patterns and gotchas.

NEVER Do This

❌ NEVER assume metrics are instant (10-15 minute lag)

Metrics published every 1-5 minutes

Processing delay: 5-10 minutes

Total lag: 10-15 minutes from event to visible metric

Don't debug "missing metrics" within first 15 minutes of resource creation

❌ NEVER use = for alarm thresholds with sparse metrics

# WRONG - alarm never fires if metric has gaps
MetricName[1m].mean() = 0

# RIGHT - handle missing data
MetricName[1m]{dataMissing=zero}.mean() > 0

❌ NEVER forget metric dimensions (causes "no data")

# WRONG - missing required dimension
CPUUtilization[1m].mean()

# RIGHT - include resourceId dimension
CPUUtilization[1m]{resourceId="<instance-ocid>"}.mean()

❌ NEVER set alarm thresholds without trigger delay (alert fatigue)

# BAD - fires on every CPU spike
CPUUtilization[1m].mean() > 80

# BETTER - sustained high CPU
CPUUtilization[5m].mean() > 80
Trigger delay: 5 minutes (fires after 5 consecutive breaches)

❌ NEVER create alarms without notification channels

# WRONG - alarm fires but nobody knows
oci monitoring alarm create ... --destinations '[]'

# RIGHT - always link to notification topic
oci monitoring alarm create ... --destinations '["<notification-topic-ocid>"]'

Cost impact: Undetected outages cost $5,000-50,000/hour in production

❌ NEVER ignore Cloud Guard findings (security audit failure)

Cloud Guard detects misconfigurations BEFORE they become incidents

Integrate Cloud Guard → Notifications → Email/Slack/PagerDuty

Cost impact: $100,000+ per security breach vs $0 for proactive remediation

Metric Namespace Gotchas

OCI Metrics Use Service-Specific Namespaces:

Service
Namespace
Example Metric

Compute
oci_computeagent
CPUUtilization, MemoryUtilization

Autonomous DB
oci_autonomous_database
CpuUtilization, StorageUtilization

Load Balancer
oci_lbaas
HttpRequests, UnHealthyBackendServers

Object Storage
oci_objectstorage
ObjectCount, BytesUploaded

Common Mistake: Using wrong namespace (oci_compute vs oci_computeagent)

Alarm Missing Data Handling

Setting
Behavior
Use When

treatMissingDataAsBreaching
Alarm fires if no data
Critical services (outage = breach)

treatMissingDataAsNotBreaching
Alarm silent if no data
Optional monitoring

{dataMissing=zero}
Treat missing as 0
Counters (requests/sec)

Log Collection Common Gaps

Problem: Logs not showing in Log Analytics

Logs not appearing?
├─ Is log enabled on resource?
│  └─ Compute: oci-compute-agent must be running
│  └─ Function: Logging enabled in function config
│
├─ Is Service Connector configured?
│  └─ Source: Log Group → Target: Log Analytics
│  └─ Check: Service Connector status = ACTIVE
│
├─ IAM policy for Service Connector?
│  └─ "Allow any-user to use log-content in tenancy"
│  └─ "Allow service loganalytics to READ logcontent in tenancy"
│
└─ 10-15 minute ingestion lag?
   └─ Wait before debugging

Metric Query Optimization

Expensive (slow):

# Queries ALL instances
CPUUtilization[1m].mean()

Optimized (filter by dimension):

# Query specific instance
CPUUtilization[1m]{resourceId='<instance-ocid>'}.mean()

Cost: Queries free, but rate limited (1000 req/min)

Progressive Loading References

OCI Monitoring Reference (Official Oracle Documentation)

WHEN TO LOAD oci-monitoring-reference.md:

Need comprehensive list of all OCI service metrics

Understanding MQL (Monitoring Query Language) in depth

Implementing complex alarm conditions and composites

Need official Oracle guidance on Logging and Service Connector

Setting up Log Analytics and APM integration

Do NOT load for:

Quick alarm setup (examples in this skill)

Common metric patterns (tables above)

Troubleshooting decision trees (covered above)

When to Use This Skill

Alarms: threshold configuration, missing data handling, trigger delay

Troubleshooting: metrics not showing, alarms not firing, namespace errors

Log collection: Service Connector, IAM policies, missing logs

Performance: query optimization, dimension filtering