google-agents-cli-observability

SKILL.md

Set up tracing, logging, and monitoring for deployed ADK agents across Cloud Trace, BigQuery, and third-party platforms.

Four observability tiers: Cloud Trace (always enabled, distributed tracing), Prompt-Response Logging (GenAI interactions to GCS/BigQuery), BigQuery Agent Analytics (structured agent events), and third-party integrations (AgentOps, Phoenix, MLflow, Weave, Freeplay, and others)

For Agent Runtime deployments, run agents-cli infra single-project before first deploy to provision Terraform-managed infrastructure (service account, GCS bucket, BigQuery dataset); post-deployment setup requires manual IAM and env var configuration

Cloud Trace works out of the box with OpenTelemetry spans tracking invocation flow, LLM calls, and tool execution; accessible via Cloud Console Trace explorer

Prompt-response logging is privacy-preserving by default (metadata only via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT); disabled locally unless LOGS_BUCKET_NAME is set

Includes troubleshooting guide covering missing traces, privacy misconfiguration, BigQuery setup, and cost optimization strategies

Observability Guide

Cloud Trace works out of the box — no infrastructure needed. Prompt-response logging and BigQuery Agent Analytics require Terraform-provisioned infrastructure (service account, GCS bucket, BigQuery dataset). Run agents-cli infra single-project --apply --project PROJECT_ID to provision these resources. Go projects get the BigQuery telemetry stack too; the GCS completion upload behind prompt-response logging and the BigQuery Agent Analytics plugin are Python only. See references/cloud-trace-and-logging.md for details, env vars, and verification commands. If your project isn't scaffolded yet, see /google-agents-cli-scaffold first.

Deployment order

Do you want Terraform-managed observability?
Action

No
Run agents-cli deploy directly (works for all targets).

Yes
Run agents-cli infra single-project --apply first, then agents-cli deploy.

If you already ran agents-cli deploy imperatively:
Do not apply Terraform afterward. For Agent Runtime, Terraform owns the whole engine (service account, deployment spec, env vars), so a later apply can't be reconciled without taking over the resource.
Either delete the deployment and start over, or keep the SDK-deployed instance — skip infra single-project and set the observability env vars by re-running agents-cli deploy --update-env-vars "KEY=VALUE,..."; deploy matches the existing Reasoning Engine by display name and updates it in place, preserving env vars set outside the deploy. You must also grant its service account the telemetry IAM roles the Terraform module would otherwise provision: roles/storage.admin (write completions to the logs bucket), roles/logging.logWriter, roles/cloudtrace.agent, plus roles/bigquery.dataOwner + roles/bigquery.jobUser when scaffolded with --bq-analytics. The full set lives in deployment/terraform/single-project/iam.tf (from app_sa_roles) and telemetry.tf. Terraform-managed env vars aren't available in this mode.

Reference Files