beyla

SKILL.md

Grafana Beyla

Docs: https://grafana.com/docs/beyla/latest/

Zero-code HTTP / gRPC / DB instrumentation via eBPF. Emits OTLP traces + Prometheus metrics.

Prerequisites

Linux kernel 5.8+ with BTF enabled (ls /sys/kernel/btf/vmlinux must exist)

Root or CAP_SYS_ADMIN (or privileged: true in Kubernetes + hostPID: true)

x86_64 or ARM64

An OTLP receiver (Tempo, Alloy, OTel Collector) reachable from Beyla

Common Workflows

1. Instrument a single binary with Docker
# 1. Run Beyla against the app's port (the app must already be running, listening on 8080)
docker run --privileged --pid=host \
  -v /sys/kernel/debug:/sys/kernel/debug:ro \
  -e BEYLA_OPEN_PORT=8080 \
  -e BEYLA_PROMETHEUS_PORT=8999 \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
  -p 8999:8999 \
  grafana/beyla

# 2. Generate some traffic
curl http://localhost:8080/ ; curl http://localhost:8080/api/users/42

# 3. Verify Beyla emitted metrics — should list http_server_request_duration_seconds + counters
curl -s http://localhost:8999/metrics | grep -E '^http_(server|client)_request_duration'

# 4. Verify traces — in Grafana Explore on Tempo, search by service.name (default = process name)
#    Or: query Tempo's search API for spans with service.name="<app>"

2. Deploy as a cluster-wide DaemonSet

Full DaemonSet + RBAC YAML lives in references/kubernetes.md. After applying:

# 1. Verify DaemonSet rollout
kubectl -n monitoring rollout status ds/beyla

# 2. Verify pods are Running, one per node
kubectl -n monitoring get pods -l app=beyla -o wide

# 3. Verify eBPF probes attached (no errors mentioning BTF or "permission denied")
kubectl -n monitoring logs ds/beyla --tail=50 | grep -Ei 'error|fail|btf' || echo "clean"

# 4. Verify telemetry is flowing — check the Tempo/Alloy receiver for spans from the cluster
#    Or scrape one pod directly:
kubectl -n monitoring port-forward ds/beyla 8999:8999 &
curl -s localhost:8999/metrics | head

3. Send to Grafana Cloud via Alloy

# beyla-config.yml
otel_traces_export:  { endpoint: http://alloy:4318 }
otel_metrics_export: { endpoint: http://alloy:4318 }

# Verify Alloy is forwarding — check Alloy UI (localhost:12345) for the
# otelcol.receiver.otlp.beyla component showing received spans/metrics > 0.

Full Alloy + Beyla YAML: references/config.md.

Troubleshooting

failed to load BPF object → kernel < 5.8 or BTF missing; check /sys/kernel/btf/vmlinux

No spans in Tempo, but Prometheus metrics show → check OTEL_EXPORTER_OTLP_ENDPOINT, protocol (http vs grpc), and ports (4318 http / 4317 grpc)

HTTP route cardinality explosion → set routes.unmatched: heuristic and add patterns (see references/config.md)

Pod restarts with CrashLoopBackOff → likely missing hostPID: true or privileged: true / required capabilities

Resources

Beyla docs

Beyla GitHub

references/config.md — full config, env vars, samplers, routes decorator, generated metrics table, runtime matrix

references/kubernetes.md — DaemonSet + RBAC + discovery filters + Helm