Golang Benchmark

Intent

This skill covers the full performance measurement workflow in Go: write benchmarks, run them with statistical rigor, profile CPU/memory/goroutine behavior with pprof, compare results before/after using benchstat, detect regressions in CI pipelines, and correlate production metrics with code changes using Prometheus. Use this skill when you need to measure where time and memory are spent, prove that an optimization actually works, or investigate why production performance diverges from local benchmarks. For optimization patterns to apply after measurement, see samber/cc-skills-golang@golang-performance. For pprof setup on running services, see samber/cc-skills-golang@golang-troubleshooting.

Inputs

• Go toolchain: go binary (1.24+ preferred for b.Loop(), but 1.20+ supported)
• benchstat binary: golang.org/x/perf/cmd/benchstat@latest for statistical comparison
• Optional profiling tools: graphviz (for pprof web UI), curl (if pulling metrics from live services)
• CI environment: GitHub Actions, GitLab CI, or self-hosted runners with stable hardware for regression detection
• Prometheus setup (production only): Go runtime metrics endpoint at /metrics with prometheus/client_golang wired into your HTTP server
• Hardware context: CPU model, core count, OS, and architecture (captured in benchmark output; affects profile interpretation)

Procedure

Step 1: Write a Benchmark

Input: Source file with target function to measure (e.g., pkg/parser/parser.go). Output: Test file with benchmark function (e.g., pkg/parser/parser_test.go).

1. Create a function named BenchmarkXxx in a *_test.go file.
2. For Go 1.24+, use b.Loop() to wrap the code being measured. Setup code (loading fixtures, allocating buffers) goes outside the loop.
3. For Go <1.24, use the for i := 0; i < b.N; i++ pattern. Call b.ResetTimer() after expensive setup if needed.
4. To measure allocations, add b.ReportAllocs() in the function body or use the -benchmem flag at runtime.
5. For custom metrics (e.g., throughput), use b.ReportMetric(value, unit) inside the loop.
6. For parameterized benchmarks, use b.Run(name, func(b *testing.B) { ... }) to create sub-benchmarks with different inputs.

Example:

func BenchmarkParse(b *testing.B) {
    data := loadFixture("large.json")
    b.ReportAllocs()
    for b.Loop() {
        Parse(data)
    }
}

func BenchmarkParseVariations(b *testing.B) {
    for _, size := range []int{64, 256, 4096} {
        b.Run(fmt.Sprintf("size=%d", size), func(b *testing.B) {
            data := make([]byte, size)
            for b.Loop() {
                Encode(data)
            }
        })
    }
}

Step 2: Run Benchmarks Locally

Input: Benchmark function(s) from Step 1, target package path. Output: Plain text benchmark results saved to file (e.g., bench.txt).

1. Run go test -bench=<BenchmarkName or .> -benchmem -count=10 ./pkg/... | tee bench.txt to execute all matching benchmarks 10 times.
2. Use -benchtime=3s (or higher) if the default 1 second per benchmark is too short for statistical significance.
3. Use -cpu=1,2,4,8 to measure how the code scales with GOMAXPROCS if concurrency is relevant.
4. Run on a quiet machine: close other apps, disable frequency scaling if possible, and verify CPU governor is set to performance mode (on Linux: grep cpu /proc/cpuinfo | head -1 and check scaling frequency).
5. Capture the output to a file for later comparison with benchstat.

Example output:

BenchmarkParse-8                  5000000       230.5 ns/op        128 B/op        2 allocs/op
BenchmarkParseVariations/size=64-8  10000000       105.2 ns/op       64 B/op         1 allocs/op
BenchmarkParseVariations/size=256-8  2000000       598.3 ns/op       256 B/op        1 allocs/op

Step 3: Profile Benchmarks (Optional but Recommended)

Input: Benchmark function(s), target package path, desired profile type (CPU, memory, trace). Output: Profile file (.prof or .out extension).

1. For CPU profiling, run go test -bench=<BenchmarkName> -cpuprofile=cpu.prof ./pkg/....
2. For memory profiling (heap allocations), run go test -bench=<BenchmarkName> -memprofile=mem.prof ./pkg/....
3. For execution tracing (goroutine scheduling, GC events), run go test -bench=<BenchmarkName> -trace=trace.out ./pkg/....
4. Open the profile interactively with go tool pprof <profile-file> and use pprof commands (top, list, graph, web) to identify hot paths.
5. For memory profiles, distinguish between -alloc_objects (total allocations, includes freed memory) and -inuse_space (live memory at peak, detects leaks).
6. For traces, run go tool trace trace.out to open the web UI and visualize goroutine execution timeline, GC pauses, and blocking events.

Example:

go test -bench=BenchmarkParse -cpuprofile=cpu.prof ./pkg/parser
go tool pprof cpu.prof
# (in pprof) top10  # shows top 10 functions by CPU time
# (in pprof) list Parse  # shows annotated source of Parse function
# (in pprof) web  # generates a call graph (requires graphviz)

Step 4: Compare Results with benchstat

Input: Two or more benchmark result files (e.g., old.txt, new.txt). Output: Statistical comparison table showing p-value, confidence interval, and percentage change.

1. Run benchmarks on the old version and save to old.txt.
2. Apply your optimization, run benchmarks again, save to new.txt.
3. Run benchstat old.txt new.txt to compute confidence intervals and p-values.
4. Read the output: if ~ appears next to the percentage, the change is not statistically significant (noise). If (p=0.000 n=10) appears, the result is significant at alpha=0.05.
5. Only commit optimizations where benchstat shows p < 0.05 (not ~).

Example output:

name                  old ns/op   new ns/op   delta
BenchmarkParse-8       230.5 ± 2%  152.3 ± 1%  -33.99% (p=0.000 n=10)

name                  old B/op    new B/op    delta
BenchmarkParse-8       128 ± 0%    64 ± 0%    -50.00% (p=0.000 n=10)

name                  old allocs  new allocs  delta
BenchmarkParse-8       2.00 ± 0%   1.00 ± 0%  -50.00% (p=0.000 n=10)

Step 5: Document Results in Commit

Input: benchstat output, commit message, code change. Output: Commit with performance impact documented.

1. Paste the benchstat output (CPU time, allocations, allocs/op) into the commit body.
2. Include the hardware context line at the top (e.g., goos: linux / goarch: amd64 / cpu: AMD Ryzen 9 5950X).
3. Strip rows that are unrelated to the change (only include affected benchmarks).
4. Only include results where benchstat shows statistical significance (no ~ symbol).
5. Use perf(scope): as the commit type for performance-only changes.

Example commit:

perf(parser): reduce Parse allocations 50% with sync.Pool

Replace per-call []byte allocation with a pooled buffer.

goos: linux / goarch: amd64 / cpu: AMD Ryzen 9 5950X
          │    old     │              new               │
          │  sec/op    │  sec/op     vs base            │
Parse-32    4.592µ ± 2%  3.041µ ± 1%  -33.78% (p=0.000 n=10)

          │   old    │             new              │
          │   B/op   │   B/op     vs base           │
Parse-32   1.024Ki ± 0%  0.512Ki ± 0%  -50.00% (p=0.000 n=0)

Step 6: Set Up CI Regression Detection (Optional)

Input: Benchmarks from previous steps, CI pipeline configuration (GitHub Actions, GitLab CI, etc.), baseline benchmark results. Output: CI job that gates pull requests if performance regresses beyond threshold.

1. Store baseline benchmark results (e.g., benchmarks/main.txt) in the repository.
2. In your CI pipeline, run benchmarks on the PR branch and save to a temp file.
3. Use benchdiff for quick visual PR comparisons (shows delta without strict thresholds).
4. Use cob (continuous benchmarking) for strict threshold-based gating (e.g., fail if latency increases >5%).
5. Use gobenchdata to append results to a JSON file and build long-term trend dashboards.
6. Account for noisy neighbor effects: cloud CI runners vary 5-10% even on quiet machines. Set thresholds conservatively (e.g., >10% regression to fail).
7. For reproducible benchmarks, use self-hosted runners or tune cloud runners: disable CPU frequency scaling, pin to isolated cores, and warm up the CPU before benchmarking.

Example GitHub Actions workflow:

- name: Run benchmarks
  run: go test -bench=. -benchmem -count=5 ./... | tee new.txt

- name: Compare with baseline
  run: benchstat benchmarks/main.txt new.txt | tee benchstat.txt

- name: Fail if regression > 10%
  run: |
    if grep -E "^\w+.*-[0-9]{1,3}\.[0-9]{2}%" benchstat.txt; then
      echo "Performance regression detected"
      exit 1
    fi

Step 7: Investigate Production Performance (Optional)

Input: Prometheus endpoint at /metrics, PromQL queries, runtime metrics. Output: Correlation between code changes and production behavior.

1. Ensure Go runtime metrics are exposed via prometheus/client_golang in your HTTP server.
2. Query Prometheus for key metrics: go_gc_duration_seconds_sum (total GC time), go_goroutines (active goroutines), go_memstats_heap_alloc_bytes (heap size).
3. Use PromQL to correlate metric spikes with code deployments: rate(go_gc_duration_seconds_sum[5m]) to track GC frequency over time.
4. Enable GODEBUG flags on production (e.g., GODEBUG=gctrace=1) to log GC events to stderr and correlate with latency increases.
5. Cross-reference local benchmarks with production metrics to identify why production differs (e.g., larger dataset sizes, different CPU topology, different garbage collection pressure).

Example PromQL:

# GC frequency (higher = more pressure on heap)
rate(go_gc_duration_seconds_count[5m])

# Heap size (sustained high values = memory leak)
go_memstats_heap_alloc_bytes

# Goroutine count (spike = possible goroutine leak)
go_goroutines

Decision Points

If the function call is trivially cheap (nanoseconds), then increase -benchtime to 5-10 seconds for statistical stability and to prevent compiler optimizations from skewing results.

If using Go <1.24, then use for i := 0; i < b.N; i++ loops and add b.ResetTimer() after expensive setup; otherwise use b.Loop().

If the benchmark result shows allocation where none is expected, then run with go test -gcflags="-m" to see escape analysis output and verify the compiler isn't moving stack values to the heap unnecessarily.

If benchstat output shows ~ (not statistically significant), then do not claim the optimization works; re-run with higher -count (e.g., -count=20) or longer -benchtime to reduce noise before drawing conclusions.

If profiling reveals a hot path in third-party code, then before optimizing, check the third-party library's GitHub issues or file a bug (the maintainer may already have a fix in development).

If running benchmarks on shared cloud CI hardware (AWS, GCP, Azure), then expect 5-10% variance between runs even on "quiet" machines; set regression thresholds to >10% to avoid false positives.

If production metrics diverge from local benchmarks (e.g., GC frequency is higher in prod), then enable GODEBUG=gctrace=1 on a canary instance to log GC events and correlate with code changes, heap size, or workload differences.

If tracing reveals goroutine blocking on locks or I/O, then use the golang-troubleshooting skill for Delve debugger setup to inspect goroutine state at runtime.

Output Contract

Benchmark runs produce a plain-text file (e.g., bench.txt) with rows in the format:

BenchmarkName-<GOMAXPROCS> <iterations> <time-per-op> <memory-per-op> <allocs-per-op>

Profile files are binary outputs:

• CPU profile: cpu.prof (input to go tool pprof)
• Memory profile: mem.prof (input to go tool pprof)
• Execution trace: trace.out (input to go tool trace)

benchstat output is a markdown-style table showing three columns per metric (old, new, delta with p-value and sample size):

name            old ns/op   new ns/op   delta
BenchmarkXxx-8  100 ± 5%    90 ± 3%     -10.00% (p=0.000 n=10)

Commit message includes benchstat table in body with hardware context (goos/goarch/cpu line) and only statistically significant results (no ~ rows).

CI regression job outputs a pass/fail status and optionally a benchstat diff URL or artifact for review.

Outcome Signal

• Benchmarks run successfully when go test -bench=. ./... completes without errors and outputs a readable table with ns/op, B/op, and allocs/op columns.
• Profiling works when go tool pprof <profile-file> opens interactively and top10 shows function names and CPU/memory percentages (not zero values).
• Statistical comparison is valid when benchstat output shows p-values (e.g., p=0.000) for significant results and ~ for noise; only then is the optimization claim defensible.
• Commit documentation is complete when the commit message includes the benchstat table, hardware context line, and only rows where p < 0.05 (statistically significant).
• CI regression gating is active when a pull request is rejected or flagged for review if benchmark latency or memory increases beyond the configured threshold (e.g., >10%).
• Production alignment is confirmed when Prometheus metrics (GC frequency, heap size, goroutine count) track similarly to local benchmark behavior after a code change is deployed, or divergence is explained by workload differences (larger data, higher concurrency).