gke-service-networking

SKILL.md

GKE Service Networking Skill

This skill provides workflows for exposing applications running on GKE securely
to the internet or internal networks.

Deployable manifest templates live in assets/ — edit the # Replace ...
placeholders before applying.

Workflows

1. Configure Gateway API (Recommended)

The Gateway API is the modern way to manage routing in Kubernetes.

Prerequisites: Gateway API must be enabled on the cluster (enabled by
default on new clusters running GKE 1.26+; on older supported versions enable it
with --gateway-api=standard).

Templates:

assets/gateway.yaml — external Gateway using the
gke-l7-global-external-managed GatewayClass with an HTTP listener.

assets/httproute.yaml — HTTPRoute attaching to the Gateway via
parentRefs and routing a path prefix to a Service backendRef.

assets/httproute-traffic-split.yaml — HTTPRoute demonstrating weighted
traffic splitting (e.g. 90/10) for canary deployments across backend
services.

kubectl apply -f assets/gateway.yaml
kubectl apply -f assets/httproute.yaml

Traffic Splitting (Canary Deployments):

HTTPRoute supports weighted traffic splitting across multiple backend Services
for canary rollouts:

spec:
  rules:
    - backendRefs:
        - name: app-v1
          port: 80
          weight: 90
        - name: app-v2
          port: 80
          weight: 10

2. Configure Standard GKE Ingress

Use standard Ingress for simpler use cases or legacy setups.

Template: assets/ingress.yaml — GCE Ingress (kubernetes.io/ingress.class: "gce" annotation) routing to a Service.

3. Secure with Cloud Armor

Cloud Armor provides WAF and DDoS protection.

Create a Security Policy in Cloud Armor:

gcloud compute security-policies create {security_policy_name} \
  --description "WAF policy for {app_name}"

# Example rule: block an abusive IP range
gcloud compute security-policies rules create 1000 \
  --security-policy {security_policy_name} \
  --action deny-403 \
  --src-ip-ranges "203.0.113.0/24" \
  --description "Block abusive range"

Reference it in a BackendConfig: assets/backendconfig.yaml (sets
spec.securityPolicy.name).

Associate the BackendConfig with your Service via annotations:

# In your Kubernetes Service manifest metadata.annotations:
cloud.google.com/backend-config: '{"default": "{backend_config_name}"}'
# Or for specific port mappings:
cloud.google.com/backend-config: '{"ports": {"80": "{backend_config_name}"}}'

4. Configure Google-Managed SSL Certificates

Automatically provision and renew SSL certificates.

Legacy Ingress approach: apply assets/managed-certificate.yaml (a
ManagedCertificate listing your domains), then reference it in the Ingress
annotations:

networking.gke.io/managed-certificates: {certificate_name}

Gateway API approach: for standard Certificate Manager integration, create a
CertificateMap and reference it in the Gateway metadata annotations using the
exact annotation networking.gke.io/certmap (spelled without any hyphens in
certmap):

metadata:
  annotations:
    networking.gke.io/certmap: {certificate_map_name}

[!IMPORTANT] The annotation key is strictly networking.gke.io/certmap (do
not use cert-map or certificate-map).

Alternatively, reference a Kubernetes Secret in the HTTPS listener's
tls.certificateRefs. Both variants are in assets/gateway-https.yaml.

5. Enable Container-Native Load Balancing (Recommended)

Container-native load balancing allows load balancers to target Kubernetes Pods
directly, rather than targeting nodes. This improves latency and distribution.

Prerequisites: Cluster must be VPC-native.

How it works: the cloud.google.com/neg annotation on a Service triggers
creation of a NEG that mirrors the Pod IPs. GKE often adds it for you — but not
always, and knowing which case you are in is the whole point.

# In your Kubernetes Service manifest metadata.annotations:
cloud.google.com/neg: '{"ingress": true}'

When the annotation is automatic (do not add it by hand):

Internal Ingress — container-native load balancing is always used, not
optional. Internal Ingress always uses GCE_VM_IP_PORT NEGs and requires a
VPC-native cluster.

External Ingress, but only when all four hold: the cluster is
VPC-native, is not on Shared VPC, does not use GKE Network Policy, and has
the HttpLoadBalancing add-on enabled (on by default — do not disable it).
GKE then annotates Services automatically.

When you must add it explicitly:

Standalone NEGs — you manage the load balancer yourself instead of
letting Ingress own it. Required if the LB must be configured outside GKE,
since Ingress overwrites managed load balancer settings on sync or upgrade.
You become responsible for every part of the load balancer.

Any external-Ingress cluster failing one of the four conditions above —
Shared VPC, GKE Network Policy, or non-VPC-native. Enable per Service.

Legacy configurations — some older external Ingress objects created on
VPC-native clusters still use instance group backends.

Not supported / no NEG fallback:

Windows Server node pools.

Routes-based (non-VPC-native) clusters with external Ingress — the Ingress
controller falls back to unmanaged instance groups spanning all nodes.

Scale consequence: without NEGs a cluster is capped at 1,000 nodes, and
non-NEG Services behind Ingress stop functioning correctly beyond that. With
NEGs there is no GKE node limit.

6. Configure Private Service Connect (PSC)

Private Service Connect allows you to expose services in one VPC to consumers in
another VPC securely, without VPC peering.

Prerequisite: The backing Service must be an internal passthrough Network
Load Balancer — i.e. type: LoadBalancer with the
networking.gke.io/load-balancer-type: "Internal" annotation. The
ServiceAttachment requires this; a ClusterIP or external LoadBalancer Service
will not work.

Steps:

Create an internal LoadBalancer Service for your workload.

Create a ServiceAttachment referencing that Service:
assets/service-attachment.yaml (sets connectionPreference, the PSC NAT
subnet, and the Service resourceRef).

Share the ServiceAttachment URI with consumers to create a PSC endpoint in
their VPC.

7. Topology Aware Routing (Cost & Latency Optimization)

To minimize cross-zone data transfer costs and network latency, configure
Kubernetes Services with Topology Aware Routing. This routes traffic to Pods in
the same zone as the originating client:

# In your Kubernetes Service manifest metadata.annotations:
service.kubernetes.io/topology-mode: auto

Troubleshooting

Diagnose Ingress / load-balancer data-plane failures. These map to both Ingress
(BackendConfig / FrontendConfig) and Gateway (GCPBackendPolicy /
HealthCheckPolicy / GCPGatewayPolicy). Stay at the read-only → propose-manifest
boundary; never apply live mutations directly.

502 / 5xx with UNHEALTHY backends (health checks)

The Google Cloud load-balancer health check is separate from Kubernetes
liveness/readiness probes — it runs from outside the cluster, so a Pod can be
Ready while the backend service still shows UNHEALTHY.

Allow the Google health-check source ranges to the node/Pod serving port.
GKE usually creates this rule automatically, but on Shared VPC or with
hand-managed firewalls it can be missing:

gcloud compute firewall-rules create allow-lb-health-checks \
  --allow tcp:SERVING_PORT \
  --source-ranges 130.211.0.0/22,35.191.0.0/16 \
  --target-tags NODE_TAG

Point the health check at a healthy endpoint. If the default / returns a
non-200, set a custom health check with a BackendConfig (Ingress) or a
HealthCheckPolicy (Gateway):

# BackendConfig (Ingress)
spec:
  healthCheck:
    requestPath: /healthz
    port: 8080
    checkIntervalSec: 15
    timeoutSec: 5

Confirm the Service is container-native (NEG) so the check targets Pod IPs
rather than nodes (see workflow 5).

502 / dropped requests during rollouts or on long requests

Enable connection draining so in-flight requests finish before a backend
Pod is removed during a rolling update or scale-down:

# BackendConfig (Ingress)
spec:
  connectionDraining:
    drainingTimeoutSec: 60

Raise the backend timeout for slow or streaming responses — a 502/408 on a
request that runs longer than the backend response timeout is the classic
symptom. Set timeoutSec in the BackendConfig (Ingress) or GCPBackendPolicy
(Gateway).

TLS / SSL handshake failures or weak-cipher enforcement

Ingress: attach an SSL policy (minimum TLS version / cipher profile) with a
FrontendConfig sslPolicy, and optionally force HTTP→HTTPS with
redirectToHttps:

# FrontendConfig (external Ingress only)
spec:
  sslPolicy: gke-ingress-ssl-policy
  redirectToHttps:
    enabled: true

Gateway: attach the SSL policy name in a GCPGatewayPolicy. For a regional
Gateway, create and reference a regional SSL policy.

Gotchas

Certificate Manager API must be enabled for the
networking.gke.io/certmap annotation to work (gcloud services enable certificatemanager.googleapis.com); without it the Gateway fails to
provision the certificate map.

Regional Gateway classes need a proxy-only subnet: classes like
gke-l7-regional-external-managed and gke-l7-rilb require a subnet with
--purpose=REGIONAL_MANAGED_PROXY in the region; the Gateway stays
unprogrammed without it.

ManagedCertificate provisioning depends on DNS: the certificate stays in
Provisioning until the domain's A/AAAA records point at the load balancer
IP, and can take 15-60 minutes after DNS is correct.

References

Ingress features (BackendConfig / FrontendConfig)

Container-native load balancing (NEGs)

Configure Gateway resources (GCPGatewayPolicy / HealthCheckPolicy / GCPBackendPolicy)

Health check concepts