Documentation
Traffic management
Mirroring

Mirroring

Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.

Deploy the sample services

Deploy httpbin-v1

You will use the httpbin service to view requests.

$ kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
        ports:
        - containerPort: 80
EOF

Deploy httpbin-v2

A second instance of httpbin will be used to mirror traffic to.

$ kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v2
  template:
    metadata:
      labels:
        app: httpbin
        version: v2
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
        ports:
        - containerPort: 80
EOF

Deploy the httpbin service 

$ kubectl create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
EOF

Deploy a curl client

We will use curl to send requests to our httpbin service.

$ kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      containers:
      - name: curl
        image: curlimages/curl
        command: ["/bin/sleep","3650d"]
        imagePullPolicy: IfNotPresent
EOF

Creating a default routing policy

By default Kubernetes load balances across both versions of the httpbin service. Change that behavior so that all traffic goes to v1.

Create a default route rule

Create a default route rule to route all traffic to v1 of the service:

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: httpbin-v1
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: httpbin
    version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-v2
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: httpbin
    version: v2
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: httpbin
    port: 8000
  rules:
  - backendRefs:
    - name: httpbin-v1
      port: 80
EOF

Send a request

Now, with all traffic directed to httpbin:v1, send a request to the service:

$ kubectl exec deploy/curl -- curl -sS http://httpbin:8000/headers
{
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin:8000", 
    "User-Agent": "curl/8.7.1"
  }
}

Check the logs

Check the logs from httpbin-v1 and httpbin-v2 pods. You should see access log entries for v1 and none for v2:

$ kubectl logs deploy/httpbin-v1 -c httpbin
10.244.1.36 - - [29/Oct/2024:08:59:16 +0000] "GET /headers HTTP/1.1" 200 105 "-" "curl/8.7.1"

$ kubectl logs deploy/httpbin-v2 -c httpbin
<none>

Mirror traffic to httpbin-v2

Mirror traffic to httpbin-v2

Change the route to mirror traffic to httpbin-v2:

$ kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: httpbin
    port: 8000
  rules:
  - filters:
    - type: RequestMirror
      requestMirror:
        backendRef:
          name: httpbin-v2
          port: 80
    backendRefs:
    - name: httpbin-v1
      port: 80
EOF

This route sends 100% of the traffic to v1. The RequestMirror filter specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2 service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow.

The requests are mirrored as “fire and forget”, which means that the responses are discarded.

Send the traffic 

$ kubectl exec deploy/curl -- curl -sS http://httpbin:8000/headers

Check the logs

Now, you should see access logging for both v1 and v2. The access logs created in v2 are the mirrored requests that are actually going to v1.

$ kubectl logs deploy/httpbin-v1 -c httpbin
127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"

$ kubectl logs deploy/httpbin-v2 -c httpbin
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"

Cleaning up

Remove the routes and supporting services:

$ kubectl delete httproute httpbin
$ kubectl delete svc httpbin-v1 httpbin-v2

Delete httpbin and curl deployments and httpbin service:

$ kubectl delete deploy httpbin-v1 httpbin-v2 curl
$ kubectl delete svc httpbin
Traffic splittingRedirects and rewrites