Documentation
Traffic management
Mirroring

Mirroring

Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.

Deploy the sample services

Deploy httpbin-v1

You will use the httpbin service to view requests.

$ kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
        ports:
        - containerPort: 80
EOF

Deploy httpbin-v2

A second instance of httpbin will be used to mirror traffic to.

$ kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v2
  template:
    metadata:
      labels:
        app: httpbin
        version: v2
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
        ports:
        - containerPort: 80
EOF

Deploy the httpbin service 

$ kubectl create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
EOF

Deploy a curl client

We will use curl to send requests to our httpbin service.

$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      containers:
      - name: curl
        image: curlimages/curl
        command: ["/bin/sleep","3650d"]
        imagePullPolicy: IfNotPresent
EOF

Creating a default routing policy

By default Kubernetes load balances across both versions of the httpbin service. Change that behavior so that all traffic goes to v1.

Create a default route rule to route all traffic to v1 of the service:

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: httpbin-v1
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: httpbin
    version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-v2
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: httpbin
    version: v2
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: httpbin
    port: 8000
  rules:
  - backendRefs:
    - name: httpbin-v1
      port: 80
EOF

Now, with all traffic directed to httpbin:v1, send a request to the service:

$ kubectl exec deploy/curl -- curl -sS http://httpbin:8000/headers
{
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin:8000", 
    "User-Agent": "curl/8.7.1"
  }
}

Check the logs from httpbin-v1 and httpbin-v2 pods. You should see access log entries for v1 and none for v2:

$ kubectl logs deploy/httpbin-v1 -c httpbin
10.244.1.36 - - [29/Oct/2024:08:59:16 +0000] "GET /headers HTTP/1.1" 200 105 "-" "curl/8.7.1"
$ kubectl logs deploy/httpbin-v2 -c httpbin
<none>

Mirror traffic to httpbin-v2

Change the route to mirror traffic to httpbin-v2:

$ kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: httpbin
    port: 8000
  rules:
  - filters:
    - type: RequestMirror
      requestMirror:
        backendRef:
          name: httpbin-v2
          port: 80
    backendRefs:
    - name: httpbin-v1
      port: 80
EOF

This route sends 100% of the traffic to v1. The RequestMirror filter specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2 service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow.

The requests are mirrored as “fire and forget”, which means that the responses are discarded.

Send the traffic:

$ kubectl exec deploy/curl -- curl -sS http://httpbin:8000/headers

Now, you should see access logging for both v1 and v2. The access logs created in v2 are the mirrored requests that are actually going to v1.

$ kubectl logs deploy/httpbin-v1 -c httpbin
127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
$ kubectl logs deploy/httpbin-v2 -c httpbin
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"

Cleaning up

Remove the routes and supporting services:

$ kubectl delete httproute httpbin
$ kubectl delete svc httpbin-v1 httpbin-v2

Delete httpbin and curl deployments and httpbin service:

$ kubectl delete deploy httpbin-v1 httpbin-v2 curl
$ kubectl delete svc httpbin
Fault injectionRedirects and rewrites