Mirroring
Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.
Deploy the sample services
Deploy httpbin-v1
You will use the httpbin service to view requests.
$ kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-v1
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
version: v1
template:
metadata:
labels:
app: httpbin
version: v1
spec:
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin
command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
ports:
- containerPort: 80
EOF
Deploy httpbin-v2
A second instance of httpbin will be used to mirror traffic to.
$ kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-v2
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
version: v2
template:
metadata:
labels:
app: httpbin
version: v2
spec:
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin
command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
ports:
- containerPort: 80
EOF
Deploy the httpbin service
$ kubectl create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: httpbin
labels:
app: httpbin
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: httpbin
EOF
Deploy a curl client
We will use curl
to send requests to our httpbin
service.
$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: curl
spec:
replicas: 1
selector:
matchLabels:
app: curl
template:
metadata:
labels:
app: curl
spec:
containers:
- name: curl
image: curlimages/curl
command: ["/bin/sleep","3650d"]
imagePullPolicy: IfNotPresent
EOF
Creating a default routing policy
By default Kubernetes load balances across both versions of the httpbin
service.
Change that behavior so that all traffic goes to v1
.
Create a default route rule to route all traffic to v1
of the service:
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: httpbin-v1
spec:
ports:
- port: 80
name: http
selector:
app: httpbin
version: v1
---
apiVersion: v1
kind: Service
metadata:
name: httpbin-v2
spec:
ports:
- port: 80
name: http
selector:
app: httpbin
version: v2
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin
spec:
parentRefs:
- group: ""
kind: Service
name: httpbin
port: 8000
rules:
- backendRefs:
- name: httpbin-v1
port: 80
EOF
Now, with all traffic directed to httpbin:v1
, send a request to the service:
$ kubectl exec deploy/curl -- curl -sS http://httpbin:8000/headers
{
"headers": {
"Accept": "*/*",
"Host": "httpbin:8000",
"User-Agent": "curl/8.7.1"
}
}
Check the logs from httpbin-v1
and httpbin-v2
pods. You should see access log entries for v1
and none for v2
:
$ kubectl logs deploy/httpbin-v1 -c httpbin
10.244.1.36 - - [29/Oct/2024:08:59:16 +0000] "GET /headers HTTP/1.1" 200 105 "-" "curl/8.7.1"
$ kubectl logs deploy/httpbin-v2 -c httpbin
<none>
Mirror traffic to httpbin-v2
Change the route to mirror traffic to httpbin-v2
:
$ kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin
spec:
parentRefs:
- group: ""
kind: Service
name: httpbin
port: 8000
rules:
- filters:
- type: RequestMirror
requestMirror:
backendRef:
name: httpbin-v2
port: 80
backendRefs:
- name: httpbin-v1
port: 80
EOF
This route sends 100% of the traffic to v1. The RequestMirror filter
specifies that you want to mirror (i.e., also send) 100% of the same traffic to the
httpbin:v2
service. When traffic gets mirrored,
the requests are sent to the mirrored service with their Host/Authority headers
appended with -shadow
.
The requests are mirrored as “fire and forget”, which means that the responses are discarded.
Send the traffic:
$ kubectl exec deploy/curl -- curl -sS http://httpbin:8000/headers
Now, you should see access logging for both v1
and v2
. The access logs
created in v2
are the mirrored requests that are actually going to v1
.
$ kubectl logs deploy/httpbin-v1 -c httpbin
127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
$ kubectl logs deploy/httpbin-v2 -c httpbin
127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"
Cleaning up
Remove the routes and supporting services:
$ kubectl delete httproute httpbin
$ kubectl delete svc httpbin-v1 httpbin-v2
Delete httpbin
and curl
deployments and httpbin
service:
$ kubectl delete deploy httpbin-v1 httpbin-v2 curl
$ kubectl delete svc httpbin