Fault injection

Test system resilience in ambient mesh by injecting latency and errors via waypoint proxies using Istio’s VirtualService resource.

Fault injection lets you actively test resilience behaviors, such as whether circuit breakers, retries, and outlier detection function as expected, by deliberately introducing latency and errors into your mesh. Istio injects faults by instrumenting the waypoints in the path between microservices.

This guide uses Istio’s Bookinfo sample application. The scenario takes advantage of the fact that reviews-v2 and reviews-v3 call an upstream ratings service while reviews-v1 does not.

Note

The Kubernetes Gateway API does not currently support fault injection, and retries are only available in the Experimental channel. This guide uses Istio’s VirtualService resource. Mixing VirtualService routing with HTTPRoute objects is not recommended.

Before you begin

Install an ambient mesh in your cluster.
Label the default namespace to add it to the ambient mesh.
```
kubectl label ns default istio.io/dataplane-mode=ambient
```
Create a waypoint for the default namespace. A waypoint is required because this resiliency feature configures Layer 7 (L7) proxy behavior. For more information on using waypoints, see Configure waypoints.
```
istioctl waypoint apply -n default --enroll-namespace --wait
```

Configure a delay

Deploy the Bookinfo sample application and a curl client.

kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/curl/curl.yaml

To inspect requests that return a 503 response code in subsequent steps, enable waypoint access logging. By default, waypoint logging is disabled and can be enabled by using Istio’s Telemetry API.

Enable waypoint access logging by applying a Telemetry resource.

kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: enable-access-logging
  namespace: default
spec:
  accessLogging:
    - providers:
      - name: envoy
EOF

In one terminal, tail the waypoint’s logs.
```
kubectl logs --follow deploy/waypoint
```

In a second terminal, send a test request.

kubectl exec deploy/curl -- curl -s reviews:9080/reviews/123

In the output, verify that a log entry similar to the following appears, which confirms that requests traverse the waypoint.

[2026-07-07T16:57:34.543Z] "GET /reviews/123 HTTP/1.1" 200 - via_upstream - "-" 0 360 573 573 "-" "curl/8.16.0" "a900c543-5e81-9828-bb58-6190575f1aeb" "reviews:9080" "envoy://connect_originate/10.244.0.21:9080" inbound-vip|9080|http|reviews.default.svc.cluster.local envoy://internal_client_address/ 10.96.195.57:9080 10.244.0.25:35810 - default

Keep the waypoint logs open to refer to them in subsequent steps.

Apply a VirtualService to inject a 4-second latency into all calls to the ratings service.

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        fixedDelay: 4s
        percentage:
          value: 100.0
    route:
    - destination:
        host: ratings
EOF

Call the reviews service several times.

kubectl exec deploy/curl -- curl reviews:9080/reviews/123

When reviews-v1 handles the request, the response arrives immediately:

{"id": "123","podname": "reviews-v1-7bbc4c5497-p2q6l","clustername": "null","reviews": [{"reviewer": "Reviewer1","text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!"},{"reviewer": "Reviewer2","text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare."}]}

When reviews-v2 or reviews-v3 handles the request, the response is delayed by ~4 seconds (the injected latency on the upstream ratings call). These versions also include star ratings in their response.

In the waypoint logs, the DI (Delay Injected) response flag confirms that the waypoint is injecting latency into ratings calls.

[2026-07-07T16:57:47.201Z] "GET /ratings/123 HTTP/1.1" 200 DI via_upstream - "-" 0 50 4010 7 "-" "curl/8.16.0" "238a6703-229b-49a6-8e8e-c9de84db98b4" "ratings:9080" "envoy://connect_originate/10.244.0.20:9080" inbound-vip|9080|http|ratings.default.svc.cluster.local envoy://internal_client_address/ 10.96.130.38:9080 10.244.0.22:40704 - -

Configure a timeout in the client

Apply a VirtualService to configure a 1-second timeout on reviews calls.

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
    timeout: 1s
EOF

Test the timeout.

kubectl exec deploy/curl -- curl -s -v reviews:9080/reviews/123

When reviews-v2 or reviews-v3 handles the request, the 1-second timeout elapses before ratings responds, and you get a 504:

...
< HTTP/1.1 504 Gateway Timeout
< content-length: 24
< content-type: text/plain
< server: istio-envoy
<
upstream request timeout

Configure clients to retry

You can augment a timeout with a retry policy so that when calls to reviews-v2 or reviews-v3 time out, the mesh retries and eventually reaches reviews-v1. In a worst-case scenario, a request is first routed to reviews-v2 (fails with 504 after 1s), retried to reviews-v3 (also fails), then succeeds on reviews-v1. The client waits more than 2 seconds total. Retrying on gateway-error responses with a per-try timeout bounds each attempt to 1s.

Update the VirtualService to add two retry attempts with a 1-second, per-try timeout.

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
    retries:
      attempts: 2
      perTryTimeout: 1s
      retryOn: gateway-error
EOF

Send 10 requests to productpage and confirm that all succeed, all reviews come from reviews-v1, and some requests take slightly longer due to retries.
```
for i in {1..10}; do
  kubectl exec deploy/curl -- curl -s productpage:9080/productpage | grep reviews-
done
```
Each request produces two lines (one per reviewer). Retries land on reviews-v1 since reviews-v2 and reviews-v3 always exceed the 1-second per-try timeout:
```
reviews-v1
reviews-v1
reviews-v1
...
```

Augment with outlier detection

This configuration works, but is not optimal, because retries are still attempted on every request. Adding outlier detection ejects reviews-v2 and reviews-v3 from the load balancing pool after they start failing, so subsequent requests route directly to reviews-v1 with minimal retries.

Apply a DestinationRule that ejects an endpoint after a single gateway error, with a 15-second base ejection time.

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    outlierDetection:
      consecutiveGatewayErrors: 1
      baseEjectionTime: 15s
      maxEjectionPercent: 100
EOF

Send 10 more requests. Most return without delay as ejections route all traffic to reviews-v1.
```
for i in {1..10}; do
  kubectl exec deploy/curl -- curl -s productpage:9080/productpage | grep reviews-
done
```
Each request produces two lines (one per reviewer on the page). With reviews-v2 and reviews-v3 ejected, all traffic routes to reviews-v1:
```
reviews-v1
reviews-v1
reviews-v1
...
```
Note

To observe ejection counters directly from the waypoint, you must first enable outlier detection metrics via proxyStatsMatcher. Without that configuration, the outlier_detection.* stats are not active in Envoy.
Remove the ratings delay. After about 15 seconds, the ejected workloads return to the load balancing pool.
```
kubectl delete virtualservice ratings
```

To confirm recovery, send another series of requests and confirm that all three versions of reviews handle traffic again.

for i in {1..10}; do
  kubectl exec deploy/curl -- curl -s productpage:9080/productpage | grep reviews-
done

reviews-v1
reviews-v1
reviews-v3
reviews-v2
...

Cleanup

Delete the sample resources.

kubectl delete destinationrule reviews
kubectl delete virtualservice reviews
kubectl delete virtualservice ratings
istioctl waypoint delete -n default waypoint
kubectl delete -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl delete -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/curl/curl.yaml

Outlier detection