Skip to content

Circuit breakers

Page as Markdown 

    

  
  


              
            

          

          
          
Configure circuit breakers in ambient mesh to temporarily stop traffic to unhealthy services and prevent cascading failures.

          
Circuit breaking is a resiliency mechanism that temporarily stops requests to a downstream service when the service is detected as unhealthy. This mechanism helps prevent cascading failures by blocking traffic to services that are overloaded, unresponsive, or returning too many errors. Once the service recovers, the circuit breaker automatically resumes traffic.


Because circuit breaking is implemented between services in the ambient mesh, and the configuration is generally specified by the downstream service owner, it can also be thought of as an implementation of the load shedding pattern.


In this guide, you learn how to set up and monitor circuit breakers in your ambient mesh.



  
Note


  

    
Circuit breakers are supported at the edge of a cluster using a gateway, or when a workload is enrolled in the waypoint layer.

    

  




Before you begin
    
    

  1. 

    Install an ambient mesh in your cluster.
    

    2. 

  2. 

    Label the default namespace to add it to the ambient mesh.
    

    

    kubectl label ns default istio.io/dataplane-mode=ambient
    
  

    

    

    3. 

  3. 

    Create a waypoint for the default namespace. A waypoint is required because this resiliency feature configures Layer 7 (L7) proxy behavior. For more information on using waypoints, see Configure waypoints.
    

    

    istioctl waypoint apply -n default --enroll-namespace --wait
    
  

    

    

    4. 



Deploy httpbin
    
    

  1. 

    Deploy the httpbin service and two clients: curl, for sending a single request, and fortio, a load testing tool that lets you specify the number of requests and concurrency level.
    

    


    kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/httpbin/httpbin.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/curl/curl.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/httpbin/sample-client/fortio-deploy.yaml
    
  

    

    

    2. 

  2. 

    To inspect requests that return a 503 response code in subsequent steps, enable waypoint access logging. By default, waypoint logging is disabled and can be enabled by using Istio’s Telemetry API.
    

      

    1. Enable waypoint access logging by applying a Telemetry resource.

      


      kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: enable-access-logging
  namespace: default
spec:
  accessLogging:
    - providers:
      - name: envoy
EOF
      
  

      

      

      2. 

    2. In one terminal, tail the waypoint’s logs.

      


      kubectl logs --follow deploy/waypoint
      
  

      

      

      3. 

    3. In a second terminal, send a test request.

      


      kubectl exec deploy/curl -- curl -s httpbin:8000/get
      
  

      

      
In the output, verify that a log entry similar to the following appears, which confirms that requests traverse the waypoint.

      


      [2026-07-07T17:34:13.930Z] "GET /get HTTP/1.1" 200 - via_upstream - "-" 0 369 3 3 "-" "curl/8.16.0" "c2304c0a-8d65-4450-b1e6-692f736b56e7" "httpbin:8000" "envoy://connect_originate/10.244.0.36:8080" inbound-vip|8000|http|httpbin.default.svc.cluster.local envoy://internal_client_address/ 10.96.103.171:8000 10.244.0.37:60878 - default

      
  

      

      

      4. 

    

    Keep the waypoint logs open to refer to them in subsequent steps.
    

    3. 

  3. 

    Send 100 requests to httpbin to establish a baseline before configuring circuit breaking.
    

      

    • -c 3 uses three concurrent threads for requests
      • 

    • -qps 0 sends the requests as fast as possible
      • 

    • -n is the number of requests
      • 

    • -quiet reduces the output verbosity
      • 

    

    


    kubectl exec deploy/fortio-deploy -- fortio load -c 3 -qps 0 -n 100 -quiet http://httpbin:8000/get
    
  

    

    

    In the output, confirm that all 100 requests return 200.
    

    


    Code 200 : 100 (100.0 %)

    
  

    

    

    4. 



Configure circuit breakers
    
    

  1. 

    Apply a DestinationRule to configure circuit breaking for httpbin. A DestinationRule configures post-routing proxy behavior, including circuit breaking thresholds. These thresholds are set artificially low for this example; for production guidance, see the Istio reference documentation.
    

    


    kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin.default.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
EOF
    
  

    

    

    2. 

  2. 

    Send another 100 requests to trip the circuit breakers. Sending more than a couple of concurrent requests exceeds the artificially low thresholds.
    

    


    kubectl exec deploy/fortio-deploy -- fortio load -c 3 -qps 0 -n 100 -quiet http://httpbin:8000/get
    
  

    

    

    In the output, note that a percentage of requests return a 503 “Service unavailable”.
    

    


    Code 200 : 60 (60.0 %)
Code 503 : 40 (40.0 %)

    
  

    

    

    3. 

  3. 

    In the waypoint logs, verify that the UO (UpstreamOverflow) response flag field is present, which confirms that the circuit breaker tripped and prevented the request from reaching httpbin.
    

    


    [2026-07-07T17:34:25.767Z] "GET /get HTTP/1.1" 503 UO upstream_reset_before_response_started{overflow} - "-" 0 81 0 - "-" "fortio.org/fortio-1.69.5" "6e74f8d4-7e74-4e6f-8160-d8fbc1c1d164" "httpbin:8000" "-" inbound-vip|8000|http|httpbin.default.svc.cluster.local - 10.96.103.171:8000 10.244.0.38:50118 - default

    
  

    

    

    4. 



Circuit breaking metrics
    


By default, Istio configures Envoy to record a minimal set of statistics to reduce overall CPU and memory footprint. Circuit breaker metrics are not included by default, and you must configure Istio to include them in the following steps. After you enable the metrics, overflows are reflected in metrics such as upstream_cx_overflow (connection overflow) and upstream_rq_pending_overflow (request overflow). Two other overflow breaker metrics, the connection pool overflow and retry budget overflow, are available as additional thresholds that are not explored in this guide.


    

  1. 

    Configure circuit breaking metrics collection.
    

      

    1. 

      Save the following Helm values to a file named mesh-config.yaml.
      

      


      cat > mesh-config.yaml <<EOF
meshConfig:
  defaultConfig:
    proxyStatsMatcher:
      inclusionRegexps:
        - ".*outlier_detection.*"
        - ".*upstream_rq_retry.*"
        - ".*upstream_rq_pending.*"
        - ".*upstream_cx_.*"
      inclusionSuffixes:
        - "upstream_rq_timeout"
EOF
      
  

      

      

      2. 

    2. 

      Upgrade the istiod chart with the updated configuration.
      

      


      helm upgrade istiod istio/istiod \
  --namespace istio-system \
  --reuse-values \
  --values mesh-config.yaml \
  --wait
      
  

      

      

      3. 

    3. 

      Restart the waypoint to pick up the updated configuration.
      

      


      kubectl rollout restart deploy waypoint
      
  

      

      

      4. 

    

    2. 

  2. 

    Review metrics by using Prometheus or the Envoy proxy’s admin interface.

   
   
   
    
  
    

    

      

    1. 

      Deploy Prometheus to the cluster.
      

      


      kubectl apply -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/addons/prometheus.yaml
      
  

      

      

      2. 

    2. 

      Connect to the Prometheus dashboard.
      

      


      istioctl dashboard prometheus
      
  

      

      

      3. 

    3. 

      Query the pending connection overflow metric for httpbin. Note the current counter value (if you just deployed Prometheus, there may be no result yet).
      

        

      • envoy_cluster_upstream_rq_pending_overflow{cluster_name=~".*httpbin.*"}
        • 

      

      4. 

    4. 

      Send 100 more requests to increase the counters.
      

      


      kubectl exec deploy/fortio-deploy -- fortio load -c 3 -qps 0 -n 100 -quiet http://httpbin:8000/get
      
  

      

      

      5. 

    5. 

      Wait 20 seconds for Prometheus to collect another round of metrics, and then re-run the envoy_cluster_upstream_rq_pending_overflow{cluster_name=~".*httpbin.*"} query. The counts increase by the same number as the 503 errors reported by Fortio.
      

       
      Figure: Pending requests overflow counter metric in Prometheus
       
      Figure: Pending requests overflow counter metric in Prometheus
      

      6. 

    

    

    

    3. 



Cleanup
    
Delete the sample resources.





kubectl delete -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/addons/prometheus.yaml
kubectl delete destinationrule httpbin
istioctl waypoint delete -n default waypoint
kubectl delete -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/httpbin/httpbin.yaml
kubectl delete -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/curl/curl.yaml
kubectl delete -f https://raw.githubusercontent.com/istio/istio/1.30.0/samples/httpbin/sample-client/fortio-deploy.yaml

  





        

        

  

Request timeoutsOutlier detection