For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Review recommendations for taking ambient mesh to production, covering resource allocation, scaling, and high-availability configuration.

Configuration changes for production

Resource allocation and scaling

Ztunnel is a DaemonSet, which runs one pod per node. Ztunnel is designed to have a small footprint and to handle large-scale clusters out of the box, scaling with cluster size (pods and services) and traffic rates (connections, requests, and throughput).

Ztunnel configuration changes are typically not required unless one of the following conditions applies:

• The cluster has over 100,000 pods or 20,000 services.
• An individual node serves over 20,000 connections, 100,000 requests per second, or 5Gb/s of traffic.

These are not limits of ztunnel, which can scale beyond these thresholds. They are the recommended point at which you might analyze usage and consider vertically scaling the CPU and memory reservations to match observed usage.

Ztunnel runs with a small CPU and memory reservation by default, and no limits. However, its worker thread count caps its maximum CPU usage.

In Istio 1.31 and later, ztunnel sizes its worker thread pool from the CPU resources of its own pod. CPU quantities round up to whole cores, so 500m gives 1 core, 1.5 gives 2, etc.

• If you set neither a CPU request nor limit, ztunnel uses 25% of the logical cores of the node, with a minimum of 2 threads.
• If you set only a CPU request, ztunnel uses 25% of the logical cores of the node or the request size, whichever is greater, with a minimum of 2 threads. The chart requests 200m by default, which rounds to 1 core and is therefore too small to raise the count.
• If you set a CPU limit, ztunnel uses that limit as its thread count, or the CPU request if that is higher (not recommended).

In earlier versions, ztunnel always ran 2 worker threads unless you overrode the count by using the ZTUNNEL_WORKER_THREADS environment variable. On nodes with more than 8 cores, expect the ztunnel worker thread count to rise after you upgrade to 1.31.

To inspect the current thread count:

istioctl ztunnel-config all -ojson | jq .config.numWorkerThreads

Review the following resource limit recommendations:

• CPU limits: Do not set CPU limits on the pod. Ztunnel does not need a limit to bound its CPU usage, because the worker thread count already does this. If you must set a limit, do not set it to 1 core or less, which takes ztunnel below 2 worker threads.
• Memory limits: Do not set memory limits. If you must set one, set it as high as possible. If the limit is exceeded simultaneously across nodes, all ztunnel pods could be killed and cause a cluster-wide networking outage.

Tune the ztunnel worker thread count

The automatic sizing based on node resources described above is a good default. Override it when you want a thread count that does not follow from the CPU resources of the pod, such as a fixed count across a fleet of differently sized nodes.

The ZTUNNEL_WORKER_THREADS environment variable on the ztunnel pod takes precedence over the automatic sizing. To tune the count, use one of the following methods:

• Set a CPU request: Set resources.requests.cpu in the ztunnel chart. The request reserves the CPU and sets the minimum thread count in one value. This is the best option for most clusters.
• Set an explicit thread count: Set ZTUNNEL_WORKER_THREADS to a fixed number such as 8, to a percentage such as 25% (one thread per four cores, useful in mixed-node environments), or to 0 to use all available cores. Percentages and 0 are relative to the CPU limit of the pod when one is set, and to the logical cores of the node otherwise. A fixed number is not limit-aware: setting more threads than the CPU limit allows can result in substantial performance degradation.

Upgrading

To understand how to safely upgrade Istio when a new version is released, see the upgrading guide.

Component health monitoring

Monitor ambient mesh components at the default log level. Only increase log verbosity when actively troubleshooting, because debug logging under production traffic generates significant volume.

ztunnel

Ztunnel runs as a DaemonSet, so an unhealthy pod on any node means workloads on that node lose mesh connectivity. Check pod status across all nodes.

kubectl get pods -n istio-system -l app=ztunnel

Metrics

Ztunnel exposes Prometheus metrics on port 15020. The Istio sample Grafana dashboards include a ztunnel dashboard covering pod CPU and memory usage, connection counts, and bytes transmitted. Monitor for unexpected spikes or drops in any of these values.

Key metrics to alert on include:

• istio_tcp_connections_opened_total: Rate of new connections through ztunnel. A sudden drop on a node indicates ztunnel may have stopped processing traffic.
• istio_tcp_sent_bytes_total / istio_tcp_received_bytes_total: Bytes flowing through ztunnel. A sustained drop relative to your baseline suggests traffic is no longer routing through the mesh.

Logs

Ztunnel writes an access log entry for each connection on completion (not on open). Each entry includes source and destination workload identity, bytes transferred, duration, and an error field when the connection did not complete normally.

To surface errors across all ztunnel pods:

kubectl logs -n istio-system -l app=ztunnel --prefix | grep error

Common error patterns to watch for:

• TLS handshake error: Certificate or identity issue. Check that both workloads are enrolled in the mesh and that certificates have not expired.
• io error: deadline has elapsed: The connection timed out. Typically a network policy or firewall rule is blocking traffic between pods or to the destination.
• no healthy upstream: The destination has no ready endpoints. Check the target pod’s health and whether it is enrolled in the mesh.
• io error: Cannot assign requested address: A local network configuration problem on the node, often seen during node instability.

For common error scenarios and remediation steps, see the Istio ambient troubleshooting wiki.

istiod

An unhealthy istiod prevents configuration from reaching ztunnel and waypoint proxies, which causes policy and routing to become stale. Check the status of the istiod pod.

kubectl get pods -n istio-system -l app=istiod

Metrics

The Istio sample Grafana dashboards include a Pilot dashboard covering xDS push health, push latency, and connected proxy counts. Key metrics to alert on include:

• pilot_xds_rejects: Configurations rejected by connected proxies. Any non-zero value indicates a configuration error that proxies refused to accept.
• pilot_xds_connected_proxies: Number of ztunnel and waypoint proxies connected to the control plane. A sudden drop means proxies have lost contact with istiod.
• pilot_proxy_convergence_time: Time for configuration changes to propagate to all proxies. A sustained increase indicates control plane stress from high churn or large scale.

Logs

Under normal operation, istiod logs are primarily info-level xDS push activity. Watch for warn and error messages:

kubectl logs -n istio-system -l app=istiod | grep -iE 'warn|error'

Repeated warnings or errors referencing rejected configurations, certificate signing failures, or proxy disconnects are worth investigating promptly.

Applications

Adding workloads to the mesh does not change how you monitor your applications. Continue watching the same signals as before: request latency, error rates (4xx/5xx), pod readiness, and replica counts. If you observe unexpected changes in these signals after enrolling a namespace in the mesh, check the ztunnel logs on the affected nodes first.

Considerations for users already familiar with Istio

If you are already familiar with Istio in sidecar mode, review the following considerations for ambient mode:

• Layer 7 metrics are collected only for workloads with a waypoint deployed, or when using HTTP observability in Solo Enterprise for Istio.
• Authorization policies with Layer 7 conditions work only when bound to a waypoint.

Solo Enterprise for Istio

Solo Enterprise for Istio extends ambient mesh with production capabilities unavailable in upstream Istio. All paid Solo licenses include long-term support (LTS) and FIPS-compliant builds. An Enterprise license additionally unlocks:

• L7 observability without waypoints: Collect HTTP metrics, access logs, and distributed traces directly from ztunnels, without deploying a waypoint proxy per namespace. For more information, see the Solo Enterprise for Istio documentation.
• Multicluster mesh: Peer multiple clusters without remote Kubernetes API access or shared kubeconfig secrets. Each cluster runs its own istiod and east-west gateway; control planes connect over mTLS-secured xDS and services are discoverable across clusters with global naming. For more information, see Multicluster support.
• VM workloads: Add external VM workloads to the ambient mesh, each with their own SPIFFE identity, certificates, and authorization policy enforcement. A single VM can host multiple workloads, and the istioctl vm add-workload command automates onboarding. For more information, see the Solo Enterprise for Istio documentation.
• EnvoyFilter for waypoints: Apply the EnvoyFilter API to waypoint proxies to configure custom HTTP filters, modify route behavior, and apply Envoy-level customizations to L7 traffic. In upstream Istio, EnvoyFilter is supported for sidecars and ingress gateways only. For more information, see the Solo Enterprise for Istio documentation.

To review all features by license level, see the Solo.io Istio support plans.