Prepare for production deployments

Review recommendations for taking ambient mesh to production, covering resource allocation, scaling, and high-availability configuration.

Configuration changes for production

Resource allocation and scaling

Ztunnel is a DaemonSet, which runs one pod per node. Ztunnel is designed to have a small footprint and to handle large-scale clusters out of the box, scaling with cluster size (pods and services) and traffic rates (connections, requests, and throughput).

Ztunnel configuration changes are typically not required unless one of the following conditions applies:

• The cluster has over 100,000 pods or 20,000 services.
• An individual node serves over 20,000 connections, 100,000 requests per second, or 5Gb/s of traffic.

These are not limits of ztunnel, which can scale beyond these thresholds. They are the recommended point at which you might analyze usage and consider vertically scaling the CPU and memory reservations to match observed usage.

Ztunnel runs with a small CPU and memory reservation by default, and no limits. It also runs with a limited number of worker threads, which implicitly caps its maximum CPU usage. To inspect the current thread count:

istioctl ztunnel-config all -ojson | jq .config.numWorkerThreads
2

Review the following resource limit recommendations:

• CPU limits: Do not set CPU limits on the pod. Instead, tune the number of worker threads using the ZTUNNEL_WORKER_THREADS environment variable. Set it to a fixed number like 8, or a percentage like 25% (one thread per four cores, useful in mixed-node environments). If you must set a CPU limit, set it equal to the number of threads. Do not set the CPU limit lower than the thread count — this results in substantial performance degradation.
• Memory limits: Do not set memory limits. If you must set one, set it as high as possible. If the limit is exceeded simultaneously across nodes, all ztunnel pods could be killed and cause a cluster-wide networking outage.

Upgrading

To understand how to safely upgrade Istio when a new version is released, see the upgrading guide.

Component health monitoring

Monitor ambient mesh components at the default log level. Only increase log verbosity when actively troubleshooting, because debug logging under production traffic generates significant volume.

ztunnel

Ztunnel runs as a DaemonSet, so an unhealthy pod on any node means workloads on that node lose mesh connectivity. Check pod status across all nodes.

kubectl get pods -n istio-system -l app=ztunnel

Metrics

Ztunnel exposes Prometheus metrics on port 15020. The Istio sample Grafana dashboards include a ztunnel dashboard covering pod CPU and memory usage, connection counts, and bytes transmitted. Monitor for unexpected spikes or drops in any of these values.

Key metrics to alert on include:

• istio_tcp_connections_opened_total: Rate of new connections through ztunnel. A sudden drop on a node indicates ztunnel may have stopped processing traffic.
• istio_tcp_sent_bytes_total / istio_tcp_received_bytes_total: Bytes flowing through ztunnel. A sustained drop relative to your baseline suggests traffic is no longer routing through the mesh.

Logs

Ztunnel writes an access log entry for each connection on completion (not on open). Each entry includes source and destination workload identity, bytes transferred, duration, and an error field when the connection did not complete normally.

To surface errors across all ztunnel pods:

kubectl logs -n istio-system -l app=ztunnel --prefix | grep error

Common error patterns to watch for:

• TLS handshake error: Certificate or identity issue. Check that both workloads are enrolled in the mesh and that certificates have not expired.
• io error: deadline has elapsed: The connection timed out. Typically a network policy or firewall rule is blocking traffic between pods or to the destination.
• no healthy upstream: The destination has no ready endpoints. Check the target pod’s health and whether it is enrolled in the mesh.
• io error: Cannot assign requested address: A local network configuration problem on the node, often seen during node instability.

For common error scenarios and remediation steps, see the Istio ambient troubleshooting wiki.

istiod

An unhealthy istiod prevents configuration from reaching ztunnel and waypoint proxies, which causes policy and routing to become stale. Check the status of the istiod pod.

kubectl get pods -n istio-system -l app=istiod

Metrics

The Istio sample Grafana dashboards include a Pilot dashboard covering xDS push health, push latency, and connected proxy counts. Key metrics to alert on include:

• pilot_xds_rejects: Configurations rejected by connected proxies. Any non-zero value indicates a configuration error that proxies refused to accept.
• pilot_xds_connected_proxies: Number of ztunnel and waypoint proxies connected to the control plane. A sudden drop means proxies have lost contact with istiod.
• pilot_proxy_convergence_time: Time for configuration changes to propagate to all proxies. A sustained increase indicates control plane stress from high churn or large scale.

Logs

Under normal operation, istiod logs are primarily info-level xDS push activity. Watch for warn and error messages:

kubectl logs -n istio-system -l app=istiod | grep -iE 'warn|error'

Repeated warnings or errors referencing rejected configurations, certificate signing failures, or proxy disconnects are worth investigating promptly.

Applications

Adding workloads to the mesh does not change how you monitor your applications. Continue watching the same signals as before: request latency, error rates (4xx/5xx), pod readiness, and replica counts. If you observe unexpected changes in these signals after enrolling a namespace in the mesh, check the ztunnel logs on the affected nodes first.

Considerations for users already familiar with Istio

If you are already familiar with Istio in sidecar mode, review the following considerations for ambient mode:

• Layer 7 metrics are collected only for workloads with a waypoint deployed, or when using HTTP observability in Solo Enterprise for Istio.
• Authorization policies with Layer 7 conditions work only when bound to a waypoint.

Solo Enterprise for Istio

Solo Enterprise for Istio extends ambient mesh with production capabilities unavailable in upstream Istio. All paid Solo licenses include long-term support (LTS) and FIPS-compliant builds. An Enterprise license additionally unlocks:

• L7 observability without waypoints: Collect HTTP metrics, access logs, and distributed traces directly from ztunnels, without deploying a waypoint proxy per namespace. For more information, see the Solo Enterprise for Istio documentation.
• Multicluster mesh: Peer multiple clusters without remote Kubernetes API access or shared kubeconfig secrets. Each cluster runs its own istiod and east-west gateway; control planes connect over mTLS-secured xDS and services are discoverable across clusters with global naming. For more information, see Multicluster support.
• VM workloads: Add external VM workloads to the ambient mesh, each with their own SPIFFE identity, certificates, and authorization policy enforcement. A single VM can host multiple workloads, and the istioctl vm add-workload command automates onboarding. For more information, see the Solo Enterprise for Istio documentation.
• EnvoyFilter for waypoints: Apply the EnvoyFilter API to waypoint proxies to configure custom HTTP filters, modify route behavior, and apply Envoy-level customizations to L7 traffic. In upstream Istio, EnvoyFilter is supported for sidecars and ingress gateways only. For more information, see the Solo Enterprise for Istio documentation.

To review all features by license level, see the Solo.io Istio support plans.