Troubleshooting

Diagnose and resolve common issues in ambient mesh, from ztunnel traffic capture problems to connectivity and policy failures.

Install and setup issues

Before troubleshooting, verify that your environment meets the following requirements:

Pod fails to start with `Failed to create pod sandbox`

For pods in the mesh, Istio runs a CNI plugin during pod sandbox creation to configure networking rules. This operation might intermittently fail, in which case Kubernetes automatically retries.

Common causes include the following:

no ztunnel connection: The CNI plugin is not connected to ztunnel. Verify that ztunnel is running on the same node and is healthy.
failed to add IP ... to ipset istio-inpod-probes: exist: Istio attempted to add the workload, but encountered a race condition in Kubernetes IP allocation. Kubernetes automatically retries, which usually resolves this. Other occurrences of this error might indicate a bug.

Ztunnel fails with `failed to bind to address [::1]:15053: Address family not supported`

Your kernel does not support IPv6. Disable IPv6 support by setting IPV6_ENABLED=false on ztunnel.

Ztunnel traffic issues

Understanding access logs

When troubleshooting traffic issues, start by analyzing the ztunnel access logs. Two ztunnel pods might be involved in a single request (the source and the destination), so check logs on both sides.

Access logs are written on each connection completion by default. Connection opening logs are available at debug level. For details, see how to set the log level.

An example access log entry looks like the following:

2024-09-25T22:08:30.213996Z     info    access  connection complete     src.addr=10.244.0.33:50676 src.workload="shell-5b7cf9f6c4-7hfkc" src.namespace="default" src.identity="spiffe://cluster.local/ns/default/sa/default" dst.addr=10.244.0.29:15008 dst.hbone_addr=10.96.99.218:80 dst.service="echo.default.svc.cluster.local" dst.workload="waypoint-66f44865c4-l7btm" dst.namespace="default" dst.identity="spiffe://cluster.local/ns/default/sa/waypoint" direction="outbound" bytes_sent=67 bytes_recv=518 duration="2ms"

The src/dst addr, workload, namespace, and identity fields identify the source and destination of the traffic. Not all fields are present for every connection:
- identity is only set when mTLS is used.
- dst.namespace and dst.workload are absent when traffic is sent to an unknown destination (passthrough traffic).
dst.service is the destination service, when the request targets a service rather than a pod directly.
dst.hbone_addr is set when using mTLS. In this case, hbone_addr is the target of the traffic, while dst.addr is the address ztunnel connected to for the tunnel.
bytes_sent and bytes_recv indicate how many bytes were transferred during the connection.
duration indicates how long the connection was open.
error, if present, indicates that the connection encountered an error and describes the cause.

In the example above, dst.service is echo, but dst.workload and dst.addr belong to waypoint-..., which confirms that traffic was sent through a waypoint proxy.

Ztunnel does not capture traffic

Check the pod for the ambient.istio.io/redirection annotation. The presence of this annotation indicates that the CNI node agent enabled redirection.

kubectl get pods shell-5b7cf9f6c4-npqgz -oyaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    ambient.istio.io/redirection: enabled

If the annotation is missing, the pod was not enrolled in the mesh.

Check the logs of the istio-cni-node pod on the same node as the affected pod for errors. Errors during enablement might block the pod from receiving traffic from ztunnel.
Check the logs of the istio-cni-pod on the same node to verify that ambient is enabled. The pod logs AmbientEnabled: true during startup. If this value is false, verify that you installed Istio with --set profile=ambient.
Verify that the pod is configured to use ambient mode. All of the following must be true:
- The pod or namespace must have the istio.io/dataplane-mode=ambient label.
- The pod must not have the sidecar.istio.io/status annotation, which is added automatically when a sidecar is injected.
- The pod must not have istio.io/dataplane-mode=none.
- The pod must not have spec.hostNetwork=true.

If the annotation is present, Istio claims it enabled redirection for the pod, but the redirection is not working as expected.

Check the iptables rules in the pod. Run a debug shell and run iptables-save.

# iptables-save

Your output might vary from the following example, but any Istio-related entries confirm that iptables rules are installed.

# Generated by iptables-save v1.8.10 on Wed Sep 25 22:06:16 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_PRERT - [0:0]
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -d 169.254.7.127/32 -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
-A ISTIO_PRERT -s 169.254.7.127/32 -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d 127.0.0.1/32 -p tcp -m tcp ! --dport 15008 -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15006

Verify that ztunnel is running within the pod network.

netstat -ntl

Verify that you see listeners on several Istio ports, including 15001, 15006, and others.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:15053         0.0.0.0:*               LISTEN
tcp6       0      0 ::1:15053               :::*                    LISTEN
tcp6       0      0 :::15001                :::*                    LISTEN
tcp6       0      0 :::15006                :::*                    LISTEN
tcp6       0      0 :::15008                :::*                    LISTEN

Check the ztunnel logs. When you send traffic, verify that you see log entries like info access connection complete .... Ztunnel logs connections when they are closed, not when they are opened, so long-lived connections might not produce log entries immediately.

Traffic timeout with ztunnel

When traffic is blocked, ztunnel logs errors similar to the following:

error   access  connection complete      direction="outbound" bytes_sent=0 bytes_recv=0 duration="10002ms" error="io error: deadline has elapsed"
error   access  connection complete      direction="outbound" bytes_sent=0 bytes_recv=0 duration="10002ms" error="connection timed out, maybe a NetworkPolicy is blocking HBONE port 15008: deadline has elapsed"

For the connection timed out error, the connection could not be established. A common cause is a NetworkPolicy or other firewall rule blocking port 15008. Istio mTLS traffic is tunneled over port 15008, so both ingress and egress must allow this port.
For generic errors like io error: deadline has elapsed, the root cause is usually the same as for connection timed out. However, if traffic works without ambient mode, a typical firewall rule is unlikely to be the cause, because traffic follows the same path as without ambient mode. This error likely indicates an incompatibility with your Kubernetes setup.

Readiness probes fail with ztunnel

After enabling ambient mode, pod readiness probes fail. For example, you might see the following:

  Warning  Unhealthy               92s (x6 over 4m2s)   kubelet                  Readiness probe failed: Get "http://1.1.1.1:8080/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Ambient mode excludes readiness probe traffic from interception by applying a SNAT rule on the host that rewrites any traffic from kubelet as originating from 169.254.7.127, then skips redirection for traffic matching this source address.

Readiness probe failures that begin after enabling ambient mode typically indicate an environmental issue with this traffic rewrite. Known causes include the following:

Cilium with bpf.masquerade=true breaks this behavior. See the platform prerequisites guide and issue #52208.
Calico before version 3.29 with bpfEnabled set breaks this behavior. See issue #52765.
AWS Security Groups might block this traffic. See issue #53105.

Traffic fails with `timed out waiting for workload from xds`

When a pod sends traffic, ztunnel must first retrieve information about the pod from istiod over XDS. If ztunnel does not receive this information within 5 seconds, it rejects the connection with this error.

Istiod is expected to respond significantly faster than 5 seconds. If this error occurs intermittently, istiod might be overloaded, or a configuration change might have increased PILOT_DEBOUNCE_AFTER, which slows down updates.

If the error occurs persistently, this is likely a bug. File an issue in the open source Istio repo.

Traffic fails with `no healthy upstream`

This error indicates that a service has no applicable backends.

Run the following command to see how ztunnel views the service’s health:
```
istioctl zc services
```
This example output shows 4 endpoints for the service, with 1 marked as unhealthy.
```
NAMESPACE    SERVICE NAME         SERVICE VIP    WAYPOINT ENDPOINTS
default      echo                 10.96.99.1     None     3/4
```

Run the following command to see how Kubernetes views the service:

kubectl get endpointslices

Example output:

NAME          ADDRESSTYPE   PORTS    ENDPOINTS                           AGE
echo-v76p9    IPv4          8080     10.244.0.20,10.244.0.36 + 1 more... 7h50m

If Kubernetes shows zero healthy endpoints, the issue is not in the Istio configuration; the service is actually unhealthy. Verify that its labels select the expected workloads and that those pods are marked as ready.

Traffic fails with `http status: ...`

Ztunnel acts as a TCP proxy and does not parse user HTTP traffic. Because ztunnel uses the HBONE tunneling protocol, which runs over HTTP CONNECT, ztunnel can report an HTTP error even for non-HTTP traffic.

An HTTP error indicates that ztunnel established an HBONE connection, but the stream was rejected. When communicating with another ztunnel, common causes include the following:

400 Bad Request: The request was malformed. This likely indicates a bug.
401 Unauthorized: The request was rejected by an AuthorizationPolicy.
503 Service Unavailable: The destination is not available.

When communicating with a waypoint proxy (Envoy), a wider range of response codes is possible. 401 indicates an AuthorizationPolicy rejection; 503 is a general failure.

Traffic fails with `connection closed due to connection drain`

When ztunnel shuts down a proxy instance, it closes any outstanding connections. Before closing, ztunnel logs inpod::statemanager pod delete request, shutting down proxy for the affected pod.

This can happen in the following cases:

The pod is deleted. In this case, the connections are generally already closed.
Ztunnel itself is shutting down.
The pod was un-enrolled from ambient mode.

Ztunnel does not send egress traffic to waypoints

Consider a ServiceEntry such as the following:

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: example.com
  labels:
    istio.io/use-waypoint: my-waypoint
spec:
  hosts:
  - example.com
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS

Unlike a typical Service, a ServiceEntry does not have the two components required for traffic capture to work:

A stable Service IP address known to Istio (example.com might have many, changing IPs).
DNS configuration to return such a stable IP address.

Istio provides two features to resolve these issues:

values.pilot.env.PILOT_ENABLE_IP_AUTOALLOCATE=true enables a controller that allocates an IP address for the ServiceEntry and writes it into the object:
```
status:
  addresses:
  - host: example.com
    value: 240.240.0.3
  - host: example.com
    value: 2001:2::3
```
values.cni.ambient.dnsCapture=true enables ztunnel to handle DNS, so it can respond with the allocated IP addresses for queries to example.com. Restart client workloads after changing this setting.

Together, these settings enable egress traffic to traverse a waypoint. To troubleshoot this:

Verify that the ServiceEntry has an IP address in the status.
Check that the pod receives this IP address in DNS lookups.
Check whether this IP appears as the destination IP address in ztunnel.

Waypoint issues

Traffic does not go through the waypoint

Look for the following indicators that traffic is traversing a waypoint:

Requests sent through a waypoint proxy go through Envoy’s HTTP processing, which modifies the request. By default, headers are translated to lowercase and several Envoy headers are injected:
```
x-envoy-upstream-service-time: 2
server: istio-envoy
x-envoy-decorator-operation: echo.default.svc.cluster.local:80/*
```
Note that this does not apply when traffic is configured as TCP.
Waypoint access logs, if enabled, log each request. See here for how to enable access logs.
Ztunnel access logs, if enabled, log each request. See Understanding access logs for an example log entry showing traffic to a waypoint.

Traffic can be sent to a service or directly to a workload. While sending to a service is typical, check the ztunnel access logs to identify the traffic type. A waypoint can be associated with a service, a workload, or both. Mismatches between the traffic destination and waypoint association can cause the waypoint to be bypassed.

Tip

Cilium with bpf-lb-sock requires bpf-lb-sock-hostns-only to be set, or all traffic is incorrectly treated as direct-to-workload traffic. See issue #27619.

Run the following commands to check whether ztunnel is configured to send traffic to a waypoint.
```
istioctl zc services
istioctl zc workloads
```
Example output:
```
NAMESPACE    SERVICE NAME         SERVICE VIP  WAYPOINT ENDPOINTS
default      echo                 10.96.0.1    waypoint 1/1
default      no-waypoint          10.96.0.2    None     1/1

NAMESPACE  POD NAME                     ADDRESS     NODE  WAYPOINT     PROTOCOL
default    echo-79dcbf57cc-l2cdp        10.244.0.1  node  None         HBONE
default    product-59896bc9f7-kp4lb     10.244.0.2  node  waypoint     HBONE
```
The output shows that the echo service and the product-59896bc9f7-kp4lb pod are bound to waypoint. If ztunnel is configured to use the waypoint for the destination but traffic is not reaching the waypoint, the traffic might be going to the wrong destination. Check the ztunnel access logs to verify the destination service or workload.

If the waypoint shows as None, ztunnel is not programmed to use a waypoint.

Check the status of the object. The status indicates whether the resource is attached to the waypoint. This field is available in Istio 1.24 and later, and currently applies to Service and ServiceEntry only.

kubectl get svc echo -oyaml

Example output:

status:
  conditions:
  - lastTransitionTime: "2024-09-25T19:28:16Z"
    message: Successfully attached to waypoint default/waypoint
    reason: WaypointAccepted
    status: "True"
    type: istio.io/WaypointBound

Check what resources are configured to use a waypoint. Check namespaces in all cases. For service-based traffic, also check services and serviceentries. For workload-based traffic, also check pods and workloadentries. The istio.io/use-waypoint label must be present to associate a resource with a waypoint.
```
kubectl get namespaces -L istio.io/use-waypoint
```
Example output:
```
NAME                           STATUS   AGE    USE-WAYPOINT
namespace/default              Active   1h     waypoint
namespace/istio-system         Active   1h
```
If the label is present but traffic is still not reaching the waypoint, the waypoint might be missing or unhealthy. Check the Gateway objects to verify that the waypoint is deployed.
```
kubectl get gateways.gateway.networking.k8s.io
```
This example output shows a waypoint that is deployed but not healthy. A waypoint remains inactive until it becomes healthy at least once. If the waypoint is not healthy, check the status for more details. If the Gateway object is not present, deploy one.
```
NAME       CLASS            ADDRESS   PROGRAMMED   AGE
waypoint   istio-waypoint             False        17s
```

Sidecar proxy returns `404 NR route_not_found` for a `ServiceEntry` bound to a waypoint

When a sidecar proxy is already connected to istiod at the time a ServiceEntry’s WaypointBound status is first established, istiod may not push the updated route to the connected sidecar. The sidecar retains the previous inlined route and returns 404 NR route_not_found errors until it reconnects.

To resolve this, restart the affected sidecar proxy pod. The proxy reconnects to istiod and receives the correct tunnel-to-waypoint route.

Diagnostic tools

Running a debug shell

Most pods have limited privileges and few debug tools available. For some diagnostics, run an ephemeral container with elevated privileges and utilities using the istio/base image with kubectl debug --profile sysadmin.

For example:

kubectl debug --image istio/base --profile sysadmin --attach -t -i shell-5b7cf9f6c4-npqgz

Setting the log level

To view the current log level:

istioctl zc log ztunnel-cqg6c

ztunnel-cqg6c.istio-system:
current log level is info

To set the log level:

istioctl zc log ztunnel-cqg6c --level=info,access=debug

ztunnel-cqg6c.istio-system:
current log level is hickory_server::server::server_future=off,access=debug,info

To set the log level at ztunnel pod startup, configure the environment variable:

kubectl -n istio-system set env ds/ztunnel RUST_LOG=info

Using Solo's builds of Istio