Operational differences between sidecar and ambient mode

If you’re used to running Istio in sidecar mode, here are a few things you need to know about how ambient mode differs from it.

The istio-cni component is now required

The Istio CNI node agent was an optional component for sidecar mode. It was used to configure traffic redirection in one central, privileged place, without having to grant that privilege to every pod in the init container.

Traffic redirection for ztunnel is implemented by the CNI node agent, so it is now required in ambient mode. If you weren’t using it already, it means one extra Helm chart.

“CNI” is often used as a shorthand for “interface with the cluster’s network stack”, and it is important to note that this agent does not replace your primary Kubernetes CNI plugin — the software that assigns the IP addresses to each pod. It includes a component that is chained with it, meaning that it is invoked after the network interface is set up. Most of the magic happens in the agent that runs on each node. The agent is informed about new pod creation or deletion, and it reaches into the pod’s network namespace to configure a transparent redirection to the ztunnel on the same node.

You can read about how this works, but you probably will only want to do that if it ever stops working.

If you’re using Cilium

By default, Cilium expects to be an exclusive CNI plugin, and will remove other plugins as they attempt to install on the node. This configuration must be changed for ambient mesh to work with Cilium.

There are another couple of cases where if you’ve turned Cilium functionality on, it can conflict with Istio. Check the prerequisite docs for the up to date list, or the Solo.io migration tool will catch most of them in the pre-requisites phase.

Mesh routing is configured with Gateway API

As the Kubernetes community coalesced around Gateway API, so too did the service mesh community. We started with support for using Gateway API for ingress use cases, and support for mesh followed as the API was released upstream.

For ambient mesh, we made the decision that the only supported configuration method for in-mesh traffic (i.e. deploying and configuring waypoints) would be Gateway API. The targeting model maps very well to ambient mesh — it’s almost like it was designed by the same people! — where you can think of a waypoint as being a gateway to a namespace, or a set of services.

The architecture of Gateway API will not feel unfamiliar to people who are used to Istio’s legacy APIs, but the way that certain resources are attached to objects in the cluster may require some explanation. Further, for APIs where there is not yet a Gateway API equivalent (for example, Istio’s AuthorizationPolicy API), the way you interact with ztunnel and waypoint proxies can be different.

I will go into this in a lot more depth in an upcoming post on traffic management, but here are some of the key things you need to know.

Configuring Gateway API creates things

Istio gateways are first deployed by the administrator, and then configured using an object called Gateway. Confusingly, Gateway API also has an object called a Gateway, but creating one of those also deploys the proxy¹.

Because waypoints are Gateway API constructs, you define a Gateway resource and the waypoint infrastructure is created and managed for you.

You can’t do everything with Gateway API

…but it’s hopefully a 90/10 thing, and you can get pretty close.

Gateway API is deliberately designed to be implementation-independent, and there are some things in Istio’s API that are specific to Envoy. Most of these gaps will close over time, but there are generally workarounds.

For example, mirroring a percentage of traffic is new in Gateway API 1.3, but has long been supported in VirtualService. Retries on an HTTPRoute and TCP proxying both require you to use the experimental channel of Gateway API at the time of writing,

For a small number of situations where you need an escape valve, you can still use VirtualService objects to configure waypoints. There are caveats to this, and we will address them in the upcoming post on traffic management.

How you address things is different

Gateway API defines the relationship between objects (such as routes and gateways) in terms of attachment.

• Route objects (such as HTTPRoute) include a way to reference the parent resources it wants to attach to.
• Policy objects (such as AuthorizationPolicy) augment the behavior of a target resource in a standard way.

The resources you can attach to are Services² and Gateways, including both ingress gateways and waypoints.

The routes in an Istio VirtualService match based on hostname; routes in Gateway API attach based on service name. These things are generally the same, in that the hostname you used was probably the name of the service anyway, but it’s a useful distinction to make.

In a VirtualService, you would specify a gateway stanza to have a route apply to a named gateway, or leave the stanza empty to have it apply to “all sidecars in the mesh”. Similarly, with Gateway API, each route has a parentRefs field which specifies the attachment:

• kind: Gateway lets you specify a named Gateway
• kind: Service matches a Kubernetes Service, and the route is attached to any waypoints that are used by that service³.

Given that a waypoint is a gateway for a set of services, you can address your policies in one of two ways: by targeting the services that use the waypoint (kind: Service), or by targeting the waypoint itself (kind: Gateway)⁴.

For simplicity, we often refer to both types of object as attaching to or targeting a particular resource, and then refer to where they bind to; i.e. the gateway or waypoint that actually receives that traffic.

Selectors and targets

When there is a 1:1 mapping between a workload and a proxy, it is sensible to use Kubernetes workload selectors — the “core grouping primitive in Kubernetes” — to apply configuration to a set of proxies.

For some of Istio’s legacy APIs, you use a selector field to decide if the policy applied to a particular pod (by matching its labels).

In ambient mode, ztunnel is present for every workload, and so it can be thought of as an extension of that workload, just like a sidecar could before. That means, for Layer 4 use cases, you still use the selector method to address workloads.

Gateway targeting is different, because when you’re addressing something at Layer 7, there are multiple services and hostnames that could resolve to a single workload.

Some Istio objects can thus be mapped to workloads using a selector (a Kubernetes construct) or attached to a set of services using targetRefs (a Gateway API construct). We’ll get into this as we talk about the things you need to apply this knowledge to.

You need to do one final upgrade to your sidecars

The ambient mesh data plane tunnels its traffic over a new-and-improved protocol. HBONE is a mechanism to transparently tunnel TCP streams related to many different application connections over an mTLS-encrypted network connection.

HBONE is implemented in sidecars, but it is not enabled by default. When you install Istio with the ambient profile, it also configures both the control plane and the sidecar profile to support HBONE. To be able to interoperate between the sidecar and the ambient data plane, you need to enable this.

Stop the world; I want to get off!

Sidecars can now speak the ambient mode protocol, but because they predate the existence of ambient mode, they are not aware of waypoints. By default, they will try and do what they always do, which is to load-balance on the client side. They would then send traffic directly to the backend workload, bypassing any configured waypoint.

While you’re migrating from sidecars to ambient, you are going to be changing routing and filtering from client-side to server-side. You will also be moving any Layer 7 security from your server-side sidecar to a waypoint. You can’t move your server to ambient until you know this is done, and the clients can’t move to ambient until you have done it. Therefore, to do a big-bang migration to ambient without waypoint-aware sidecars, you would have to stop the world, and endure a small amount of downtime.

Solo.io has built waypoint-awareness into sidecars in Gloo Mesh, and made this functionality available in free builds of Istio that can be installed by anyone. Using these builds, you can do a zero-downtime migration (i.e. one where there is no need to stop all services to switch from client-side to waypoint-side processing.)

In the next post, we’ll look at how authentication and authorization differs between sidecar and ambient mode, and how you will need to write and attach policies.