Migrating to ambient mesh from Istio in sidecar mode

The Solo ambient mesh migration tool provides a prescriptive path for migrating from Istio’s sidecar mode to ambient mode. This migration is zero-downtime when used with the free Solo builds of Istio, but can also translate Kubernetes manifests for users of upstream Istio.

While the tool will walk you through each step, having an understanding of ambient mode and its differences from sidecars will be helpful. We recommend going through the documentation to get an overview of the architecture and differences. In particular, understanding the security architecture of ambient mesh is critical if you use authorization policies.

The tool does not manipulate any configuration in your cluster. It only reads configuration and suggests actions that you must take to adjust configuration. When following the procedures recommended by the tool and using a Solo build of Istio, a cluster can be migrated from sidecar mode to ambient without downtime. However, the tool cannot account for all possible cluster setups; operators are expected to review the suggestions and test them before acting on them.

Before you begin

Throughout this guide, we will show a simplified representation of the traffic path of communication between a client and server. As you progress through each step, more components will be deployed and enabled. Make sure that the tool is passing all its checks before moving to the next step.

In sidecar mode, client-side sidecars are responsible for traffic routing decisions (depicted here as “routing”), and server-side sidecars are responsible for authentication and authorization (depicted as “policy”).

There are two main considerations for migration:

• Client-side sidecars make all routing decisions locally, without awareness of waypoint proxies
• Authorization policies are applied to server-side sidecars using a selector, whereas in ambient mode, policies target waypoints using a reference

The Solo builds of Istio include support for migration by enhancing sidecar mode to be aware of waypoint proxies, and to route traffic to a waypoint if a workload is enrolled to use it, using the same logic as ztunnel.

Istio policies that are bound to a waypoint must be adjusted to use a targetRef instead of a selector. The migration tool detects the most common policies and provides new configuration, which you then validate and apply.

Download the migration helper

Install the gloo CLI client:

$ curl -sL https://storage.googleapis.com/gloo-cli/install.sh | sh
$ export PATH=$HOME/.gloo/bin:$PATH

There are a number of options that can be provided to the migration commands, including the ability to run from YAML files instead of a live cluster. These can be found in an appendix to this page.

Prepare your cluster for ambient mesh

To migrate to ambient mesh, your cluster must have:

• Istio v1.25 or later
• Istio components configured with support for ambient mode
• The Gateway API CRDs installed

Certain Istio features are not available in ambient mode.

• Multi-cluster mesh
• VM integration
• SPIRE

If you attempt to upgrade a cluster that uses these features, the tool will advise you.

Run the prerequisites step to confirm your cluster is ready for operating in ambient mode, and you are not using any features that are not supported:

An example cluster that is ready for ambient mode:

$ gloo ambient migrate
• Starting phase pre-reqs...
✅ Phase pre-reqs succeeded!
  ✅ Cluster CNI compatibility: passed
  ✅ Istio version compatibility: passed
  ✅ Multicluster usage compatibility: passed
  ✅ Virtual Machine usage compatibility: passed
  ✅ SPIRE usage compatibility: passed

All existing istiod installations in the cluster must run Istio 1.25 or higher. The next phase will ensure the Istio installation has support for ambient mode. If an upgrade is needed, you may consider enabling ambient mode in the same step.

Next, run the cluster-setup check to confirm your Istio environment is ready:

$ gloo ambient migrate --phase cluster-setup
• Starting phase cluster-setup...
❌ Phase cluster-setup failed!
  ❌ Ambient mode enabled: failed. 1 error occurred:
  	* Istiod istio-system/istiod must have 'PILOT_ENABLE_AMBIENT=true'. Upgrade Istio with '--set profile=ambient'.
  
  
  ❌ DaemonSets deployed: failed. 2 errors occurred:
  	* ztunnel not found
  	* Istio CNI agent not found
  
  
  ❌ Sidecars support ambient mode: failed. 6 errors occurred:
  	* Sidecar default/details-v1-79dfbd6fff-gvmh7 is missing 'ENABLE_HBONE'. Upgrade Istio with '--set profile=ambient' and restart the pod.
  	* Sidecar default/productpage-v1-dffc47f64-tt9r8 is missing 'ENABLE_HBONE'. Upgrade Istio with '--set profile=ambient' and restart the pod.
  	* Sidecar default/ratings-v1-65f797b499-fsxqf is missing 'ENABLE_HBONE'. Upgrade Istio with '--set profile=ambient' and restart the pod.
  	* Sidecar default/reviews-v1-5c4d6d447c-tz928 is missing 'ENABLE_HBONE'. Upgrade Istio with '--set profile=ambient' and restart the pod.
  	* Sidecar default/reviews-v2-65cb66b45c-dzptm is missing 'ENABLE_HBONE'. Upgrade Istio with '--set profile=ambient' and restart the pod.
  	* Sidecar default/reviews-v3-f68f94645-h9c8l is missing 'ENABLE_HBONE'. Upgrade Istio with '--set profile=ambient' and restart the pod.
  
  
  ✅ Required CRDs installed: passed

Most users will have to install ambient mode and restart their sidecar workloads to pass this step.

Enable support for ambient mode

Most sidecar mode installations are not installed with the ambient profile, which includes the ztunnel component. To install it, you will follow the steps in the upgrade guide, with parameters to install the Solo builds.

This ensures Istio is installed with support for ambient mode, which enables you to start enrolling workloads into an ambient mesh, and enables the existing dataplanes (sidecar and gateways) to communicate with ambient workloads.

For example, for an installation with istioctl, ensure you have the same version of istioctl as the version of Istio you wish to install, and then update the installation:

$ export HUB=us-docker.pkg.dev/soloio-img/istio
$ export TAG=1.25.2-solo
$ istioctl install --set profile=ambient --set hub=$HUB --set tag=$TAG

For an installation with Helm:

$ export HUB=us-docker.pkg.dev/soloio-img/istio
$ export TAG=1.25.2-solo
$ helm repo add istio https://istio-release.storage.googleapis.com/charts
$ helm repo update istio
$ helm install istio-base istio/base -n istio-system --set global.hub=$HUB --set global.tag=$TAG --create-namespace --wait
$ helm install istiod istio/istiod --namespace istio-system --set global.hub=$HUB --set global.tag=$TAG --set profile=ambient --wait
$ helm install istio-cni istio/cni -n istio-system --set global.hub=$HUB --set global.tag=$TAG --set profile=ambient --wait
$ helm install ztunnel istio/ztunnel -n istio-system --set global.hub=$HUB --set global.tag=$TAG --wait

At this stage, you will have ambient mode support installed, but not in use:

Ensure your workloads are running an ambient-aware sidecar

Configuring an Istio installation for ambient mode enables support for sidecars to use the HBONE protocol. The sidecars must be restarted for this change to take effect. This can be achieved by restarting the Deployments in a namespace:

$ kubectl rollout restart deployment -n <namespace>

Before progressing to the next step, ensure the check now fully passes:

$ gloo ambient migrate --phase cluster-setup
• Starting phase cluster-setup...
✅ Phase cluster-setup succeeded!
  ✅ Ambient mode enabled: passed
  ✅ DaemonSets deployed: passed
  ✅ Sidecars support ambient mode: passed
  ✅ Required CRDs installed: passed

Deploy waypoints

In sidecar mode, all services have L7 processing, whether or not they need this functionality. Ambient mode allows you to choose which services are opted into L7 processing.

To migrate from sidecars, you must evaluate each service to see if it requires the use of a waypoint proxy. The migration tool will analyze cluster configuration to determine which services require waypoints to be deployed to retain their existing functionality. For each recommendation, configuration to deploy these waypoints will be provided.

The suggestions from the tool are a best-effort analysis based on the policies in the cluster. Review the recommended waypoint configurations carefully before deploying waypoints for best results. For example, the tool might detect customized load balancing for a service and recommend a waypoint, as waypoints are required to implement custom load balancing policies. However, you may decide that the value of this policy doesn’t justify the waypoint, and choose to skip the recommendation. If you aren’t sure, we recommend deploying the waypoint.

In some cases, the tool might not recommend a waypoint where you actually want apps to use one. Even without any policies configured, sidecars provide a variety of functionalities such as mTLS, HTTP observability, and HTTP request-level load balancing. While ztunnel provides mTLS and can optionally provide HTTP observability, it does not provide request-level load balancing. Applications requiring this can benefit from deploying a waypoint, even when the tool does not recommend one.

An example cluster where a single waypoint is recommended:

$ gloo ambient migrate --phase deploy-waypoints
• Starting phase deploy-waypoints...
⚠️ Phase deploy-waypoints has recommendations!
  🔮 Namespace "application" may require a waypoint for the following services:
     * Service "application/hello-world" depends on VirtualService "application/hello-world"

  ℹ️ Generated waypoints written to /tmp/istio-migrate/recommended-waypoints.yaml

The recommended-waypoints.yaml file contains the recommended waypoint configuration:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: application
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

Apply this configuration through your regular deployment pipeline, or using kubectl:

$ kubectl apply -f /tmp/istio-migrate/recommended-waypoints.yaml 

For more information about configuring waypoints, see Configure waypoint proxies.

At the end of this stage, you have deployed waypoint proxies, but they are not in use.

Migrate policies

At this point, you have any necessary waypoints deployed, but they are not fully configured or in use. The next step is to configure them. As in the previous step, traffic will still not be routed through the waypoint, which will occur in a later step once the waypoints are fully configured.

For policies that modify services, VirtualServices and DestinationRules, no changes are required. The tool copies the configuration and generates policies to be applied to waypoints. However, policies that modify specific workloads must be modified to apply to the waypoint.

The tool will automatically provide recommendations on how to apply the following resources to waypoints:

• AuthorizationPolicy
• RequestAuthentication
• Telemetry
• WasmPlugin
• EnvoyFilter (when running with --enterprise)

It will also detect and warn against policies that are not supported or required in ambient mode.

An example in which a few policies are migrated to the waypoint, as well as a warning about an unsupported configuration:

$ gloo ambient migrate --phase migrate-policies
• Starting phase migrate-policies...
⚠️ Phase migrate-policies has recommendations!
  ⚠️ Sidecar/legacy/sidecar will be removed in a later step: Found unsupported configurations:
    * Configuration scoping is detected. Typically, this is used for scalability purposes, which is not necessary with ambient.
  🔮 Add AuthorizationPolicy/app1/echo-from-waypoint: Service/app1/echo must allow traffic from its waypoint
  🔮 Apply AuthorizationPolicy/app1/echo-legacy-policies: Existing configuration is copied from policy app1/legacy-policies to be enforced at the waypoint.
  🔮 Apply AuthorizationPolicy/app1/echo-policies: Existing configuration is copied from policy app1/policies to be enforced at the waypoint.
  🔮 Apply AuthorizationPolicy/app1/waypoint-allow-nothing: Existing configuration is copied from namespace policy app1/allow-nothing to be enforced at the waypoint.
  ℹ️ Recommended policies written to /tmp/istio-migrate/recommended-policies.yaml

The recommended-policies.yaml file contains the recommended policies that are generated. As before, apply these through your regular deployment pipeline, or using kubectl:

$ kubectl apply -f recommended-policies.yaml

At the end of this stage, you have deployed policies which will allow you to safely enroll your workloads in ambient mode.

A note on authorization policy

AuthorizationPolicy in particular requires careful handling due to the changes in how policies are enforced in ambient mode. In ambient mode, Layer 4 and Layer 7 policies can be enforced against services, applied at the waypoint. Additionally, Layer 4 policies can be enforced against workloads.

This requires the following changes:

• If a workload policy contains HTTP attributes, a waypoint is required, as recommended in the previous phase. The policy will then need to be moved to configure the waypoint, rather than the workload.
• If a waypoint is deployed, the workload must allow traffic from the waypoint.

The migration tool will help move policies to the waypoints as required, and additionally generate policies to allow traffic from the waypoints.

Recommendation details

Note that, at this point, all policy recommendations are in addition to the existing policies. This ensures that during the migration process, policies continue to apply to all traffic requests, regardless of whether traffic is routed through a waypoint. Later phases will transition traffic to start flowing through the waypoint, and then optionally recommend cleaning up the unused policies once the migration is complete.

For example, consider a case where an authorization policy allows a single client to access an application:

When a waypoint is deployed, the tool recommends additional policies to support the same behavior for traffic from the waypoint, in addition to traffic that does not go through the waypoint. This ensures a zero-downtime transition.

Enable waypoints

Now that waypoints have been deployed and configured, you can start to use them. The tool will detect which waypoints you need to enable and give steps to enable them.

Once enabled, traffic will start to flow through the waypoint. This can be done on a service-by-service basis to allow you to monitor the health of the system as you progress. There is no need to quickly move everything at once. If a service is behaving unexpectedly, the waypoint enablement can be reverted. See the waypoint configuration documentation for more details.

An example cluster where one waypoint is ready for enablement:

$ gloo ambient migrate --phase use-waypoints
• Starting phase use-waypoints...
⚠️ Phase use-waypoints has recommendations!
  ⚠️ Warning: detected waypoint app1/waypoint but it is not used by anything
  🔮 Detected namespace app1 requires a waypoint, but not configured to use one. Configure it with: kubectl label namespace app1 istio.io/use-waypoint=waypoint

Enable the waypoint with the command provided:

$ kubectl label namespace app1 istio.io/use-waypoint=waypoint

Traffic will now flow from the client-side sidecar to the waypoint — without any local routing or policy decisions — and then to the server-side sidecar. Each connection will be separately encrypted with mTLS.

Sequencing waypoint enablement and sidecar removal

It is recommended that all waypoints are enabled before you start to remove sidecars from workloads. However, this is not strictly required, which may be useful in scenarios where some workloads will take a while to complete migration.

In order for all policies to be correctly applied, the request flow needs either:

• A sidecar on the client and server side
• A waypoint proxy enabled

This means that it is not suitable to simply remove sidecars from a namespace once that namespace has a waypoint enabled, because clients in that namespace may still call servers in other namespaces that do not yet have an enabled waypoint.

The following diagram illustrates how a client sidecar successfully communicates with a migrated namespace and with a namespace that is not yet migrated. If you were to remove the client sidecar before the unmigrated namespace is migrated, some policies would not be correctly applied to traffic requests.

Simplify policy

At this point, all traffic will be flowing through waypoints. This means that policies can be simplified by removing any policies that are specific to sidecars, as they will no longer be relevant.

In the previous step, you deployed copies of existing policies. The original sources of these can now be removed.

$ gloo ambient migrate --phase policy-simplification
• Starting phase policy-simplification...
⚠️ Phase policy-simplification has recommendations!
  🔮 AuthorizationPolicy was migrated and can be deleted: kubectl delete authorizationpolicies.security.istio.io -n app1 allow-nothing
  🔮 AuthorizationPolicy was migrated and can be deleted: kubectl delete authorizationpolicies.security.istio.io -n app1 legacy-policies
  🔮 AuthorizationPolicy was migrated and can be deleted: kubectl delete authorizationpolicies.security.istio.io -n app1 policies

Remove sidecars

When all services have been configured to use waypoints, and all waypoints have been configured to apply equivalent configurations, it is safe to remove the sidecars from workloads.

The sidecar removal phase will detect workloads still using sidecars and recommend moving to ambient mode:

$ gloo ambient migrate --phase remove-sidecars
• Starting phase remove-sidecars...
⚠️ Phase remove-sidecars has recommendations!
  🔮 namespace app1 can disable its sidecars: kubectl label ns app1 istio-injection- istio.io/dataplane-mode=ambient
  ⚠️ sidecar found for Pod app1/echo-7fb78cb7c5-plsjl

To remove injection from a namespace and add it to the ambient mesh, execute the commands as recommended:

$ kubectl label ns app1 istio-injection- istio.io/dataplane-mode=ambient

When your applications are next restarted or redeployed, they will run without sidecars, and their traffic will be captured by ztunnel.

Appendix: About the migration tool

The ambient migration tool will inspect the state of the cluster and make recommendations about next steps. Each phase of the migration runs a series of checks, and validates that the checks are completed before proceeding. If checks in a phase fail, the tool gives recommendations for how to resolve them. Details about each phase are documented above.

The migration process involves running gloo ambient migrate a number of times. Each time, the tool will analyze the current state of the cluster and recommend next steps.

There are a few useful options to consider: