System and app requirements

If you are using EKS:

• with Amazon’s VPC CNI
• with Pod ENI trunking enabled
• and you are using EKS pod-attached SecurityGroups via SecurityGroupPolicy

POD_SECURITY_GROUP_ENFORCING_MODE must be explicitly set to standard, or pod health probes (which are by-default silently exempted from all policy enforcement by the VPC CNI) will fail. This is because Istio uses a link-local SNAT address for kubelet health probes, which Amazon’s VPC CNI is not aware of, and the VPC CNI does not have an option to exempt link-local addresses from policy enforcement.

Refer to the Istio documentation for details.

On GKE, critical components can only be installed into a namespace with a resource quota defined. To install ambient mesh in the default istio-system namespace, first apply this resource:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gcp-critical-pods
  namespace: istio-system
spec:
  hard:
    pods: 1000
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - system-node-critical

Refer to the Istio documentation for details.

Add -n kube-system --set global.platform=openshift to your Helm installation commands to set the required configuration for OpenShift.

$ helm install istio-cni istio/cni -n kube-system --set profile=ambient --set global.platform=openshift --wait

Add -n kube-system --set global.platform=k3d to your Helm installation commands to set the required configuration for k3d. For example:

$ helm install istio-cni istio/cni -n istio-system --set profile=ambient --set global.platform=k3d --wait

Add -n kube-system --set global.platform=k3s to your Helm installation commands to set the required configuration for k3s. For example:

$ helm install istio-cni istio/cni -n istio-system --set profile=ambient --set global.platform=k3s --wait

Add -n kube-system --set global.platform=microk8s to your Helm installation commands to set the required configuration for microk8s. For example:

$ helm install istio-cni istio/cni -n istio-system --set profile=ambient --set global.platform=microk8s --wait

Add -n kube-system --set global.platform=minikube to your Helm installation commands to set the required configuration for minikube. For example:

$ helm install istio-cni istio/cni -n istio-system --set profile=ambient --set global.platform=minikube --wait

• Cilium currently defaults to proactively deleting other CNI plugins and their config, and must be configured with cni.exclusive = false to properly support chaining. See the Cilium documentation for more details.

• Cilium’s BPF masquerading is currently disabled by default, and has issues with Istio’s use of link-local IPs for Kubernetes health checking. Enabling BPF masquerading via bpf.masquerade=true is not currently supported, and results in non-functional pod health checks in Istio ambient. Cilium’s default iptables masquerading implementation should continue to function correctly.

• Due to how Cilium manages node identity and internally allow-lists node-level health probes to pods, applying any default-DENY NetworkPolicy in a Cilium CNI install underlying Istio in ambient mode will cause kubelet health probes (which are by-default silently exempted from all policy enforcement by Cilium) to be blocked. This is because Istio uses a link-local SNAT address for kubelet health probes, which Cilium is not aware of, and Cilium does not have an option to exempt link-local addresses from policy enforcement.

  This can be resolved by applying the following CiliumClusterWideNetworkPolicy:

  apiVersion: "cilium.io/v2"
  kind: CiliumClusterwideNetworkPolicy
  metadata:
    name: "allow-ambient-hostprobes"
  spec:
    description: "Allows SNAT-ed kubelet health check probes into ambient pods"
    enableDefaultDeny:
      egress: false
      ingress: false
    endpointSelector: {}
    ingress:
    - fromCIDR:
      - "169.254.7.127/32"

  This policy override is not required unless you already have other default-deny NetworkPolicies or CiliumNetworkPolicies applied in your cluster.

  Please see issue #49277 and CiliumClusterWideNetworkPolicy for more details.