Enable tracing

Distributed tracing provides a way to monitor and understand behavior by monitoring individual requests as they flow through a mesh. Traces empower mesh operators to understand service dependencies and the sources of latency within their service mesh.

Ambient mesh supports distributed tracing through waypoint proxies. The proxies automatically generate trace spans on behalf of the applications they proxy, requiring only that the applications forward the appropriate request context.

In this guide you will use Jaeger, an open source, distributed tracing platform. The steps will largely be the same for any other similar platform.

Configuring an extension provider

To send traces, you must configure an extension provider. Istio supports OpenTelemetry, as well as a number of legacy tracing providers such as Zipkin, Datadog and Apache SkyWalking.

If you followed the Quickstart guide, an OpenTelemetry provider called jaeger-tracing is configured for you already. If you are installing with Helm, use a values.yaml file to update the meshConfig when installing the istiod chart:

meshConfig:
  extensionProviders:
  - name: jaeger-tracing
    opentelemetry:
      port: 4317
      service: jaeger-collector.istio-system.svc.cluster.local

Install Jaeger

Managing a production-scale observability stack is a skill in and of itself. For tracing, options include the Jaeger Operator and various commercial offerings. Istio provides a sample installaton of Jaeger for learning about how to observe your mesh:

$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/jaeger.yaml

This sample runs a service called jaeger-collector which listens for OpenTelemetry traces (via gRPC) on port 4317. This matches the parameters set in the extension provider above.

You can launch the Jaeger dashboard with istioctl:

$ istioctl dashboard jaeger

Configure tracing with the Telemetry API

Telemetry policies are attached similarly to routes using the targetRefs field to select a Gateway.

With an extension provider configured, you use the Telemetry API to configure what percentage of requests you want sent to the tracing backend. You can also customize other options, including setting tags on trace spans.

For production use, most users who use distributed tracing will want sampling at a rate between 1 and 10%. For debugging or demonstrating, it is useful to set the sampling percentage to 100%, i.e. all traffic.

Enable tracing on a gateway

When requests enter the mesh, a tracing header must be added. Enable tracing at the gateway:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: telemetry-gateway
  namespace: default
spec:
  targetRefs:
  - kind: Gateway
    name: bookinfo-gateway
    group: gateway.networking.k8s.io
  tracing:
  - providers:
    - name: "jaeger-tracing"
    randomSamplingPercentage: 100

Enable tracing on waypoints

When using waypoints, you are able to capture trace information for subsequent services that are called. Enable tracing at the waypoint:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: telemetry-waypoints
  namespace: default
spec:
  targetRefs:
  - kind: Gateway
    name: waypoint
    group: gateway.networking.k8s.io
  tracing:
  - providers:
    - name: "jaeger-tracing"
    randomSamplingPercentage: 100

View traces

Refresh the Bookinfo productpage a few times, then go to the Jaeger dashboard.

The “Service” drop-down is populated with services that Jaeger has received traces for. Given that all the requests start at the gateway, select bookinfo-gateway-istio.default and click “Find traces”.

In this example, there are two traces that are only between the gateway and the productpage service, relating to static files. You can open the trace to see the file that is requested:

There is also a trace that displays the calls that are made from the productpage service, as reported by the waypoint proxy: productpage calls details and reviews, and reviews calls ratings.

Cleaning up

Remove the Telemetry objects to disable tracing.

$ kubectl delete telemetry telemetry-gateway
$ kubectl delete telemetry telemetry-waypoints