Configure and view metrics
Metrics provide a way of monitoring and understanding behavior in aggregate. The two layers of ambient mesh generate different metrics:
- ztunnel generates TCP metrics for all service traffic
- Waypoint proxies generates request and response metrics for HTTP, HTTP/2 and gRPC traffic.
To monitor service behavior, Istio generates metrics for all service traffic in, out, and within an Istio service mesh. These metrics provide information on behaviors such as the overall volume of traffic, the error rates within the traffic, and the response times for requests.
In addition to monitoring the behavior of services within a mesh, it is also important to monitor the behavior of the mesh itself. Istio components export metrics on their own internal behaviors to provide insight on the health and function of the mesh control plane.
Metric sources
Istio provides a set of service-oriented metrics for monitoring service communications. These metrics cover the four basic service monitoring needs: latency, traffic, errors, and saturation.
Each proxy generates a rich set of metrics about all traffic passing through the proxy (both inbound and outbound). The proxies also provide detailed statistics about the administrative functions of the proxy itself, including configuration and health information.
Metrics are accessible by making an HTTP request to either “/stats/prometheus” or “/metrics” on port 15020.
Istio proxies are marked up with prometheus.io
annotations, which mean they will be scraped by default by Prometheus installations.
Install Prometheus
Managing a production-scale monitoring stack is a skill in and of itself. Ambient mesh works with any Prometheus-compatible observability stack, including the Prometheus Operator and commercial offerings. Istio provides a demo installaton of Prometheus for learning about how to observe your mesh.
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/addons/prometheus.yaml
Query and graph metrics with Prometheus
If you’ve installed Prometheus with the Istio add-on, you can use istioctl
to access the Prometheus console:
$ istioctl dashboard prometheus
You can enter queries in the PromQL language. For example:
- Total count of all requests to the
productpage
service:istio_requests_total{destination_service="productpage.default.svc.cluster.local"}
- Total count of all requests to
v3
of thereviews
service:istio_requests_total{destination_service="reviews.default.svc.cluster.local", destination_version="v3"}
- Rate of requests over the past 5 minutes to all instances of the
productpage
service:rate(istio_requests_total{destination_service=~"productpage.*", response_code="200"}[5m])
View metrics with Grafana
Istio comes with Grafana dashboards which offer summaries of Istio’s metrics, allowing you to drill down by service or workload.
To explore Grafana, use the add-on that comes with Istio:
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.23/samples/addons/grafana.yaml
You can launch the dashboard with istioctl
:
$ istioctl dashboard grafana
Click “Dashboards” in the left hand menu, and then select “istio” from the list on the right hand pane. There are dashboards for control plane, service, workload and performance, all of which can be copied and customized to your requirements.
You can also add the dashboards to your own Grafana installation by downloading them from grafana.com.
Differences between the layers
ztunnel
Ztunnel exposes the TCP metrics from Istio’s standard metrics, as well as a variety of DNS and debugging metrics.
An example ztunnel metric:
istio_tcp_connections_opened_total{
reporter="source",
source_workload="productpage-v1",
source_canonical_service="productpage",
source_canonical_revision="v1",
source_workload_namespace="default",
source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",
source_app="productpage",
source_version="v1",
source_cluster="Kubernetes",
destination_service="reviews.default.svc.cluster.local",
destination_service_namespace="default",
destination_service_name="reviews",
destination_workload="waypoint",
destination_canonical_service="waypoint",
destination_canonical_revision="latest",
destination_workload_namespace="default",
destination_principal="spiffe://cluster.local/ns/default/sa/waypoint",
destination_app="waypoint",
destination_version="latest",
destination_cluster="Kubernetes",
request_protocol="tcp",
response_flags="-",
connection_security_policy="mutual_tls"
} 1
Waypoint proxies
Waypoint proxies expose the full set of Istio metrics.
An example waypoint metric:
istio_requests_total {
reporter="waypoint",
source_workload="productpage-v1",
source_canonical_service="productpage",
source_canonical_revision="v1",
source_workload_namespace="default",
source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",
source_app="productpage",
source_version="v1",
source_cluster="Kubernetes",
destination_workload="details-v1",
destination_workload_namespace="default",
destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-details",
destination_app="details",
destination_version="v1",
destination_service="details.default.svc.cluster.local",
destination_canonical_service="details",
destination_canonical_revision="v1",
destination_service_name="details",
destination_service_namespace="default",
destination_cluster="Kubernetes",
request_protocol="http",
response_code="200",
grpc_response_status="",
response_flags="-",
connection_security_policy="mutual_tls"
} 1
Waypoint proxies also expose Envoy statistics as metrics.