Description
This was originally reported on "Discuss Istio" - I was asked to create a github issue regarding this bug. See: https://discuss.istio.io/t/need-source-telemetry-to-capture-the-first-hop-when-mirroring-traffic/369
Describe the bug
I have traffic mirroring set up in bookinfo (see this yaml if you are interested) where productpage-v1 sends its requests to reviews-v1 but mirrors to reviews-v2.
Visualizing the resulting telemetry in Kiali looks like this:
Notice that “first hop” in the mirrored traffic is missing - that is to say, the request going from productpage-v1 to reviews-v2 is missing. This is because there is no reporter=“source” metric. However, as that mirrored traffic flows “downstream”, there is source telemetry for the rest of the request as it flows from service to service (which is why you see edges from reviews-v2 to ratings-v1 and -v2 as well as ratings-v2 to mongodb-v1).
The Istio implementation should be changed so the full mirrored traffic (starting at that “first hop”) is represented by source telemetry. As it is now, the source telemetry has a “hole” in it as you see when the telemetry is visualized.
Side note: There is reporter=“destination” telemetry for that first hop - but that is from the point of view of the reviews-v2 workload (thus reporter=“destination”). The Kiali graph is visualizing reporter=“source” telemetry because that is the only side that provides information about client-side errors (like injected faults, network errors, etc).
Expected behavior
I expect traffic mirroring to have the same telemetry as "normal" traffic. Specifically, I expected to see source telemetry (reporter="source") for the "first hop" of the mirrored traffic.
Steps to reproduce the bug
- Install bookinfo demo
- Add the virtual service/destination rule that turns on mirroring using this yaml.
- Send one request from your browser to the bookinfo's productpage web page. (The result of this is that the virtual service sends the "normal" request to reviews-v1 but mirrors traffic to reviews-v2).
- Look in Prometheus for all timeseries metrics dealing with the mirrored traffic going to reviews-v2. You do this using the query
istio_requests_total{destination_workload=“reviews-v2”}. Notice there is a missing reporter="source" timeseries - there is only a reporter="destination" timeseries. You will see something like this:
istio_requests_total{connection_security_policy="none",
destination_app="reviews",
destination_principal="unknown",
destination_service="reviews.bookinfo.svc.cluster.local",
destination_service_name="reviews",
destination_service_namespace="bookinfo",
destination_version="v2",
destination_workload="reviews-v2",
destination_workload_namespace="bookinfo",
instance="172.17.0.12:42422",
job="istio-mesh",
reporter="destination",
request_protocol="http",
response_code="200",
source_app="productpage",
source_principal="unknown",
source_version="v1",
source_workload="productpage-v1",
source_workload_namespace="bookinfo"}
Version
Istio 1.0.5
Installation
Using helm
Metadata
Metadata
Assignees
Labels
Type
Projects
Status