feat(http/retry): model `ReplayBody` with `Frame<T>` #3598

cratelyn · 2025-02-06T03:59:58Z

pr #3559 (dd4fbcd) refactored our trailer peeking body middleware to
model its buffering in terms of the Frame<T> type used in
http-body's 1.0 release.

this branch performs a similar change for the other piece of body
middleware that super linkerd's retry facilities: ReplayBody. the
inner body B is now wrapped in the ForwardCompatibleBody adapter,
and we now poll it in terms of frames.

NB: polling the underlying in terms of frames has a subtle knock-on
effect regarding when we observe the trailers, in the liminal period
between this refactor and the subsequent upgrade to hyper 1.0, whilst we
must still implement the existing 0.4 interface for Body that includes
poll_trailers().

see the comment above replay_trailers for more on this, describing why
we now initialize this to true. relatedly, this is why we no longer
delegate down to B::poll_trailers ourselves. it will have already been
called by our adapter.

ReplayBody::is_end_stream() now behaves identically when initially
polling a body compared to subsequent replays. this is fine, as
is_end_stream() is a hint that facilitates optimizations
(hyperium/http-body#143). we do still report the end properly, we just
won't be quite as prescient on the initial playthrough.

in the same manner as the existing frame() method mimics
http_body_util::BodyExt::frame(), this branch introduces
a new ForwardCompatibleBody::poll_frame() method.

this allows us to poll the compatibility layer for a Frame<T>.

see:

linkerd/http/retry/src/replay.rs

some tests do not set up a tracing subscriber, because they do not use the shared `Test::new()` helper function used elsewhere in this test suite. to provide a trace of the test's execution in the event of a failure, initialize a tracing subscriber in some additional unit tests. Signed-off-by: katelyn martin <kate@buoyant.io>

this commit removes the `cfg(test)` gate on the method exposing `B::is_end_stream()`, and introduces another method also exposing the `size_hint()` method. we will want these in order to implement these methods for `ReplayBody`. Signed-off-by: katelyn martin <kate@buoyant.io>

in the same manner as the existing `frame()` method mimics `http_body_util::BodyExt::frame()`, this commit introduces a new `ForwardCompatibleBody::poll_frame()` method. this allows us to poll the compatibility layer for a `Frame<T>`. Signed-off-by: katelyn martin <kate@buoyant.io>

pr #3559 (dd4fbcd) refactored our trailer peeking body middleware to model its buffering in terms of the `Frame<T>` type used in `http-body`'s 1.0 release. this commit performs a similar change for the other piece of body middleware that super linkerd's retry facilities: `ReplayBody`. the inner body `B` is now wrapped in the `ForwardCompatibleBody` adapter, and we now poll it in terms of frames. NB: polling the underlying in terms of frames has a subtle knock-on effect regarding when we observe the trailers, in the liminal period between this refactor and the subsequent upgrade to hyper 1.0, whilst we must still implement the existing 0.4 interface for `Body` that includes `poll_trailers()`. see the comment above `replay_trailers` for more on this, describing why we now initialize this to `true`. relatedly, this is why we now longer delegate down to `B::poll_trailers` ourselves. it will have already been called by our adapter. `ReplayBody::is_end_stream()` now behaves identically when initially polling a body compared to subsequent replays. this is fine, as `is_end_stream()` is a hint that facilitates optimizations (hyperium/http-body#143). we do still report the end properly, we just won't be quite as prescient on the initial playthrough. see: - linkerd/linkerd2#8733. - #3559 Signed-off-by: katelyn martin <kate@buoyant.io>

this commit introduces some trace-level diagnostics tracking how the replay body has determined whether or not it has reached the end of the stream. Signed-off-by: katelyn martin <kate@buoyant.io>

Signed-off-by: katelyn martin <kate@buoyant.io>

note: this commit will not compile, code changes are intentionally elided from this commit. this commit upgrades hyper, http, tonic, prost, related dependencies, and their assorted cargo features. see <linkerd/linkerd2#8733>. see also: * #3379 * #3380 * #3382 * #3405 * hyperium/hyper#3796 * #3411 * #3421 * #3427 * #3428 * #3432 * #3433 * #3444 * #3445 * #3454 * #3455 * #3456 * #3457 * #3461 * #3459 * #3465 * #3466 * #3467 * #3468 * linkerd/linkerd2-proxy-api#421 * linkerd/linkerd2#13492 * linkerd/linkerd2#13493 * hyperium/hyper#3816 * #3472 * #3473 * #3479 * tokio-rs/tokio#7059 * #3509 * hyperium/http-body#140 * #3515 * hyperium/http-body#141 * #3530 * #3531 * #3540 * #3556 * #3558 * #3559 * #3564 * #3567 * #3573 * #3583 * hyperium/http-body#144 * #3585 * #3586 * #3597 * #3598 * #3611 * #3614 * #3615 * #3616 * #3647 * #3651 * #3653 * #3654 * #3655 * #3656 * #3657 * #3660 * #3671 * #3672 * #3673 * #3676 * hyperium/http-body#147 * #3692 * #3699 * #3700 * #3701 * #3708 * linkerd/drain-rs#36 * #3715 * #3717 * eminence/procfs#340 --- squash: chore(deps): add hyper-util workspace dependency chore(deps): add http-body-util workspace dependency chore(deps): upgrade linkerd2-proxy-api this commit represents main as of linkerd/linkerd2-proxy-api#421. Signed-off-by: katelyn martin <kate@buoyant.io>

## ✨ chore(deps): upgrade to hyperium 1.x crates this branch performs an exciting upgrade for our proxy. this branch upgrades a number of our dependencies so that we use the 1.0 release family of the `hyper` http framework, and its ecosystem. see the [v1.0 announcement][hyper-v1] for more information. this branch upgrades the following dependencies: * `h2`: 0.3 -> 0.4 * `http`: 0.2 -> 1 * `http-body`: 0.4 -> 1 * `hyper`: 0.14.32 -> 1 * `prost`: 0.12 -> 0.13 * `prost-build`: 0.12 -> 0.13 * `prost-types`: 0.12 -> 0.13 * `tonic`: 0.10 -> 0.12 * `tonic-build`: 0.10 -> 0.12 a `hyper-util` dependency is added, which provides among other things, legacy-compatible interfaces such as `hyper_util::client::legacy::Client`, or glue to use `hyper` with the tokio runtime. see <https://docs.rs/hyper-util/latest/hyper_util/> for more information. a `http-body-util` dependency is added, which provides a `BodyExt` trait and a channel-backed body for use in unit tests. the `deprecated` feature flag that was active on our `0.14` hyper dependency has been removed, along with the `stream` and `runtime` feature flags. the `linkerd2-proxy-api` dependency is updated. see: <linkerd/linkerd2-proxy-api#421> ### 📝 notes for review bear particular attention to changes involving `http_body::Body` middleware. the change from two separate `poll_data()` and `poll_trailers()` functions, to a single `poll_frame()` method, induces some subtle changes to various pieces of middleware. also bear in mind that failing to set a timer, in our case `hyper_util::rt::TokioTimer`, can cause http/2 clients, or http/1 and http/2 servers, to panic. make sure that any uses of `hyper::server::conn::http1::Builder`, `hyper::client::conn::http1::Builder`, or `hyper::client::conn::http2::Builder` install a timer. ### ❗ breaking change: `l5d-proxy-error` values the `l5d-proxy-error` header can be examined to observe the cause of proxy errors encountered when sending meshed traffic. by virtue of this using a newer `hyper` client in the proxy, some error messages may in turn look different. for example, an error like `"connect timed out after 1s"` may now appear as `"client error (Connect)"`. ### 📚 other notes this work, by virtue of touching so many parts of the system, is carried out in distinct commits. an initial commit upgrades the dependencies at th workspace level. subsequent commits will not compile if the `--workspace` flag is provided, but the intent of this branch is to update each crate individually. use commands like, e.g. `cargo check --tests -p linkerd-proxy-http` to build particular crates at intermediate commits within this branch. this commit is also only the final leaf in an _extended_ line of work. this has been done to mitigate the effort of reviewing this change, and the risk of churn in the event of any unanticipated errors. see the top-level comment in linkerd/linkerd2#8733 for an overview of all of the work that brought us to this juncture. [hyper-v1]: https://seanmonstar.com/blog/hyper-v1/ --- * chore(deps): upgrade to hyper 1.x note: this commit will not compile, code changes are intentionally elided from this commit. this commit upgrades hyper, http, tonic, prost, related dependencies, and their assorted cargo features. see <linkerd/linkerd2#8733>. see also: * #3379 * #3380 * #3382 * #3405 * hyperium/hyper#3796 * #3411 * #3421 * #3427 * #3428 * #3432 * #3433 * #3444 * #3445 * #3454 * #3455 * #3456 * #3457 * #3461 * #3459 * #3465 * #3466 * #3467 * #3468 * linkerd/linkerd2-proxy-api#421 * linkerd/linkerd2#13492 * linkerd/linkerd2#13493 * hyperium/hyper#3816 * #3472 * #3473 * #3479 * tokio-rs/tokio#7059 * #3509 * hyperium/http-body#140 * #3515 * hyperium/http-body#141 * #3530 * #3531 * #3540 * #3556 * #3558 * #3559 * #3564 * #3567 * #3573 * #3583 * hyperium/http-body#144 * #3585 * #3586 * #3597 * #3598 * #3611 * #3614 * #3615 * #3616 * #3647 * #3651 * #3653 * #3654 * #3655 * #3656 * #3657 * #3660 * #3671 * #3672 * #3673 * #3676 * hyperium/http-body#147 * #3692 * #3699 * #3700 * #3701 * #3708 * linkerd/drain-rs#36 * #3715 * #3717 * eminence/procfs#340 --- squash: chore(deps): add hyper-util workspace dependency chore(deps): add http-body-util workspace dependency chore(deps): upgrade linkerd2-proxy-api this commit represents main as of linkerd/linkerd2-proxy-api#421. Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/box): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(hyper-balance): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/retain): ugrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/stream-timeouts): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/classify): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/upgrade): upgrade to hyper 1.x NOTE: there is a comment noting that the upgrade middleware does not expect to be cloneable. it is unfortunately, however, at odds with the new bounds expected of extensions. so, `Http11Upgrade` is now Clone'able, but a comment is left in place noting this weakened invariant. it's worth investigating how upgrades have changed since, in more detail, but for the current moment we are interested in being especially conservative about changing behavior, and focusing on api changes like `Body::poll_frame(..)`. Signed-off-by: katelyn martin <kate@buoyant.io> * chore(metrics): upgrade to hyper 1.x a brief note; this commit happened to tickle an unfortunate sharp edge in `BoxBody` and `Full`'s respective constructors. type inference could not figure out how to construct the body, so we refrain from boxing the response body now. Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/metrics): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/prom): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/insert): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/retry): deprecate linkerd-http-body-compat Signed-off-by: katelyn martin <kate@buoyant.io> * chore(mock/http-body): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/retry): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(proxy/tap): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(proxy/http): update to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(app/core): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(app/test): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(app/admin): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(app/outbound): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(app/inbound): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(app/integration): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(app): upgrade to hyper 1.x Signed-off-by: katelyn martin <kate@buoyant.io> * chore(transport-header): update generated code Signed-off-by: katelyn martin <kate@buoyant.io> * chore(spiffe-proto): update generated code Signed-off-by: katelyn martin <kate@buoyant.io> * chore(opencensus-proto): update generated code Signed-off-by: katelyn martin <kate@buoyant.io> * chore(opentelemetry-proto): update generated code Signed-off-by: katelyn martin <kate@buoyant.io> * chore(deny.toml): update cargo-deny directives this commit updates the contents of `deny.toml`. Signed-off-by: katelyn martin <kate@buoyant.io> * chore: `compile` has been renamed to `compile_protos` this addresses deprecation warnings, updating calls to a function that has since been renamed. Signed-off-by: katelyn martin <kate@buoyant.io> * chore(deps): remove `linkerd-http-body-compat` dependencies this commit removes this crate, which we added to future proof code for this upgrade, from its dependents. Signed-off-by: katelyn martin <kate@buoyant.io> * chore(http/body-compat): remove `linkerd-http-body-compat` crate Signed-off-by: katelyn martin <kate@buoyant.io> * chore(deps): update to drain 0.2.1 see linkerd/drain-rs#41. Signed-off-by: katelyn martin <kate@buoyant.io> --------- Signed-off-by: katelyn martin <kate@buoyant.io>

the initial replay body, circa the usage of our "compatibility" layer (4b53081, #3598), used to need an extra poll to confirm the absence of trailers before it would report itself as reaching the end of the stream. these tests were added in (afda8a7, #3583). this was an artifact of how the compatibility middleware masked the previous `poll_data()` and `poll_trailer()` methods behind a forward-compatible `poll_frame()`- and `frame()`-oriented interface. this commit removes these extra calls to `initial.frame().await`, now that the initial body will report the end of stream without an extra call to await a `None`. X-ref: #3598 X-ref: #3583 Signed-off-by: katelyn martin <kate@buoyant.io>

cratelyn mentioned this pull request Feb 6, 2025

proxy: Figure out the plan for Hyper 1.0 migration linkerd/linkerd2#8733

Closed

cratelyn marked this pull request as ready for review February 6, 2025 05:14

cratelyn requested a review from a team as a code owner February 6, 2025 05:14

olix0r approved these changes Feb 6, 2025

View reviewed changes

linkerd/http/retry/src/replay.rs Outdated Show resolved Hide resolved

Base automatically changed from kate/http-retry.outline-the-replay-buffer to main February 6, 2025 15:50

cratelyn added 6 commits February 6, 2025 10:51

feat(http/retry): is_end_stream() traces

4566b8e

this commit introduces some trace-level diagnostics tracking how the replay body has determined whether or not it has reached the end of the stream. Signed-off-by: katelyn martin <kate@buoyant.io>

nit(http/retry): capitalize trace event messages

25bb2b1

Signed-off-by: katelyn martin <kate@buoyant.io>

cratelyn force-pushed the kate/http-retry.new-frame-polling-same-great-taste branch from 28d2967 to 25bb2b1 Compare February 6, 2025 15:53

cratelyn enabled auto-merge (squash) February 6, 2025 15:54

cratelyn merged commit 4b53081 into main Feb 6, 2025
15 checks passed

cratelyn deleted the kate/http-retry.new-frame-polling-same-great-taste branch February 6, 2025 16:00

cratelyn mentioned this pull request Apr 21, 2025

nit(http/retry): remove extra body poll in tests #3877

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(http/retry): model `ReplayBody<B>` with `Frame<T>` #3598

feat(http/retry): model `ReplayBody<B>` with `Frame<T>` #3598

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat(http/retry): model ReplayBody<B> with Frame<T> #3598

feat(http/retry): model ReplayBody<B> with Frame<T> #3598

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat(http/retry): model `ReplayBody<B>` with `Frame<T>` #3598

feat(http/retry): model `ReplayBody<B>` with `Frame<T>` #3598