[serve] split call_user_method #54104
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Cindy Zhang <cindyzyx9@gmail.com>
Signed-off-by: Cindy Zhang <cindyzyx9@gmail.com>
Signed-off-by: Cindy Zhang <cindyzyx9@gmail.com>
32bc5ae to
f5542e0
Compare
Signed-off-by: Cindy Zhang <cindyzyx9@gmail.com>
Signed-off-by: Cindy Zhang <cindyzyx9@gmail.com>
There was a problem hiding this comment.
Pull Request Overview
This PR splits the single call_user_method into three distinct functions to properly handle HTTP entrypoints, streaming generators, and non-streaming user methods.
- Updated test cases to call call_http_entrypoint and call_user_generator instead of call_user_method for streaming or HTTP requests.
- Modified ReplicaBase internals to conditionally call the new methods based on request metadata.
- Adjusted local testing mode to support the new call_user_generator and call_http_entrypoint functions.
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| python/ray/serve/tests/unit/test_user_callable_wrapper.py | Updated test invocations to use the appropriate call_ functions. |
| python/ray/serve/_private/replica.py | Splits the call_user_method logic into call_http_entrypoint, call_user_generator, and call_user_method. |
| python/ray/serve/_private/local_testing_mode.py | Adjusted handling of streaming requests to call the correct function. |
Comments suppressed due to low confidence (1)
python/ray/serve/_private/replica.py:609
- [nitpick] The variable 'call_user_method_future' now holds the result from either call_http_entrypoint or call_user_generator. Consider renaming it (e.g., 'call_user_future') to better reflect its usage.
)
|
the refactor makes sense. left some nits. |
abrarsheikh
reviewed
Jun 26, 2025
python/ray/serve/_private/replica.py
Outdated
| *, | ||
| generator_result_callback: Optional[Callable] = None, | ||
| ) -> Any: | ||
| """Call a user method (unary or generator). |
There was a problem hiding this comment.
does this method handle generator case for GRPC / deployment_handle calls?
Signed-off-by: Cindy Zhang <cindyzyx9@gmail.com>
abrarsheikh
approved these changes
Jun 26, 2025
minerharry
pushed a commit
to minerharry/ray
that referenced
this pull request
Jun 27, 2025
## Why are these changes needed? Split `UserCallableWrapper.call_user_method` into three functions: 1. `call_http_entrypoint` - processes all requests with `request_metadata.is_http_request == True` 2. `call_user_generator` - processes all requests with `request_metadata.is_http_request == False and request_metadata.is_streaming == True` 3. `call_user_method` - processes all requests with `request_metadata.is_streaming == False` Note that - `ReplicaBase.handle_request` directly calls `call_user_method` - `ReplicaBase.handle_request_with_streaming` calls `ReplicaBase._call_streaming` which calls `call_http_entrypoint` if `request_metadata.is_http_request == True` otherwise calls `call_user_generator` - `ReplicaBase.handle_request_with_rejection` calls `call_user_method` if `request_metadata.is_streaming == False` otherwise calls `ReplicaBase._call_streaming`. --------- Signed-off-by: Cindy Zhang <cindyzyx9@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Split
UserCallableWrapper.call_user_methodinto three functions:call_http_entrypoint- processes all requests withrequest_metadata.is_http_request == Truecall_user_generator- processes all requests withrequest_metadata.is_http_request == False and request_metadata.is_streaming == Truecall_user_method- processes all requests withrequest_metadata.is_streaming == FalseNote that
ReplicaBase.handle_requestdirectly callscall_user_methodReplicaBase.handle_request_with_streamingcallsReplicaBase._call_streamingwhich callscall_http_entrypointifrequest_metadata.is_http_request == Trueotherwise callscall_user_generatorReplicaBase.handle_request_with_rejectioncallscall_user_methodifrequest_metadata.is_streaming == Falseotherwise callsReplicaBase._call_streaming.