YARN-11358. [Federation] Add FederationInterceptor#allow-partial-result config. #5056
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
We have something similar with RBF. |
Thanks a lot for your suggestion, I refer to RBF's code, I named this optional parameter |
|
🎊 +1 overall
This message was automatically generated. |
|
@goiri Can you help review this pr? Thank you very much! |
goiri
reviewed
Oct 26, 2022
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
Outdated
Show resolved
Hide resolved
| @@ -2024,9 +2030,15 @@ private <R> Map<SubClusterInfo, R> invokeConcurrent(Collection<SubClusterInfo> c | |||
| Map<SubClusterInfo, R> results = new HashMap<>(); | |||
|
|
|||
| // Send the requests in parallel | |||
| CompletionService<R> compSvc = new ExecutorCompletionService<>(this.threadpool); | |||
| CompletionService<Pair<R, Exception>> compSvc = | |||
| if (response != null) { | ||
| results.put(clusterId, response); | ||
| } | ||
| Exception exception = pair.getRight(); |
There was a problem hiding this comment.
In the API seems to be getValue()?
|
|
||
| Configuration conf = this.getConf(); | ||
| // Enable strict mode | ||
| conf.setBoolean(YarnConfiguration.ROUTER_INTERCEPTOR_ALLOW_PARTIAL_RESULT_ENABLED, |
| @@ -50,5 +50,4 @@ protected void registerBadSubCluster(SubClusterId badSC) { | |||
| badSC); | |||
| interceptor.setRunning(false); | |||
| } | |||
|
|
|||
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@goiri Can you help review this pr again? Thank you very much! |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
goiri
approved these changes
Nov 23, 2022
goiri
reviewed
Nov 23, 2022
| setupCluster(Arrays.asList(bad2)); | ||
|
|
||
| LambdaTestUtils.intercept(YarnRuntimeException.class, "RM is stopped", |
There was a problem hiding this comment.
Can we keep the old test and add a new one with the partial enabled?
There was a problem hiding this comment.
Thank you very much for reviewing the code, I will add new unit tests.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
goiri
reviewed
Dec 12, 2022
...ter/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java
Show resolved
Hide resolved
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
Outdated
Show resolved
Hide resolved
| /** Router Interceptor Allow Partial Result Enable. **/ | ||
| public static final String ROUTER_INTERCEPTOR_ALLOW_PARTIAL_RESULT_ENABLED = | ||
| ROUTER_PREFIX + "interceptor.allow-partial-result.enable"; | ||
| public static final boolean DEFAULT_ROUTER_INTERCEPTOR_ALLOW_PARTIAL_RESULT_ENABLED = true; |
There was a problem hiding this comment.
From the other comments, I think this needs to be false
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
goiri
reviewed
Dec 13, 2022
| @@ -2117,7 +2128,7 @@ private <R> Map<SubClusterInfo, R> invokeConcurrent(Collection<SubClusterInfo> c | |||
| throw new YarnRuntimeException(e.getCause().getMessage(), e); | |||
| } | |||
| }); | |||
|
|
|||
There was a problem hiding this comment.
Thank you very much for helping to review the code, I will fix it.
goiri
approved these changes
Dec 13, 2022
|
🎊 +1 overall
This message was automatically generated. |
|
@goiri Thank you very much for helping to review the code! |
slfan1989
added a commit
to slfan1989/hadoop
that referenced
this pull request
Dec 20, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: YARN-11358. [Federation] Add FederationInterceptor#allow-partial-result config.
When we use Federation, we will have multiple sub-clusters, and some interfaces need to call all clusters to get results,
such as getNodes, getNodeToLabels, etc.
Q1 : if the sub-cluster is unavailable when we call the interface, should the interface return the result?
The current code is to return the result. If the result is returned, the result information is inaccurate;
if the result is not returned, it is not user-friendly.
Plan to add a new configuration
yarn.router.interceptor.allow-partial-result.enableIf
true, we will ignore the exception of some subClusters during the calling process, and return result.If
false, if an exception occurs in a subCluster during the calling process, an exception will be thrown directly.