-
Notifications
You must be signed in to change notification settings - Fork 9k
HDFS-16767. RBF: Support observer node from Router-Based Federation #4127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Alread 8000 y on GitHub? Sign in to your account
Conversation
💔 -1 overall
This message was automatically generated. |
5f34b2d
to
2088b4a
Compare
💔 -1 overall
This message was automatically generated. |
2088b4a
to
88c780c
Compare
...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
Outdated
Show resolved
Hide resolved
💔 -1 overall
This message was automatically generated. |
88c780c
to
cbf3a15
Compare
💔 -1 overall
This message was automatically generated. |
...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
Outdated
Show resolved
Hide resolved
...ect/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
Outdated
Show resolved
Hide resolved
💔 -1 overall
This message was automatically generated. |
562b783
to
fdea891
Compare
Hi @fengnanli could you please take a look and add folks in your team. |
🎊 +1 overall
This message was automatically generated. |
fdea891
to
8324a37
Compare
💔 -1 overall
This message was automatically generated. |
8324a37
to
98eee9f
Compare
💔 -1 overall
This message was automatically generated. |
7f71985
to
77e41ba
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
@goiri @omalley I've now split off the IPC related parts of this change into another pull request (#4311). Please take a look. |
77e41ba
to
de6bb46
Compare
💔 -1 overall
This message was automatically generated. |
de6bb46
to
586216c
Compare
💔 -1 overall
This message was automatically generated. |
586216c
to
986c613
Compare
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
3e5bda5
to
2edd2b5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm really worried about the performance characteristics with doing the all namespace msync. I think we'd be better off if the client doesn't have the federatedstateids set on the rpc header causing them to go to the active NN.
We can still send back the federatedstateids and so later calls will go to the observer NNs. That does mean that old clients will only use the active NN, but I think that is ok. It also means that it is easy to have clients always use the active NN by not including an alignmentcontext.
...rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/FederationRPCMetrics.java
Show resolved
Hide resolved
...op-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
Outdated
Show resolved
Hide resolved
2edd2b5
to
174a043
Compare
💔 -1 overall
This message was automatically generated. |
174a043
to
54d2980
Compare
54d2980
to
61b0758
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
.../main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java
Outdated
Show resolved
Hide resolved
...op-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
Outdated
Show resolved
Hide resolved
...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
Outdated
Show resolved
Hide resolved
...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
Outdated
Show resolved
Hide resolved
...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java
Outdated
Show resolved
Hide resolved
...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
Outdated
Show resolved
Hide resolved
723e906
to
12c90d4
Compare
12c90d4
to
b0e67a3
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
Fixes apache#4127 Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
Why it is closed? |
Actually, it was committed to trunk. 6422eaf |
Fixes apache#4127 Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
Fixes apache#4127 Signed-off-by: Owen O'Malley <oomalley@linkedin.com> ACLOVERRIDE
Fixes apache#4127 Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
Description of PR
Enables routers to direct read calls to observer namenodes.
This is a refresh of a patch by Zhuobin Zhang to make it apply on current trunk. https://issues.apache.org/jira/secure/attachment/13027012/HDFS-13522.002.patch
How was this patch tested?
New unit tests
Deployed this in our test cluster and saw read operations in the observer audit log.
For code changes: