8000 HDFS-16945 . RBF: add RouterSecurityAuditLogger for router security manager by Neilxzn · Pull Request #5468 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HDFS-16945 . RBF: add RouterSecurityAuditLogger for router security manager #5468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: trunk
Choose a base branch
from

Conversation

Neilxzn
Copy link
Contributor
@Neilxzn Neilxzn commented Mar 9, 2023

Description of PR

HDFS-16945

we should add audit log for router security manager for token APIs. For examples,

 
2023-03-02 20:53:02,712 INFO org.apache.hadoop.hdfs.server.federation.router.security.RouterSecurityManager.audit: allowed=true ugi=hadoop ip=localhost/127.0.0.1 cmd=getDelegationToken toeknId=HDFS_DELEGATION_TOKEN token 18359 for hadoop with renewer hadoop proto=webhdfs

How was this patch tested?

add testRouterSecurityAuditLog

For code changes:

add RouterSecurityAuditLogger

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 56s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 39m 11s trunk passed
+1 💚 compile 0m 45s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 compile 0m 42s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 checkstyle 0m 36s trunk passed
+1 💚 mvnsite 0m 47s trunk passed
+1 💚 javadoc 0m 51s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 0m 57s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 spotbugs 1m 34s trunk passed
+1 💚 shadedclient 20m 50s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 35s the patch passed
+1 💚 compile 0m 38s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javac 0m 38s the patch passed
+1 💚 compile 0m 32s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 javac 0m 32s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 19s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 mvnsite 0m 36s the patch passed
+1 💚 javadoc 0m 34s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 0m 52s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 spotbugs 1m 19s the patch passed
+1 💚 shadedclient 20m 18s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 21m 53s hadoop-hdfs-rbf in the patch passed.
-1 ❌ asflicense 0m 39s /results-asflicense.txt The patch generated 1 ASF License warnings.
117m 17s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5468/1/artifact/out/Dockerfile
GITHUB PR #5468
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 36c7a91cf992 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 059c655
Default Java Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5468/1/testReport/
Max. process+thread count 2806 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5468/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 18s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0 8000 s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 44m 41s trunk passed
+1 💚 compile 0m 46s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 compile 0m 42s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 checkstyle 0m 33s trunk passed
+1 💚 mvnsite 0m 46s trunk passed
+1 💚 javadoc 0m 55s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 1m 5s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 spotbugs 1m 42s trunk passed
+1 💚 shadedclient 23m 48s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 37s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 34s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 javac 0m 34s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 19s the patch passed
+1 💚 mvnsite 0m 38s the patch passed
+1 💚 javadoc 0m 35s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 0m 54s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 spotbugs 1m 30s the patch passed
+1 💚 shadedclient 23m 55s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 23m 1s hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
131m 4s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5468/2/artifact/out/Dockerfile
GITHUB PR #5468
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 7534e81a60aa 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / bc47cf6
Default Java Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5468/2/testReport/
Max. process+thread count 2913 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5468/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Comment on lines +38 to +44
private static final ThreadLocal<StringBuilder> STRING_BUILDER =
new ThreadLocal<StringBuilder>() {
@Override
protected StringBuilder initialValue() {
return new StringBuilder();
}
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Java8 construct is a bit more readable.

ThreadLocal.withInitial(() -> new StringBuilder());

sb.append("ip=").append(addr).append("\t");
sb.append("cmd=").append(cmd).append("\t");

sb.append("\t").append("toeknId=");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo tokenId

}

@VisibleForTesting
public String creatAuditLog(boolean succeeded, String userName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo createAuditLog

@@ -152,7 +160,8 @@ public Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
tokenId = dtId.toStringStable();
success = true;
} finally {
logAuditEvent(success, operationName, tokenId);
logAuditEvent(success, user, Server.getRemoteIp(), operationName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remote address should be part of the CallerContext as well after HDFS-13248.

Copy link
Member
@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments. Good to have stuff

import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.*;
import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_CALLER_CONTEXT_SIGNATURE_MAX_SIZE_DEFAULT;

public class RouterSecurityAuditLogger {
Copy link
Member

Choose a reason for hiding this commen 8000 t

The reason will be displayed to describe this comment to others. Learn more.

I think this extend any child class of AuditLogger?


import java.net.InetAddress;

import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.*;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't use *, expand the imports

Comment on lines +38 to +44
private static final ThreadLocal<StringBuilder> STRING_BUILDER =
new ThreadLocal<StringBuilder>() {
@Override
protected StringBuilder initialValue() {
return new StringBuilder();
}
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be

  private static final ThreadLocal<StringBuilder> STRING_BUILDER =
      ThreadLocal.withInitial(StringBuilder::new);

Comment on lines +46 to +47
private int callerContextMaxLen;
private int callerSignatureMaxLen;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be final?

Comment on lines +84 to +87
if (
callerContext != null &&
callerContext.isContextValid()) {
sb.append("\t").append("callerContext=");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting issue

    if (callerContext != null && callerContext.isContextValid()) {

@@ -186,7 +196,8 @@ public long renewDelegationToken(Token<DelegationTokenIdentifier> token)
tokenId = id.toStringStable();
throw ace;
} finally {
logAuditEvent(success, operationName, tokenId);
logAuditEvent(success, user, Server.getRemoteIp(), operationName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't user always an empty string here? Should have been user = getRemoteUser().getUserName(); ?

Comment on lines +228 to +229
logAuditEvent(success, user, Server.getRemoteIp(), operationName,
CallerContext.getCurrent(), tokenId);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

user is empty here?
There is a line above

 String canceller = getRemoteUser().getUserName();

This guy is your user


import java.io.IOException;

import static org.junit.Assert.*;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No grouping of imports


import static org.junit.Assert.*;

public class TestRouterSecurityAuditLogger {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a javadoc for the test class that It has tests related to RouterSecurityAuditLogs

public class TestRouterSecurityAuditLogger {

@Test
public void testRouterSecurityAuditLog() throws IOException {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very basic and invoking directly the audit logger not invoking from the actual API. any scope to add tests which invokes audit logging via API invocation and you can grep the logs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0