8000 HADOOP-18993. Add option fs.s3a.classloader.isolation (branch-3.4) by tmnd1991 · Pull Request #6529 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HADOOP-18993. Add option fs.s3a.classloader.isolation (branch-3.4) #6529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 7, 2024
Merged

HADOOP-18993. Add option fs.s3a.classloader.isolation (branch-3.4) #6529

merged 1 commit into from
Feb 7, 2024

Conversation

tmnd1991
Copy link
Contributor
@tmnd1991 tmnd1991 commented Feb 5, 2024 •
edited by steveloughran
Loading

HADOOP-18993. Add option fs.s3a.classloader.isolation (#6301)

In HADOOP-17372 the S3AFileSystem forces the configuration classloader to be the same as the one that loaded S3AFileSystem. This leads to the impossibility in Spark applications to load third party credentials providers as user jars.
I propose to add a configuration key fs.s3a.extensions.isolated.classloader with a default value of true that if set to false will not perform the classloader set.

How was this patch tested?

Unit tests at org.apache.hadoop.fs.s3a.TestS3AFileSystemIsolatedClassloader.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@tmnd1991
Copy link
Contributor Author
tmnd1991 commented Feb 5, 2024

@steveloughran opened against 3.4, I'm running tests rn, I'll let you know 😄

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.4 Compile Tests _
+1 💚 mvninstall 31m 59s branch-3.4 passed
+1 💚 compile 0m 25s branch-3.4 passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 21s branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 checkstyle 0m 22s branch-3.4 passed
+1 💚 mvnsite 0m 25s branch-3.4 passed
+1 💚 javadoc 0m 19s branch-3.4 passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 22s branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 0m 44s branch-3.4 passed
+1 💚 shadedclient 19m 19s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 19s the patch passed
+1 💚 compile 0m 14s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 javac 0m 14s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 0m 12s the patch passed
+1 💚 mvnsite 0m 20s the patch passed
+1 💚 javadoc 0m 10s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 17s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚 spotbugs 0m 40s the patch passed
+1 💚 shadedclient 19m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 17s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 24s The patch does not generate ASF License warnings.
81m 5s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6529/1/artifact/out/Dockerfile
GITHUB PR #6529
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname Linux 13cffa8b801e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / ab292b4
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6529/1/testReport/
Max. process+thread count 552 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6529/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@tmnd1991
Copy link
Contributor Author
tmnd1991 commented Feb 6, 2024 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ITestS3AFileSystemStatistic.testBytesReadWithStream:72->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in number of FS bytes read by InputStreams expected:<2048> but was:<19632008>
[ERROR] Errors: 
[ERROR]   ITestS3APrefetchingCacheFiles.testCacheFileExistence:111 » AWSRedirect Receive...
[ERROR]   ITestAWSStatisticCollection.testCommonCrawlStatistics:74 » AccessDenied s3a://...
[ERROR]   ITestAWSStatisticCollection.testLandsatStatistics:56 » AccessDenied s3a://land...
[INFO] 
[ERROR] Tests run: 343, Failures: 1, Errors: 3, Skipped: 53

@steveloughran
Copy link
Contributor

yeah, landsat bucket failures. see #6523.

@steveloughran steveloughran merged commit 56f3bd9 into apache:branch-3.4 Feb 7, 2024
@steveloughran steveloughran changed the title Hadoop 18993 - target 3.4 HADOOP-18993. Add option fs.s3a.classloader.isolation (branch-3.4) Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tmnd1991 @hadoop-yetus @steveloughran @agilelab-tmnd1991
0