8000 MAPREDUCE-7341. Add an intermediate manifest committer for Azure and GCS by steveloughran · Pull Request #4075 · apache/hadoop · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

MAPREDUCE-7341. Add an intermediate manifest committer for Azure and GCS #4075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

MAPREDUCE-7341. Add an intermediate manifest committer for Azure and GCS #4075

wants to merge 3 commits into from

Conversation

steveloughran
Copy link
Contributor

this is the patch of #2971 spilt into three subcommits on top of trunk for merging, one per module, to be applied in order

  1. HADOOP-18162. hadoop-common support for MAPREDUCE-7341 Manifest Committer
  2. MAPREDUCE-7341. Add an intermediate manifest committer for Azure and GCS
  3. HADOOP-18163. hadoop-azure support for the Manifest Committer of MAPREDUCE-7341

once yetus is happy I will merge by cherrypicking each commit individually

tested: azure cardiff.

@steveloughran
HADOOP-18162. hadoop-common support for MAPREDUCE-7341 Manifest Commi…
4143cab 
…tter

* new stats names in StoreStatisticNames (for joint use with s3a committers)
* improvements to IOStatistics
* RateLimiting wrapper to guava RateLimiter
* S3A committer Tasks moved over as TaskPool & support for RemoteIterator
* JsonSerialization.load to fail fast if source does not exist

+ tests.

Contributed by Steve Loughran

Change-Id: Ia92e2ab5083ac3d8d3d713a4d9cb3e9e0278f654
@steveloughran
MAPREDUCE-7341. Add an intermediate manifest committer for Azure and GCS
f2f7403 
This is a mapreduce/spark output committer optimized for
performance and correctness on Azure ADLS Gen 2 storage
(via the abfs connector) and Google Cloud Storage
(via the external gcs connector library).

* It is safe to use with HDFS, however it has not been optimized
for that use.
* It is *not* safe for use with S3, and will fail if an attempt
is made to do so.

Contributed by Steve Loughran

Change-Id: I6f3502e79c578b9fd1a8c1485f826784b5421fca
@steveloughran
HADOOP-18163. hadoop-azure support for the Manifest Committer of MAPR…
a785e50 
…EDUCE-7341

Follow-on patch to MAPREDUCE-7341, adding ABFS support and tests

* resilient rename
* tests for job commit through the manifest committer.

contains
- HADOOP-17976. ABFS etag extraction inconsistent between LIST and HEAD calls
- HADOOP-16204. ABFS tests to include terasort

Contributed by Steve Loughran.

Change-Id: I0a7d4043bdf19bcb00c033fc389730109b93b77f
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 1s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 2s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 39 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 12m 29s Maven dependency ordering for branch
+1 💚 mvninstall 28m 26s trunk passed
+1 💚 compile 29m 36s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 23m 0s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 3m 47s trunk passed
+1 💚 mvnsite 3m 39s trunk passed
+1 💚 javadoc 2m 44s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 3m 13s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗 spotbugs 0m 30s branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚 shadedclient 25m 28s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 2m 35s the patch passed
+1 💚 compile 27m 58s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 27m 58s the patch passed
+1 💚 compile 23m 9s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 23m 9s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 3m 50s /results-checkstyle-root.txt root: The patch generated 23 new + 8 unchanged - 1 fixed = 31 total (was 9)
+1 💚 mvnsite 3m 41s the patch passed
+1 💚 xml 0m 12s The patch has no ill-formed XML file.
+1 💚 javadoc 2m 43s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 3m 15s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗 spotbugs 0m 29s hadoop-project has no data from spotbugs
+1 💚 shadedclient 24m 28s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 27s hadoop-project in the patch passed.
+1 💚 unit 17m 38s hadoop-common in the patch passed.
+1 💚 unit 6m 9s hadoop-mapreduce-client-core in the patch passed.
+1 💚 unit 2m 14s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 50s The patch does not generate ASF License warnings.
268m 38s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4075/1/artifact/out/Dockerfile
GITHUB PR #4075
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname Linux d58760536c9f 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / a785e50
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4075/1/testReport/
Max. process+thread count 1243 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-azure U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4075/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor Author

merged to 3.3 and trunk manually

@klaus-xiong klaus-xiong mentioned this pull request May 17, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@steveloughran @hadoop-yetus
0