8000 TEZ-4604: tez-mapreduce does not delete files under staging directory by okumin · Pull Request #395 · apache/tez · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

TEZ-4604: tez-mapreduce does not delete files under staging directory #395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commit 8000 s into
base: master
Choose a base branch
from
Open

TEZ-4604: tez-mapreduce does not delete files under staging directory #395

wants to merge 3 commits into from

Conversation

okumin
Copy link
Contributor
@okumin okumin commented Feb 28, 2025

okumin
okumin commented Feb 28, 2025
View reviewed changes
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java
@@ -623,6 +623,7 @@ public JobStatus submitJob(JobID jobId, String jobSubmitDir, Credentials ts)
try {
dagAMConf.set(TezConfiguration.TEZ_AM_STAGING_DIR,
jobSubmitDir);
dagAMConf.setBoolean(TezConfiguration.TEZ_AM_STAGING_BASE_DIR_CLEANUP, true);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YARNRunner is a glue code between a MapReduce Job and Tez, implementing ClientProtocol. So, the client code of YARNRunner is Apache Hadoop.
ClientProtocol doesn't have an API to declare that a specific job has been completed. If we resolve this issue on the client side, we have to add new APIs to Apache Hadoop. That's why I added a new param and handled the issue on Apache Tez side.
I'm not confident that this approach is the best. I'd appreciate it if someone could give me a better idea.

@okumin okumin changed the title [WIP] TEZ-4604: tez-mapreduce does not delete files under staging directory TEZ-4604: tez-mapreduce does not delete files under staging directory Feb 28, 2025
@okumin okumin marked this pull request as ready for review February 28, 2025 09:27
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 22m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 2m 3s Maven dependency ordering for branch
+1 💚 mvninstall 9m 58s master passed
+1 💚 compile 2m 27s master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚 compile 2m 19s master passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+1 💚 checkstyle 2m 5s master passed
+1 💚 javadoc 2m 1s master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚 javadoc 1m 51s master passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+0 🆗 spotbugs 0m 46s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 4m 13s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for patch
+1 💚 mvninstall 1m 28s the patch passed
+1 💚 compile 1m 30s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚 javac 1m 30s the patch passed
+1 💚 compile 1m 22s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+1 💚 javac 1m 22s the patch passed
+1 💚 checkstyle 0m 55s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 javadoc 0m 59s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚 javadoc 0m 59s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
+1 💚 findbugs 3m 34s the patch passed
_ Other Tests _
+1 💚 unit 2m 11s tez-api in the patch passed.
+1 💚 unit 1m 22s tez-mapreduce in the patch passed.
+1 💚 unit 5m 5s tez-dag in the patch passed.
-1 ❌ unit 51m 37s tez-tests in the patch failed.
+1 💚 asflicense 0m 49s The patch does not generate ASF License warnings.
123m 54s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/artifact/out/Dockerfile
GITHUB PR #395
JIRA Issue TEZ-4604
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux 3f45da27f7c5 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 3aa2aa1
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu122.04-b06
unit https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/artifact/out/patch-unit-tez-tests.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/testReport/
Max. process+thread count 1303 (vs. ulimit of 5500)
modules C: tez-api tez-mapreduce tez-dag tez-tests U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-395/2/console
versions git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@okumin @tez-yetus
0