TEZ-4603: Bucket Map Join can hang if the source vertex parallelism i… #392

mayankkunwar · 2025-02-19T04:53:28Z

…s changed by reducer autoparallelism
Description
The changes in this PR are w.r.t. when auto reduce parallelism kicks in, we try to updates downstream vertex's expected number of input, so that the execution hang could be avoided.
In this case, the type of edge between reducer and downstream vertices (Mapper) is a CUSTOM edge.
[Reducer --(C)--> Mapper]
The CUSTOM edge is because of Bucket Map Join

TODO: Unit Tests

tez-yetus · 2025-02-19T05:41:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	12m 7s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 1s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	2m 5s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 52s	master passed
+1 💚	compile	1m 6s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	1m 4s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 13s	master passed
+1 💚	javadoc	1m 6s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 3s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 45s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 15s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 45s	the patch passed
+1 💚	compile	0m 43s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 43s	the patch passed
+1 💚	compile	0m 40s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 40s	the patch passed
-0 ⚠️	checkstyle	0m 9s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 33s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 35s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
-1 ❌	findbugs	0m 45s	tez-dag generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	unit	1m 51s	tez-api in the patch passed.
+1 💚	unit	4m 19s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 10s	tez-dag in the patch passed.
+1 💚	asflicense	0m 23s	The patch does not generate ASF License warnings.
		47m 29s

Reason	Tests
FindBugs	module:tez-dag
	Load of known null value in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:[line 2004]

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/1/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux f3d637dfda40 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `871866e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/1/artifact/out/diff-checkstyle-tez-runtime-library.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/1/artifact/out/new-findbugs-tez-dag.html
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/1/testReport/
Max. process+thread count	2100 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/1/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2025-02-19T10:27:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	1m 56s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 43s	master passed
+1 💚	compile	1m 3s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	1m 3s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 14s	master passed
+1 💚	javadoc	1m 0s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 3s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 42s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 12s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 45s	the patch passed
+1 💚	compile	0m 42s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 42s	the patch passed
+1 💚	compile	0m 39s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 39s	the patch passed
-0 ⚠️	checkstyle	0m 9s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 32s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 37s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
-1 ❌	findbugs	0m 44s	tez-dag generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	unit	1m 53s	tez-api in the patch passed.
+1 💚	unit	4m 16s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 6s	tez-dag in the patch passed.
+1 💚	asflicense	0m 23s	The patch does not generate ASF License warnings.
		34m 55s

Reason	Tests
FindBugs	module:tez-dag
	Load of known null value in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:[line 2004]

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/2/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 527e63d5e93b 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `871866e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/2/artifact/out/diff-checkstyle-tez-runtime-library.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/2/artifact/out/new-findbugs-tez-dag.html
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/2/testReport/
Max. process+thread count	2100 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/2/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2025-02-19T10:32:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 10s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	1m 54s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 51s	master passed
+1 💚	compile	1m 5s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	1m 0s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 2s	master passed
+1 💚	javadoc	0m 54s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 46s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 41s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 6s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 44s	the patch passed
+1 💚	compile	0m 50s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 50s	the patch passed
+1 💚	compile	0m 41s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 41s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 31s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 32s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
-1 ❌	findbugs	0m 46s	tez-dag generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	unit	1m 50s	tez-api in the patch passed.
+1 💚	unit	4m 19s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 5s	tez-dag in the patch passed.
+1 💚	asflicense	0m 24s	The patch does not generate ASF License warnings.
		34m 17s

Reason	Tests
FindBugs	module:tez-dag
	Load of known null value in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:[line 2004]

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/3/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 52670823285b 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `871866e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/3/artifact/out/diff-checkstyle-tez-runtime-library.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/3/artifact/out/new-findbugs-tez-dag.html
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/3/testReport/
Max. process+thread count	2101 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/3/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2025-02-19T10:35:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	1m 53s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 52s	master passed
+1 💚	compile	0m 58s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	0m 53s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 2s	master passed
+1 💚	javadoc	0m 56s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 48s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 46s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 19s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 6s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 43s	the patch passed
+1 💚	compile	0m 49s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 49s	the patch passed
+1 💚	compile	0m 44s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 44s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 34s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 33s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
-1 ❌	findbugs	0m 48s	tez-dag generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	unit	1m 48s	tez-api in the patch passed.
+1 💚	unit	4m 22s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 7s	tez-dag in the patch passed.
+1 💚	asflicense	0m 23s	The patch does not generate ASF License warnings.
		34m 38s

Reason	Tests
FindBugs	module:tez-dag
	Load of known null value in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:in org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelismWrapper(int, VertexLocationHint, Map, Map, boolean) At VertexImpl.java:[line 2004]

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/4/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 8426e7be2a34 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `871866e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/4/artifact/out/diff-checkstyle-tez-runtime-library.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/4/artifact/out/new-findbugs-tez-dag.html
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/4/testReport/
Max. process+thread count	2100 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/4/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…s changed by reducer autoparallelism

tez-yetus · 2025-02-19T10:40:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 10s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0 8000 m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	1m 52s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 46s	master passed
+1 💚	compile	1m 3s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	1m 2s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 1s	master passed
+1 💚	javadoc	0m 56s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 52s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 44s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 16s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 49s	the patch passed
+1 💚	compile	0m 49s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 49s	the patch passed
+1 💚	compile	0m 42s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 42s	the patch passed
-0 ⚠️	checkstyle	0m 10s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 31s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 35s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	findbugs	2m 8s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 47s	tez-api in the patch passed.
+1 💚	unit	4m 26s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 13s	tez-dag in the patch passed.
+1 💚	asflicense	0m 24s	The patch does not generate ASF License warnings.
		35m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/5/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 82fb6da6ccd2 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `871866e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/5/artifact/out/diff-checkstyle-tez-runtime-library.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/5/testReport/
Max. process+thread count	1100 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/5/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2025-02-19T11:08:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	1m 54s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 40s	master passed
+1 💚	compile	1m 6s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	1m 5s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 10s	master passed
+1 💚	javadoc	1m 10s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 2s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 47s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 16s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 45s	the patch passed
+1 💚	compile	0m 43s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 43s	the patch passed
+1 💚	compile	0m 43s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 43s	the patch passed
-0 ⚠️	checkstyle	0m 8s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 32s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 36s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	findbugs	1m 50s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 53s	tez-api in the patch passed.
+1 💚	unit	4m 17s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 11s	tez-dag in the patch passed.
+1 💚	asflicense	0m 23s	The patch does not generate ASF License warnings.
		35m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/6/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux b6a261420f20 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `871866e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/6/artifact/out/diff-checkstyle-tez-runtime-library.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/6/testReport/
Max. process+thread count	1100 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/6/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2025-02-19T11:13:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 10s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	1m 53s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 42s	master passed
+1 💚	compile	1m 2s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	1m 0s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 10s	master passed
+1 💚	javadoc	1m 6s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 1s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 46s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 18s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 7s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 45s	the patch passed
+1 💚	compile	0m 43s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 43s	the patch passed
+1 💚	compile	0m 40s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 40s	the patch passed
-0 ⚠️	checkstyle	0m 9s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 31s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 34s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	findbugs	1m 57s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 52s	tez-api in the patch passed.
+1 💚	unit	4m 21s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 11s	tez-dag in the patch passed.
+1 💚	asflicense	0m 23s	The patch does not generate ASF License warnings.
		35m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/7/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 22d76683320a 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `871866e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/7/artifact/out/diff-checkstyle-tez-runtime-library.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/7/testReport/
Max. process+thread count	2100 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/7/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus · 2025-03-13T12:12:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	14m 5s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	2m 0s	Maven dependency ordering for branch
+1 💚	mvninstall	7m 44s	master passed
+1 💚	compile	1m 5s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	compile	1m 5s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	checkstyle	1m 11s	master passed
+1 💚	javadoc	1m 3s	master passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 0s	master passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+0 🆗	spotbugs	0m 45s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 13s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
+1 💚	mvninstall	0m 42s	the patch passed
+1 💚	compile	0m 43s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javac	0m 43s	the patch passed
+1 💚	compile	0m 42s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	javac	0m 42s	the patch passed
-0 ⚠️	checkstyle	0m 9s	tez-runtime-library: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was 24)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 31s	the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	0m 36s	the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
+1 💚	findbugs	1m 53s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 52s	tez-api in the patch passed.
+1 💚	unit	4m 19s	tez-runtime-library in the patch passed.
+1 💚	unit	4m 10s	tez-dag in the patch passed.
+1 💚	asflicense	0m 24s	The patch does not generate ASF License warnings.
		49m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/8/artifact/out/Dockerfile
GITHUB PR	#392
JIRA Issue	TEZ-4603
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux b4171bf66d1f 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `3aa2aa1`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~22.04-b06
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/8/artifact/out/diff-checkstyle-tez-runtime-library.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/8/testReport/
Max. process+thread count	1100 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-dag U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-392/8/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

okumin · 2025-04-05T05:32:59Z

@mayankkunwar
I recently resolved another hang issue with additional integration tests. This change can be a reference for your work.
44c4f1e#diff-ad65a331fa51a07f3cc5301ca7df09c199e9730a6f889bc8b1859554ccfc0519

mayankkunwar force-pushed the TEZ-4603-master branch 4 times, most recently from fe4a672 to 07cfcd5 Compare February 19, 2025 10:04

mayankkunwar force-pushed the TEZ-4603-master branch from 07cfcd5 to 873aab6 8000 Compare February 19, 2025 10:33

TEZ-4603: Bucket Map Join can hang if the source vertex parallelism i…

7d3a2a7

…s changed by reducer autoparallelism

mayankkunwar force-pushed the TEZ-4603-master branch from 873aab6 to 7d3a2a7 Compare February 19, 2025 10:38

Merge branch 'apache:master' into TEZ-4603-master

ff33943

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

TEZ-4603: Bucket Map Join can hang if the source vertex parallelism i… #392

TEZ-4603: Bucket Map Join can hang if the source vertex parallelism i… #392

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TEZ-4603: Bucket Map Join can hang if the source vertex parallelism i… #392

Are you sure you want to change the base?

TEZ-4603: Bucket Map Join can hang if the source vertex parallelism i… #392

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!