8000 MiniMix, ConcurrentLoadTest hang, sometimes waiting on jsr166y/WaiterTask · Issue #11868 · eclipse-openj9/openj9 · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
MiniMix, ConcurrentLoadTest hang, sometimes waiting on jsr166y/WaiterTask #11868
Open
@pshipton

Description

@pshipton

https://ci.eclipse.org/openj9/job/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/15
MiniMix_5m_0
javacore files and core file in diagnostics
https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/15/system_test_output.tar.gz

LT  05:41:52.861 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:42:12.882 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:42:32.903 - Completed 100.0%. Number of tests started=50664 (+0)
STF 05:42:50.082 - Heartbeat: Process LT  is still running
LT  05:42:52.824 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:43:12.248 - **POSSIBLE HANG DETECTED**
STF 05:43:12.621 - **FAILED** Process LT  has hung
STF 05:43:12.621 - Collecting dumps for: LT 
STF 05:43:12.621 - Sending SIG 3 to the java process to generate a javacore
STF 05:43:12.623 - Running command: kill -3 7285
STF 05:43:12.623 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_3.stderr
STF 05:43:12.623 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_3.stdout
STF 05:43:12.631 - Pausing for 30 seconds
STF 05:43:42.632 - Sending SIG 3 to the java process to generate a javacore
STF 05:43:42.636 - Running command: kill -3 7285
STF 05:43:42.636 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_3.stderr
STF 05:43:42.636 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_3.stdout
STF 05:43:42.642 - Pausing for 30 seconds
STF 05:44:12.644 - Sending SIG 3 to the java process to generate a javacore
STF 05:44:12.646 - Running command: kill -3 7285
STF 05:44:12.646 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_3.stderr
STF 05:44:12.646 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_3.stdout
STF 05:44:12.651 - Pausing for 30 seconds
STF 05:44:42.652 - Sending SIGABRT (kill -6) to the java process to generate a core
STF 05:44:42.654 - Running command: kill -6 7285
STF 05:44:42.654 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_6.stderr
STF 05:44:42.654 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_6.stdout
STF 05:44:42.659 - Pausing for 30 seconds
STF 05:45:12.661 - Sending SIGXCPU (kill -24) to the java process to generate an OS dump
STF 05:45:12.663 - Running command: kill -24 7285
STF 05:45:12.663 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_24.stderr
STF 05:45:12.663 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk16_j9_extended.system_ppc64le_linux_Nightly_testList_1/openjdk-tests/TKG/output_16123450529369/MiniMix_5m_0/20210203-052247-MixedLoadTest/results/1.LT.kill_24.stdout
LT  05:43:13.447 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:43:17.252 - **POSSIBLE HANG DETECTED**
LT  05:43:22.258 - **POSSIBLE HANG DETECTED**
LT  05:43:27.262 - **POSSIBLE HANG DETECTED**
LT  05:43:32.269 - **POSSIBLE HANG DETECTED**
LT  05:43:32.870 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:43:37.274 - **POSSIBLE HANG DETECTED**
LT  05:43:42.283 - **POSSIBLE HANG DETECTED**
LT  05:43:47.336 - **POSSIBLE HANG DETECTED**
LT  05:43:52.342 - **POSSIBLE HANG DETECTED**
LT  05:43:52.843 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:43:57.348 - **POSSIBLE HANG DETECTED**
LT  05:44:02.357 - **POSSIBLE HANG DETECTED**
LT  05:44:07.361 - **POSSIBLE HANG DETECTED**
LT  05:44:12.369 - **POSSIBLE HANG DETECTED**
LT  05:44:13.263 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:44:17.468 - **POSSIBLE HANG DETECTED**
LT  05:44:22.473 - **POSSIBLE HANG DETECTED**
LT  05:44:27.478 - **POSSIBLE HANG DETECTED**
LT  05:44:32.482 - **POSSIBLE HANG DETECTED**
LT  05:44:32.884 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:44:37.489 - **POSSIBLE HANG DETECTED**
LT  05:44:42.493 - **POSSIBLE HANG DETECTED**
LT  05:44:47.509 - **POSSIBLE HANG DETECTED**
LT  05:44:52.557 - **POSSIBLE HANG DETECTED**
LT  05:44:52.859 - Completed 100.0%. Number of tests started=50664 (+0)
LT  05:44:57.564 - **POSSIBLE HANG DETECTED**

Many threads are waiting on load-7.

2LKMONINUSE      sys_mon_t:0x00003FFEEC403898 infl_mon_t: 0x00003FFEEC403918:
3LKMONOBJECT       net/adoptopenjdk/test/concurrent/support/jsr166y/WaiterTask@0x0000000084283B40: <unowned>
3LKNOTIFYQ            Waiting to be notified:
3LKWAITNOTIFY            "load-7" (J9VMThread:0x0000000000548300)
3XMTHREADINFO      "load-7" J9VMThread:0x0000000000548300, omrthread_t:0x00003FFFAC55E428, java/lang/Thread:0x0000000080E8DD58, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x29, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0x1CA1, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000181)
3XMTHREADINFO2            (native stack address range from:0x00003FFF8EFC0000, to:0x00003FFF8F000000, size:0x40000)
3XMCPUTIME               CPU usage total: 37.721936262 secs, current category="Application"
3XMTHREADBLOCK     Waiting on: net/adoptopenjdk/test/concurrent/support/jsr166y/WaiterTask@0x0000000084283B40 Owned by: <unowned>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
1INTERNAL                    Unable to obtain lock context information
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Object.wait(Native Method)
4XESTACKTRACE                at java/lang/Object.wait(Object.java:221(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.externalAwaitDone(ForkJoinTask.java:335)
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.doJoin(ForkJoinTask.java:403(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.join(ForkJoinTask.java:726)
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.invoke(ForkJoinPool.java:2432)
4XESTACKTRACE                at net/adoptopenjdk/test/concurrent/jsr166y/ForkJoinPoolTestAPI.testAPI(ForkJoinPoolTestAPI.java:233)
3XMTHREADINFO      "ForkJoinPool-4-worker-1" J9VMThread:0x0000000000979500, omrthread_t:0x00003FFEC8101968, java/lang/Thread:0x0000000082F9E660, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x10C, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x1D88, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000481)
3XMTHREADINFO2            (native stack address range from:0x00003FFED16F0000, to:0x00003FFED1730000, size:0x40000)
3XMCPUTIME               CPU usage total: 0.040613120 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Thread.sleep(Native Method)
4XESTACKTRACE                at java/lang/Thread.sleep(Thread.java:983(Compiled Code))
4XESTACKTRACE                at net/adoptopenjdk/test/concurrent/support/jsr166y/WaiterTask.exec(WaiterTask.java:74)
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.doExec(ForkJoinTask.java:295(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.scan(ForkJoinPool.java:1665(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.runWorker(ForkJoinPool.java:1598)
4XESTACKTRACE                at java/util/concurrent/ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Possibly related to #9999 and #11836

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0