HPSF CI test with more build types #8011

masterleinad · 2025-04-21T19:32:00Z

This pull request adds more build types for the existing HPSF CI builds

for pull requests, we continue to use RelWithDebInfo
for merge commits, we use Release
for nightly builds, we use Debug

Building in Debug mode required a bunch of changes:

The shared_space test is failing frequently (which is spurious and we care about behavior in non-Debug builds)
~~A Kokkos::Vector unit test is broken and we need the same fix as in DualView: Store modified flags as bool #7993.~~
For SYCL more bessel function unit tests are broken.
view_memory_access_violations_from_device doesn't work independent of the build type for SYCL. The error message captured is just empty.

Drive-by-change: KOKKOS_ENABLE_DEBUG_BOUNDS_CHECKS -> KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK

https://gitlab.spack.io/masterleinad/kokkos/-/pipelines/1058739 shows a run in Debug mode with these changes. I verified manually that the atomic performance regression is the only test failing for HIP.

dalg24 · 2025-04-21T23:21:42Z

core/unit_test/TestViewCtorDimMatch.hpp

It means these have not been running for a while.
The bug was introduced in #5190 and had gone unnoticed in #7310.

masterleinad · 2025-04-22T17:08:19Z

containers/src/Kokkos_Vector.hpp

We can't rely on the status after calling resize. It might be modified on the host or device. With the current implementation we are ending up setting both modified flags which we catch in Debug mode.
Since we are modifying the host-side after resize, we should only sync then. Also, we can be more specific about the modifed flags afterwards by calling modify_host.

masterleinad · 2025-04-22T17:57:48Z

core/unit_test/TestMathematicalSpecialFunctions.hpp

-    if (std::is_same_v<TEST_EXECSPACE, Kokkos::SYCL>) upper_limit_0 = 19;
-#endif
-    for (int i = 1; i < upper_limit_0; i++) {
+    // FIXME_SYCL Failing for Intel GPUs highly dependent on optimization flags


Seems to be fine for oneAPI 2025.0.4 and later

masterleinad · 2025-04-22T18:48:46Z

CI timings for a rough estimate:

Build	Debug	RelWithDebInfo	Release
AMD-MI300A	21m 41s	23m 04s	20m 34s
INTEL-DATA-CENTER-MAX	49m 52s	35m 59s	28m 22s
NVIDIA-GH200	35m 05s	10m 33s	17m 20s
NVIDIA-GRACE-GRACE	22m 46s	5m 12s	5m 40s
NVIDIA-P100	52m 28s	12m 11s	12m 4s
NVIDIA-RTX5080	25m 05s	10m 13s	10m 15s

Debug expectedly is most expensive and there isn't much of difference between Release and RelWithDebInfo. Thus, I would use Debug for Nightly builds but don't have a strong opinion about the other two.

masterleinad · 2025-04-22T20:55:04Z

Retest this please.

containers/src/Kokkos_Vector.hpp

JBludau

I would like the comment to be more clear but this is non blocking.
The rest is just comments/questions outside of the review scope

containers/src/Kokkos_Vector.hpp

.gitlab/hpsf-gitlab-ci.yml

JBludau · 2025-04-23T13:15:07Z

core/unit_test/TestSharedSpace.cpp

@@ -115,6 +115,10 @@ TEST(defaultdevicetype, shared_space) {
  GTEST_SKIP() << "skipping because clock_tic is only defined for sycl+intel "
                  "gpu and with rdc support";
 #endif
+#if defined(KOKKOS_ENABLE_DEBUG)


Yeah, I would expect this to fail in DEBUG mode...I would expect the cycle counting to only work reliably for highly optimized kernels

What do we do in debug mode that causes failure?

We do the same, but counting clock cycles to introspect when pages are migrated does not sound robust in a debug mode. We have no idea if it does do the same granularity etc.

Based on a discussion with Jakob, now wondering if that guard should be

Suggested change

#if defined(KOKKOS_ENABLE_DEBUG)

#if defined(KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK)

yeah, without bounds checking this should actually work

CMAKE_BUILD_TYPE=Debug implies that Kokkos_ENABLE_DEBUG and Kokkos_ENABLE_DEBUG_DUALVIEW_MODIFY_CHECK are activated by default but Kokkos_ENABLE_DEBUG_BOUNFS_CHECK's default value is always OFF.

Thanks for correcting me

In that case I don't know why this is failing. Any guesses?

For the record I am updating the doc to reflect reality
kokkos/kokkos-core-wiki#669

I think debug mode with turned off optimization might just make everything a bit more unpredictable. For example functions not inlined that lead to jumps in memory that can cause varying timings etc.. So while I wouldn't have bet on this, it also isn't necessarily super surprising.

core/unit_test/TestViewMemoryAccessViolation.hpp

dalg24 · 2025-04-24T13:49:07Z

core/unit_test/TestViewMemoryAccessViolation.hpp

-                    "is defined";
+    GTEST_SKIP()
+        << "skipping because SYCL device-side abort does not show an error "
+           "message independent of NDEBUG";


On Aurora, out of the 4 checks

Abort SpaceAwareAccessorAccessViolation ViewMemoryAccessViolation ViewOutOfBoundsAccess

only SpaceAwareAccessorAccessViolation and ViewMemoryAccessViolation are failing with something like

[ RUN ] sycl_DeathTest.mdspan_space_aware_accessor_invalid_access_from_device FATAL: Unexpected page fault from GPU at 0x0, ctx_id: 9 (CCS) type: 0 (NotPresent), level: 3 (PML4), access: 0 (Read), banned: 1, aborting. Abort was called at 287 line in file: /home/ubit/rpmbuild/BUILD/intel-compute-runtime-24.35.30872.31/shared/source/os_interface/linux/drm_neo.cpp /home/darndt/kokkos_new/core/unit_test/TestSpaceAwareAccessorAccessViolation.hpp:44: Failure Death test: { Kokkos::parallel_for(Kokkos::RangePolicy<ExecutionSpace>(s, 0, 1), *this); Kokkos::fence(); } Result: died but not with expected error. Expected: contains regular expression "Kokkos::SpaceAwareAccessor ERROR: attempt to access inaccessible memory space" Actual msg: [ DEATH ] FATAL: Unexpected page fault from GPU at 0x0, ctx_id: 9 (CCS) type: 0 (NotPresent), level: 3 (PML4), access: 0 (Read), banned: 1, aborting. [ DEATH ] [ FAILED ] sycl_DeathTest.mdspan_space_aware_accessor_invalid_access_from_device (585 ms)

so the invalid access has a higher priority than the error message from Kokkos::abort and the kernel keeps executing after calls to Kokkos::abort.

We are now just checking that we abort no matter what the error message is.

dalg24 · 2025-04-24T13:51:20Z

core/unit_test/TestSharedSpace.cpp

@@ -115,6 +115,10 @@ TEST(defaultdevicetype, shared_space) {
  GTEST_SKIP() << "skipping because clock_tic is only defined for sycl+intel "
                  "gpu and with rdc support";
 #endif
+#if defined(KOKKOS_ENABLE_DEBUG)


What do we do in debug mode that causes failure?

dalg24 · 2025-04-24T13:52:21Z

containers/src/Kokkos_Vector.hpp

Do we really want that patch merged in as part of this PR?
Isn't that it bug fix?

I'm happy to create a pull request just for this change but the test only fails in Debug mode right now, i.e., we coincidentally do the right thing regardless and I'm not to keen on writing more tests for a deprecated feature.

crtrott

One question, and also I want the vector thing in the separate PR merged first.

crtrott · 2025-04-25T16:37:52Z

core/unit_test/TestSharedSpace.cpp

@@ -115,6 +115,10 @@ TEST(defaultdevicetype, shared_space) {
  GTEST_SKIP() << "skipping because clock_tic is only defined for sycl+intel "
                  "gpu and with rdc support";
 #endif
+#if defined(KOKKOS_ENABLE_DEBUG)


I think debug mode with turned off optimization might just make everything a bit more unpredictable. For example functions not inlined that lead to jumps in memory that can cause varying timings etc.. So while I wouldn't have bet on this, it also isn't necessarily super surprising.

crtrott · 2025-04-25T16:38:54Z

core/unit_test/TestViewCtorDimMatch.hpp

@@ -131,7 +131,7 @@ struct StaticRank<0> {
 TEST(TEST_CATEGORY_DEATH, view_construction_with_wrong_params_stat) {
  ::testing::FLAGS_gtest_death_test_style = "threadsafe";

-#ifndef KOKKOS_ENABLE_DEBUG_BOUNDS_CHECKS
+#ifndef KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK


so did the test pass? Or did we just not test with bounds check?

The CUDA-11.6-NVCC-DEBUG build has Kokkos_ENABLE_DEBUG_BOUNDS_CHECK=ON.

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Co-authored-by: JBludau <104908666+JBludau@users.noreply.github.com> Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

The requested changes and questions were addressed.

masterleinad requested a review from ndellingwood April 21, 2025 19:32

dalg24 reviewed Apr 21, 2025

View reviewed changes

masterleinad commented Apr 22, 2025

View reviewed changes

containers/src/Kokkos_Vector.hpp Outdated Show resolved Hide resolved

masterleinad requested a review from JBludau April 23, 2025 12:07

JBludau approved these changes Apr 23, 2025

View reviewed changes

masterleinad mentioned this pull request Apr 23, 2025

DualView: Be more explicit about synchronization behavior for resize kokkos/kokkos-core-wiki#668

Merged

masterleinad force-pushed the hpsf_ci_more_build_types branch from 7c5bc63 to 6f95df2 Compare April 23, 2025 18:20

ldh4 approved these changes Apr 23, 2025

View reviewed changes

ndellingwood approved these changes Apr 23, 2025

View reviewed changes

dalg24 reviewed Apr 24, 2025

View reviewed changes

crtrott previously requested changes Apr 25, 2025

View reviewed changes

dalg24 mentioned this pull request Apr 28, 2025

Fix Vector::push_back #8036

Merged

masterleinad and others added 9 commits April 28, 2025 13:23

HPSF CI test with more build types

756e700

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Use -flink-huge-device-code

3d52bd5

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Disable shared_space in Debug mode

4777535

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Disable more bessel function tests

e32e128

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Disable view_memory_access_violations_from_device

3364f48

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Disable more Bessel function tests

5cd5d64

Only enable failing Bessel function tests for oneAPI 2025.0.4 and later

75a8ddc

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Improve skip message

6b37c7e

Co-authored-by: JBludau <104908666+JBludau@users.noreply.github.com> Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

Improve access violation behavior in SYCL

e5a7092

Signed-off-by: Daniel Arndt <arndtd@ornl.gov>

masterleinad force-pushed the hpsf_ci_more_build_types branch from d5d9d62 to e5a7092 Compare April 28, 2025 17:24

masterleinad requested a review from crtrott April 28, 2025 17:24

Merge branch 'develop' into hpsf_ci_more_build_types

ff0fefd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPSF CI test with more build types #8011

HPSF CI test with more build types #8011

	#if defined(KOKKOS_ENABLE_DEBUG)
	#if defined(KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK)

HPSF CI test with more build types #8011

Are you sure you want to change the base?

HPSF CI test with more build types #8011

Conversation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment