Repository navigation
Kokkos::View related segfaults when using the SYCL backend #330
Closed
ThomasJannaudCAM
started this conversation in
General
Mar 7, 2025
·
3 comments
·
7 replies
Comment options
{{title}}
|
can't really help much as we don't have any intel gpu here. A few thoughts:
|
You must be logged in to vote
All reactions
1 reply
Comment options
{{title}}
|
additional thoughts: maybe add -DKokkos_ENABLE_SERIAL=ON to cmake, not sure it is enabled by default with syscl. |
All reactions
Comment options
{{title}}
|
Hi @glesur - thanks very much for your suggestions. Here is the output from cmake -- The C compiler identification is IntelLLVM 2025.0.0 -- The CXX compiler identification is IntelLLVM 2025.0.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icx - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icpx - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Setting default Kokkos CXX standard to 17 -- Kokkos version: 4.5.0 -- The project name is: Kokkos -- Using internal gtest for testing -- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt -- Using -std=gnu++17 for C++17 extensions as feature -- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL -- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL - found -- Looking for SYCL_EXT_ONEAPI_GRAPH -- Looking for SYCL_EXT_ONEAPI_GRAPH - found -- Built-in Execution Spaces: -- Device Parallel: Kokkos::SYCL -- Host Parallel: NoTypeDefined -- Host Serial: SERIAL -- -- Architectures: -- INTEL_PVC -- Found TPLLIBDL: /usr/include -- Looking for C++ include oneapi/dpl/execution -- Looking for C++ include oneapi/dpl/execution - found -- Looking for C++ include oneapi/dpl/algorithm -- Looking for C++ include oneapi/dpl/algorithm - found -- Performing Test KOKKOS_NO_TBB_CONFLICT -- Performing Test KOKKOS_NO_TBB_CONFLICT - Success -- Using internal desul_atomics copy -- Experimental mdspan support is enabled -- Looking for C++ include experimental/mdspan -- Looking for C++ include experimental/mdspan - not found -- Looking for C++ include mdspan -- Looking for C++ include mdspan - not found -- Using internal mdspan directory /lus/flare/projects/grteclyn/idefix/src/kokkos/tpls/mdspan/include -- Kokkos Backends: SERIAL;SYCL -- Including problem-specific CMakeLists in '/lus/flare/projects/grteclyn/idefix/test/MHD/sod' -- Explicitely setting Idefix_MHD = ON -- Idefix final configuration -- MHD: ON -- MPI: OFF -- HDF5: OFF -- Python: OFF -- Reconstruction: Linear -- Precision: Double -- Version: 2.2.00-54482d58 -- Problem definitions: 'definitions.hpp' -- Configuring done (27.4s) -- Generating done (2.8s) Then I build using make and everything is fine at this stage. ./idefix
----> Profiler::Init...
----> ...returned
.:HMMMMHn:. ..:n..
.H*'`` `'%HM'''''!x.
:x x*` .(MH: `#h.
x.`M M> :nMMMMMMMh. `n.
*kXk.. XL nnx:.XMMMMMMMMMMML .. 4X.
)MMMMMx 'M `^?M*MMMMMMMMMMMM:HMMMHHMM.
MMMMMMMX ?k 'X ..'*MMMMMMM.#MMMMMMMMMx
XMMMMMMMX 4: M:MhHxxHHHx`MMx`MMMMMMMMM>
XM!` ?M `x 4MM'`''``HHhMMX 'MMMMMMMM
4M M `: *> `` .('MX '*MMMM'
MX `X.nnx.. ..XMx` 'M*X
?h. ''```^'*!Hx. :Mf xHMh M**MMM 4L`
`*Mx `'*n.x. 4M> :M` `` 'M ` %
'% ``*MHMX X> !
:! `#MM> X> ` :x
:M ?M `X . ..'M
XX .!*X `x XM( MMx`h
'M>:: `M: `+ MMX XMM `:
'M> M 'X 'MMX ?MMk.Xx..
'M> ?L ...:! MMX.H**'MMMM*h
M> #L :!'`MM. . X*`.xHMMMMMnMk.
`! #h. :L XM'*hxHMM*MhHMMMMMMMMMM'#h
+ XMh: 4! x :f MM' `*MMMMMMMMMM% `X
M Mf``tHhxHM M> 4k xxX' `#MMMMMMMf `M .>
:f M `MMMMM: M> M!MMM: '*MMf' 'MH*
! Xf 'MMMMMX `X X>'h.` :P*Mx. .d*~..
:M X 4MMMMM> ! X~ `Mh. .nHL..M#'%nnMhH!'`
XM d> 'X`'**h 'h M ^'MMHH+*'` '''' `'**'
%nxM> *x+x.:. XL.. `k `::X
:nMMHMMM:. X> Mn`*MMMMMHM: `: ?MMn.
`'**MML M> 'MMhMMMMMMMM # `M:^*x
^*MMttnnMMMMMMMMMMMH>. M:.4X
`MMMM>X ( .MMM:MM! .
`'''4x.dX +^ `''MMMMHM?L..
``' `'`'`'`
Idefix version 2.2.00-54482d58
Built against Kokkos 40500
Compiled on Mar 11 2025 at 11:14:32
Main: initialization stage.
----> Grid::Grid(Input)...
----> ...returned
----> GridHost::GridHost(Grid)...
----> ...returned
----> GridHost::MakeGrid...
----> ...returned
----> GridHost::SyncToDevice...
----> ...returned
----> DataBlock::DataBlock...
--------> GridHost::GridHost(Grid)...
--------> ...returned
--------> GridHost::SyncFromDevice...
--------> ...returned
--------> DataBlock::ExtractSubdomain...
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
--------> ...returned
--------> DataBlock::MakeGeometry()...
------------> idefix_for(Volumes)...
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_dx0")
Backtrace:
[0x6069d0]
[0x5e02d2]
[0x411d76]
[0x41af1d]
[0x51e5c8]
[0x45879b]
[0x5d0c2a]
[0x14d60757124d] __libc_start_main
[0x40ef0a]
Aborted
|
You must be logged in to vote
All reactions
6 replies
Comment options
{{title}}
|
Hi @glesur - I'm happy to report that that worked! There was a lot of output because I compiled on debug so let me know if you want to see it. I also checked with MPI (but always 1 MPI rank to 1 GPU) and it was fine as well. Thanks so much! |
All reactions
Comment options
{{title}}
|
Thank you both very much ! |
All reactions
Comment options
{{title}}
|
Excellent, I'll propagate the fix to the develop branch. @ThomasJannaudCAM , whenever you have time, I'd like to know how it performs on Intel PVCs on the OT3D test with 128^3 points and no tracer. |
All reactions
Comment options
{{title}}
|
What would you want ? Frequency of cell updates and parallel efficiency ? |
All reactions
Comment options
{{title}}
|
yes, cell update/s (don't really care about the parallel efficiency for now), as in https://idefix.readthedocs.io/latest/performances.html |
All reactions
Comment options
{{title}}
|
A fix for this has been merged into develop (#331) and will be part of the next Idefix release. |
You must be logged in to vote
All reactions
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Heading
Bold
Italic
Quote
Code
Link
Numbered list
Unordered list
Task list
Attach files
Mention
Reference
Select a reply
Loading
reacted with thumbs up emoji
reacted with thumbs down emoji
reacted with laugh emoji
reacted with hooray emoji
reacted with confused emoji
reacted with heart emoji
reacted with rocket emoji
reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
{{title}}
Uh oh!
There was an error while loading. Please reload this page.
Hi all,
In Cambridge we just received some new Intel Ponte Vecchio GPUs. With a software engineer, we have been trying to run IDEFIX on them. She was able to compile Idefix on a number of the test cases using the SYCL backend provided by Kokkos, but at runtime it segfaults with an error about Kokkos::View. We are wondering if this is a fault with Idefix or with Kokkos. She has been able to reproduce it on two different PVC systems (each with slightly different environments - see below).
Thanks in advance !
Below is her description of how to reproduce the error:
In the test/MHD/disk directory type:
cmake ${IDEFIX_DIR} -DIdefix_MPI=ON -DKokkos_ENABLE_SYCL=ON -DKokkos_ARCH_INTEL_PVC=ON -DCMAKE_CXX_COMPILER=mpiicpx
then when it has finished configuring the build type ‘make'.
Run the Idefix executable (I used 4 MPI ranks but it doesn’t really matter, I’ve also tried with 1):
mpirun -np 4 ./idefix
This is the output:
jk945@sw-pvc03:~/idefix/test/MHD/disk $ mpirun -np 4 ./idefix [0] MPI startup(): ===== GPU topology on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) ===== [0] MPI startup(): NUMA nodes : 2 [0] MPI startup(): GPUs : 4 [0] MPI startup(): Stacks (Tiles) : 8 [0] MPI startup(): Hierarchy mode : flat [0] MPI startup(): NUMA Id GPU Id Stacks (tiles) Ranks on this NUMA [0] MPI startup(): 0 0,1 (0,1)(2,3) 0,1 [0] MPI startup(): 1 2,3 (4,5)(6,7) 2,3 [0] MPI startup(): ===== GPU pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) ===== [0] MPI startup(): Rank Pin stack (tile) [0] MPI startup(): 0 {0,1} [0] MPI startup(): 1 {2,3} [0] MPI startup(): 2 {4,5} [0] MPI startup(): 3 {6,7} MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found [0] MPI startup(): Intel(R) MPI Library, Version 2021.13 Build 20240515 (id: df72937) [0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found [0] MPI startup(): libfabric version: 1.20.1-impi [0] MPI startup(): libfabric provider: mlx [0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400_x1.dat" not found [0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400.dat" not found [0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx.dat" not found [0] MPI startup(): Load tuning file: "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi.dat" [0] MPI startup(): Number of NICs: 1 [0] MPI startup(): ===== NIC pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) ===== [0] MPI startup(): Rank Thread id Pin nic [0] MPI startup(): 0 0 mlx [0] MPI startup(): 1 0 mlx [0] MPI startup(): 2 0 mlx [0] MPI startup(): 3 0 mlx [0] MPI startup(): ===== CPU pinning ===== [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 2296614 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27} [0] MPI startup(): 1 2296615 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54 ,55} [0] MPI startup(): 2 2296616 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82 ,83} [0] MPI startup(): 3 2296617 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107, 108,109,110,111} .:HMMMMHn:. ..:n.. .H*'`` `'%HM'''''!x. :x x*` .(MH: `#h. x.`M M> :nMMMMMMMh. `n. *kXk.. XL nnx:.XMMMMMMMMMMML .. 4X. )MMMMMx 'M `^?M*MMMMMMMMMMMM:HMMMHHMM. MMMMMMMX ?k 'X ..'*MMMMMMM.#MMMMMMMMMx XMMMMMMMX 4: M:MhHxxHHHx`MMx`MMMMMMMMM> XM!` ?M `x 4MM'`''``HHhMMX 'MMMMMMMM 4M M `: *> `` .('MX '*MMMM' MX `X.nnx.. ..XMx` 'M*X ?h. ''```^'*!Hx. :Mf xHMh M**MMM 4L` `*Mx `'*n.x. 4M> :M` `` 'M ` % '% ``*MHMX X> ! :! `#MM> X> ` :x :M ?M `X . ..'M XX .!*X `x XM( MMx`h 'M>:: `M: `+ MMX XMM `: 'M> M 'X 'MMX ?MMk.Xx.. 'M> ?L ...:! MMX.H**'MMMM*h M> #L :!'`MM. . X*`.xHMMMMMnMk. `! #h. :L XM'*hxHMM*MhHMMMMMMMMMM'#h + XMh: 4! x :f MM' `*MMMMMMMMMM% `X M Mf``tHhxHM M> 4k xxX' `#MMMMMMMf `M .> :f M `MMMMM: M> M!MMM: '*MMf' 'MH* ! Xf 'MMMMMX `X X>'h.` :P*Mx. .d*~.. :M X 4MMMMM> ! X~ `Mh. .nHL..M#'%nnMhH!'` XM d> 'X`'**h 'h M ^'MMHH+*'` '''' `'**' %nxM> *x+x.:. XL.. `k `::X :nMMHMMM:. X> Mn`*MMMMMHM: `: ?MMn. `'**MML M> 'MMhMMMMMMMM # `M:^*x ^*MMttnnMMMMMMMMMMMH>. M:.4X `MMMM>X ( .MMM:MM! . `'''4x.dX +^ `''MMMMHM?L.. ``' `'`'`'` Idefix version 2.2.00-54482d58 Built against Kokkos 40500 Compiled on Mar 5 2025 at 17:00:09 Main: initialization stage. Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0") Backtrace: Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0") Backtrace: Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0") Backtrace: Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0") Backtrace: [0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x14acd1a29d90] [0x14acd1a29e40] __libc_start_main [0x40eec5] [0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x15455c029d90] [0x15455c029e40] __libc_start_main [0x40eec5] [0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x14f2af629d90] [0x14f2af629e40] __libc_start_main [0x40eec5] [0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x151e84429d90] [0x151e84429e40] __libc_start_main [0x40eec5]I do not see this error with the CUDA backend.
Here are the versions of Intel oneAPI packages that I have loaded: