8000 Integration of Fortran wrapper · Issue #40 · slcs-jsc/mptrac · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Integration of Fortran wrapper #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
12 of 13 tasks
lars2015 opened this issue Jul 11, 2024 · 2 comments
Open
12 of 13 tasks

Integration of Fortran wrapper #40

lars2015 opened this issue Jul 11, 2024 · 2 comments
Assignees
@lars2015 @sabinegri @catrinmey
Labels
enhancement New feature or request

Comments

@lars2015
Copy link
Contributor
lars2015 commented Jul 11, 2024 •
edited by catrinmey
Loading

Overview

  • A Fortran wrapper for MPTRAC has been developed in the context of the natESM LAGOON sprint.
  • The code for the wrapper has been developed separately from the MPTRAC git repo, but should be integrated in the master branch.

Open tasks

  • fix remaining issues with wrapper code:
    • fix memory issue when using the wrapper on a notebook (Ubuntu) rather than JUWELS
    • handling of pointer-to-pointer in get_met() function
    • use of argc/argv command line arguments in the wrapper
  • adapt Makefile for compiling the wrapper:
    • add a flag WRAPPER to switch on/off the compilation of the Fortran wrapper (if needed)
    • allow for selection of the Fortran compiler (gfortran, nvfortran)
  • improve test case:
    • check that Fortran wrapper code uses the same dimensions as the C code (EX, EX, EP, NP)
    • print selected variables from ctl_t, atm_t, met_t in trac_fortran to a logfile, compare logfile with reference output to make sure C-to-Fortran mapping of variables/arrays is correct
  • documentation:
    • add markdown page to user/developer manual describing the wrapper
    • add doxygen documentation to Fortran wrapper and any additional/modified C code
@lars2015 lars2015 added the enhancement New feature or request label Jul 11, 2024
@sabinegri
Copy link
8000
Collaborator

The wrapper now runs on the notebook and JUWELS (without MPI!!!). It was tested with GCC+ParaStationMPI, Intel+ParaStationMPI, NVHPC+OpenMPI ().

To achieve this, the following issues were fixed:

  • Makefile: added option for Intel compiler for c
  • Makefile: added option for Intel and NVHPC compiler for fortran
  • mptrac_read_ctl: type of argc and argv. Variables passed to c code must be of a type c understands.
  • mptrac_read_ctl: added missing variable in ctl_t
  • mptrac_read_ctl: added script to check consistency between structs in c and fortran
  • mptrac_read_ctl: fixed data types in subroutine
  • mptrac_read_clim: added additional dimensions and structs required by clim_t. As clim_t shall be accessible from fortran, all sub-datatypes (struct clim_photo_t, etc.) must be known in fortran.
  • mptrac_read_clim: added data types to subroutine

For debugging the following additions were made:

  • print *,c_sizeof calls were added
  • add modules.sh to load stage and libraries on JUWELS
  • valgrind call in runscript

Please note:

  • on the notebook you have to set ulimit -s unlimited. The structs are quite large.
  • The NVHPC compiler is not happy with the size of the met_t struct (maybe 2 GB limit?). They are currently setfor ERA5 data. If you make them smaller, let's say so that ERA-interim fits, it works.

ToDo:

  • there are several ToDo comments in the code now please work through them and
  • clean up (remove unneccessary comments and prints)
  • please also check and fix to compile with MPI and GPU later on

@lars2015
Copy link
Contributor Author
lars2015 commented Nov 27, 2024 •
edited
Loading

Tested the wrapper, but found I have some questions/issues:

  • With the MPTRAC nightly builds, I noticed the runtime of the wrapper test on JUWELS and JURECA login nodes increases by a factor 10x or more, whereas runtime on the JUWELS and JURECA compute nodes is okay.

    • Found that export OMP_NUM_THREADS=4 was missing in the wrapper test script, need to check runtime tomorrow...
    • Setting OMP_NUM_THREADS fixed the issue

  • Nightly build GPU test case fails since compilation of the wrapper with NVHPC fails:

    • Example: https://datapub.fz-juelich.de/slcs/mptrac/nightly_builds/logs/2024-11-27_juwels-booster.txt 
      nvfortran -Wall -fopenmp -g -traceback -mcmodel=medium -Mlarge_arrays -O3 -Mdclchk -Mstandard -Mbounds -Mchkptr  -Mchkstk -Minfo -o mptrac_fortran.o -c mptrac_fortran.f90 -L ../libs/build/lib -L ../lib/build/lib64 -lgsl -lgslcblas -lnetcdf -lm -L /p/software/juwelsbooster/stages/2024/software/CUDA/12/lib64 -lcudart -lzfp -lzstd
NVFORTRAN-S-0151-Empty STRUCTURE, UNION, or MAP (mptrac_fortran.f90: 134)
NVFORTRAN-S-0151-Empty STRUCTURE, UNION, or MAP (mptrac_fortran.f90: 134)
NVFORTRAN-S-0151-Empty STRUCTURE, UNION, or MAP (mptrac_fortran.f90: 134)
NVFORTRAN-F-0000-Internal compiler error. Inconsistent size       0  (mptrac_fortran.f90: 134)
make: *** [Makefile:274: mptrac_fortran.o] Error 2
    • Also found this issue when compiling locally with NVHPC on my notebook.
    • This seems to be the problem with large structs (> 2GB) mentioned earlier?
    • Maybe ask Kaveh for help, if it is a technical issue with nvfortran?
    • Can we compile mptrac.c with nvc and mptrac_fortran.f90 with gfortran, and then link this together?

  • In the Makefile, I noticed the -static flag is not added to the FCFLAGS for STATIC=1, so wrapper is always dynamically linked.

    • Added -static to the FCFLAGS for static linking, but in this case trac_fortran fails with seg fault? Not sure whether static linking should actually work for a wrapper?
    • The wrapper test does not catch the seg fault, it reports "success" (seg fault happens after output files have been written, it seems). Update wrapper test run script to catch seg fault?

  • The wrapper test does not catch issues when changing arrays size via Makefile variables (e.g. make DEFINES="-DNP=123456"), it only catches changes in mptrac.h.

    • Think about how to properly pass on arrays sizes to the wrapper code?

  • I was wondering whether current string length (len=32 or len=40) in trac_fortran.f90 are actually too small for some use cases?

@janhclem janhclem mentioned this issue May 9, 2025
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lars2015 lars2015

@sabinegri sabinegri

430E

@catrinmey catrinmey

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lars2015 @sabinegri @catrinmey
0