Doesn't use external libopenmc.so and instead installs conflicting libopenmc.so

[yurivict]

This causes conflict with pre-installed OpenMC.

Please use external OpenMC when it is present.

[RonRahaman]

Hello! ENRICO can't, in general, be run with an arbitrary version of OpenMC. This is because the OpenMC API is under active development. We update ENRICO's OpenMC submodule from upstream periodically.

Could you show us how you're running CMake? In ENRICO, if CMAKE_INSTALL_DIR is not defined by the user, then make install will install everything in the build directory. We implemented this specifically to avoid the issue you're encountering.

[yurivict]

cmake runs like this:

cmake -DNEK_DIST=none -DCMAKE_C_COMPILER:STRING="cc"  -DCMAKE_CXX_COMPILER:STRING="c++"  -DCMAKE_C_FLAGS:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  -DCMAKE_C_FLAGS_DEBUG:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  -DCMAKE_C_FLAGS_RELEASE:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  -DCMAKE_CXX_FLAGS:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -I/usr/local/mpi/openmpi/include -isystem /usr/local/include "  -DCMAKE_CXX_FLAGS_DEBUG:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -I/usr/local/mpi/openmpi/include -isystem /usr/local/include "  -DCMAKE_CXX_FLAGS_RELEASE:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -I/usr/local/mpi/openmpi/include -isystem /usr/local/include "  -DCMAKE_EXE_LINKER_FLAGS:STRING=" -L/usr/local/mpi/openmpi/lib -lmpi -Wl,-rpath=/usr/local/lib/gcc10  -L/usr/local/lib/gcc10 -B/usr/local/bin -fstack-protector-strong "  -DCMAKE_MODULE_LINKER_FLAGS:STRING=" -L/usr/local/mpi/openmpi/lib -lmpi -Wl,-rpath=/usr/local/lib/gcc10  -L/usr/local/lib/gcc10 -B/usr/local/bin -fstack-protector-strong "  -DCMAKE_SHARED_LINKER_FLAGS:STRING=" -L/usr/local/mpi/openmpi/lib -lmpi -Wl,-rpath=/usr/local/lib/gcc10  -L/usr/local/lib/gcc10 -B/usr/local/bin -fstack-protector-strong "  -DCMAKE_INSTALL_PREFIX:PATH="/usr/local"  -DCMAKE_BUILD_TYPE:STRING="Release"  -DTHREADS_HAVE_PTHREAD_ARG:BOOL=YES  -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=YES  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON

It's fine to use bundled OpenMC, but you can't install it's libraries or headers because they would conflict with the standalone OpenMC package.

Instead you should build static library from the bundled OpenMC project and only use it locally to build your binaries and not install it.

[RonRahaman]

It’s not feasible to statically link everything into the ENRICO binaries because one of its third-party single-physics solvers is configured to use dynamic linking. I recommend setting CMAKE_INSTALL_PREFIX to something like /usr/local/enrico or /opt/local/enrico. ENRICO’s libopenmc will then be installed in /usr/local/enrico/lib or /opt/local/enrico/lib

[yurivict]

This would still conflict with the standalone OpenMC package because libopenmc.so would be installed twice. Same library can't be installed twice because ldconfig resolved its name into path.

You can link static libraries into dynamic libraries fine as long as they are built with -fPIC.

[RonRahaman]

Not all our 3rd-party libraries can be configured to be built with fPIC. The rpath in the installed ENRICO binaries is set to the install path of its libraries, including libopenmc.

[yurivict]

What does enrico install? I don't see any enrico's own headers, only OpenMC and gsl-lite headers.

If enrico only installs an executable, why does it even need to install OpenMC headers or any static libraries?

[RonRahaman]

One of our dependent CFD libraries -- nekRS and its backend OCCA -- have many header in the ENRICO installation.

A nekRS case has a number of case-specific source files which the user is able to customize for things like boundary conditions, heat source, etc. These case-specific source files are just-in-time compiled at runtime. Since ENRICO's nekRS is also bundled due to versioning issues, a ENRICO installation includes those nekRS and OCCA headers.

[RonRahaman]

Are you able to successfully install via something like this?

• I'm inferring the MPI compiler paths from the CMake command you quoted above.
• I excluded these all the explicit include paths, link paths, libs, and RPATH flags. CMake's compiler verifications, find_package(MPI), and the way we set the RPATH should be sufficient.
• I've specified set CMAKE_INSTALL_PATH to /usr/local/enrico, so your systemwide openmc isn't clobbered
• I've retained your other compilaation flags and other CMake options

cmake \
-DNEK_DIST=none \
-DCMAKE_C_COMPILER:STRING="/usr/local/mpi/openmpi/bin/mpicc"  \
-DCMAKE_CXX_COMPILER:STRING="/usr/local/mpi/openmpi/bin/mpicxx"  \
-DCMAKE_C_FLAGS:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -fno-strict-aliasing " \
-DCMAKE_C_FLAGS_DEBUG:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -fno-strict-aliasing "  \
-DCMAKE_C_FLAGS_RELEASE:STRING="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -fno-strict-aliasing "  \
-DCMAKE_CXX_FLAGS:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer "  \
-DCMAKE_CXX_FLAGS_DEBUG:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer "  \
-DCMAKE_CXX_FLAGS_RELEASE:STRING="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer  " \
-DCMAKE_INSTALL_PREFIX:PATH="/usr/local/enrico"  \
-DCMAKE_BUILD_TYPE:STRING="Release"  \
-DTHREADS_HAVE_PTHREAD_ARG:BOOL=YES  \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON

[RonRahaman]

I'm also a little unclear why your CMAKE_C_FLAGS_DEBUG and CMAKE_CXX_FLAGS_DEBUG didn't include -g?

[RonRahaman]

I did fresh installs of both standalone OpenMC (v0.12.2) and ENRICO (master branch), and everything looks fine on my end.

I configured OpenMC with this command, using system-wide MPI and HDF5. After compiling and installing, the binaries were in /usr/local

CXX=mpicxx cmake ..

Then I configured ENRICO with this command, also using system-wide MPI and HDF5. After compiling and installing, the binaries were in /usr/local/enrico

CXX=mpicxx CC=mpicc FC=mpifort cmake \
-DNEK_DIST=none \
-DCMAKE_INSTALL_PREFIX=/usr/local/enrico ..

The shared object dependencies for both standalone OpenMC and ENRICO look good. Note that libopenmc.so is resolved to /usr/local/lib/libopenmc.so for standalone OpenMC and to /usr/local/enrico/lib/libopenmc.so for ENRICO.

$ ldd /usr/local/bin/openmc 
	linux-vdso.so.1 (0x00007ffe6dd8b000)
	libopenmc.so => /usr/local/lib/libopenmc.so (0x00007efe3ffff000)
	libmpi_cxx.so.40 => /lib/x86_64-linux-gnu/libmpi_cxx.so.40 (0x00007efe3ffc7000)
	libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007efe3fea1000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007efe3fc88000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007efe3fc6d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efe3fa7f000)
	libhdf5_serial.so.103 => /lib/x86_64-linux-gnu/libhdf5_serial.so.103 (0x00007efe3f70e000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efe3f6ec000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efe3f6e5000)
	libhdf5_serial_hl.so.100 => /lib/x86_64-linux-gnu/libhdf5_serial_hl.so.100 (0x00007efe3f6bf000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efe3f571000)
	libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007efe3f528000)
	libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007efe3f46c000)
	libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007efe3f3ba000)
	libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007efe3f361000)
	/lib64/ld-linux-x86-64.so.2 (0x00007efe40362000)
	libsz.so.2 => /lib/x86_64-linux-gnu/libsz.so.2 (0x00007efe3f35c000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efe3f33e000)
	libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007efe3f306000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007efe3f301000)
	libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007efe3f2fc000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007efe3f2d3000)
	libaec.so.0 => /lib/x86_64-linux-gnu/libaec.so.0 (0x00007efe3f2c8000)

$ ldd /usr/local/enrico/bin/enrico 
	linux-vdso.so.1 (0x00007ffc627fd000)
	libopenmc.so => /usr/local/enrico/lib/libopenmc.so (0x00007f0d587ac000)
	libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f0d5874b000)
	libmpi_cxx.so.40 => /lib/x86_64-linux-gnu/libmpi_cxx.so.40 (0x00007f0d5872d000)
	libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007f0d58607000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0d583ee000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0d5829e000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0d58283000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0d58097000)
	libhdf5_serial.so.103 => /lib/x86_64-linux-gnu/libhdf5_serial.so.103 (0x00007f0d57d26000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0d57d04000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0d57cfd000)
	libhdf5_serial_hl.so.100 => /lib/x86_64-linux-gnu/libhdf5_serial_hl.so.100 (0x00007f0d57cd5000)
	libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007f0d57c19000)
	libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007f0d57b67000)
	libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f0d57b0e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0d58ca4000)
	libsz.so.2 => /lib/x86_64-linux-gnu/libsz.so.2 (0x00007f0d57b09000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f0d57aeb000)
	libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007f0d57ab3000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f0d57aae000)
	libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f0d57aa9000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f0d57a80000)
	libaec.so.0 => /lib/x86_64-linux-gnu/libaec.so.0 (0x00007f0d57a75000)

And finally, I successully ran the coupled short singlerod test case for OpenMC + heat.

Could you confirm that this ENRICO configuration works for you? After you confirm that the minimal configuration works, you can consider which additional configuration options are necessary for your use case.

[yurivict]

I am able to build and install enrico as you described above.

Now it says this:

$ /usr/local/enrico/bin/test_openmc_singlerod
PCI: Failed to initialize libpciaccess with pci_system_init(): 6 (Permission denied)
PCI: Failed to initialize libpciaccess with pci_system_init(): 6 (Permission denied)
 ERROR: Settings XML file 'settings.xml' does not exist! In order to run OpenMC,
        you first need a set of input files; at a minimum, this includes
        settings.xml, geometry.xml, and materials.xml. Please consult the user's
        guide at https://docs.openmc.org for further information.

[RonRahaman]

Which Linux distro are you using? And do you experience this issue with other MPI programs?

[kkiesling]

@yurivict I just made a change in the master branch that should allow for using a pre-existing OpenMC installation instead of installing it from the submodule. Let us know if that fixes this issue for you.