Installation
This page describes how to install Grackle from source. When installing Grackle as a dependency of a simulation code, consult the code’s documentation before continuing. Build systems of some codes, like Enzo-E, are configured to automatically download & install Grackle as part of the build-process (our Integration Guide provides more details).
There are 3 steps to setting up Grackle on your system
Attention
Please use the CMake build system (it is far more robust and maintainable) and let us know if you encounter any problems.
The classic build system is deprecated since version 3.4 and is scheduled for removal in version 3.5.
We include a note on compiler toolchain compatability at the end of this page.
Dependencies
In addition to C/C++ and Fortran compilers, the following dependency must also be installed:
HDF5, the hierarchical data format. HDF5 also may require the szip and zlib libraries, which can be found at the HDF5 website.
Note
If using the classic build system: compiling with HDF5 1.8 or greater requires that the
H5_USE_16_APIcompatability directive is manually specified. This can be done by manually adding-DH5_USE_16_APIto the list of compiler flags given in machine-specific make files.You don’t have to think about this when using CMake build system (it is handled automatically).
Although many systems already have them installed, both build systems have additional dependencies:
the CMake build system requires cmake to be installed. It’s easiest to download a binary distribution from the CMake website or use your system’s package manager. We require version 3.16 or newer.
the classic build system, employs the
makedependand the libtool utilities. It’s often easiest to download these dependencies through your system’s package manager.
Downloading Source Code
Grackle is available in a git repository here. Excellent guides to git and GitHub are available at guides.github.com. To clone the Grackle repo, do the following:
~ $ git clone --recursive https://github.com/grackle-project/grackle
The presences of the --recursive flag in the above snippet instructs git to fetch additional files, containing cooling tables, from a submodule linked to the Grackle repository.
If you have already cloned the git repository and you forgot to include the --recursive flag, you can invoke git submodule update --init.
Building with CMake
Grackle’s primary build-system uses Modern CMake. While CMake has some baggage (primarily due to the maintenace of backwards compatability), it is arguably the most-portable mainstream build-system that is easiest to integrate with simulation codes.
An overview of our design philosophy is provided here.
This build-system makes integration of Grackle into simulation codes that are themselves built with CMake extremely easy.
Steps have also been taken simplify integration of Grackle into simulation codes built with any other build-systems (they just need to call the standardized pkg-config command-line tool).
More details about integration are provided on this page.
This current section focuses on installation.
For the uninitiated, the CMake build-system performs an out-of-source build.
An out-of-source build places all build artifacts (auto-generated source/header files, object files, etc.) into a “build-directory.”
The build-directory is at a user-specified location that is organized into a hierarchy that resembles the source directory hierarchy.
Cleaning up from a CMake-build is as simple as deleting this build-directory.
In contrast, the “classic build system” performs an in-source build (because that type of build distributes build artifacts throughout the source directory hierarchy, clean up requires more complex logic encapsulated by the make clean command).
Warning
While the “classic build system” has been modified to better coexist with the CMake build-system, issues can potentially arise if build-artifacts produced in a “classic” build of an earlier Grackle-revision are not properly removed. Specifically, the issues relate to the presence of auto-generated header-files. We have built checks into the CMake build-system to prevent these issues in most cases, but they may not help in certain pathological scenarios.
Procedure
The build/installation procedure follows the standard steps of any CMake build. The remainder of this subsection is primarily intended for readers who are relatively inexperienced with using CMake.
Proceed to the grackle directory
~$ cd grackleInitialize and configure the build-system. In these example snippets, we show the minimum required configuration options (this should work on most machines) and provide more details later about additional configuration options and how to specify configuration options down below. During this step you might also specifiy machine-specific host files (but that usually isn’t absolutely necessary).
For now, we make 2 basic decisions:
Decide on the directory,
<build-dir>, where you want to build Grackle. [1] This is referred to as the build-directory and is generally placed at the root level of the grackle repository. A common choice isbuild(but this is fairly arbitrary).Decide on the installation directory prefix,
<install-prefix>, where Grackle will be installed. This is be specified via theCMAKE_INSTALL_PREFIXcmake configuration variable. On UNIX-like systems, it defaults to/usr/local/.
The following snippets illustrates how to configure Grackle as a static/shared library:
~/grackle $ cmake -DCMAKE_INSTALL_PREFIX=<install-prefix> -B <build-dir>~/grackle $ cmake -DCMAKE_INSTALL_PREFIX=<install-prefix> -DBUILD_SHARED_LIBS=ON -B <build-dir>Note
If you are building Grackle to be used with a downstream simulation-code, that doesn’t mention any preferences about how Grackle is built, you will probably have more luck compiling Grackle as a shared library.
It is idiomatic for a given CMake build to just compile Grackle as either a static or shared library, not both (you usually just need one). But if you must have both, see this section.
Compile and install grackle.
~/grackle $ cmake --build <build-dir> # the build-step ~/grackle $ cmake --install <build-dir> # the install-step
Hint
The above snippet shows the most generic commands that can be executed. Other tutorials that you see online may show slight variations in these commands (where you manually make the build directory) and then manually execute the build-system from within the build-directory…
Note
In some cases, projects can use Grackle, built with CMake, without requiring a full installation. But, for historical reasons, you should generally assume that an external project requires a full installation step (unless that’s project tells you otherwise).
If you install Grackle in a non-standard location, then you also need to ensure that you properly set the
LD_LIBRARY_PATH(orDYLD_LIBRARY_PATHon macOS) env variable to make use of it.Test your Build.
Once you have compiled Grackle, you can run one of the provided example to test if it functions correctly. These examples are automatically compiled with Grackle.
~/grackle $ cd <build-dir>/examples ~/grackle/<build-dir>/examples $ ./cxx_example
Warning
The examples make certain assumptions about the location of the input files. The examples are only guaranteed to work if both:
you execute the example-binary from the same-directory where the example-binary is found
<build-dir>is a top-level directory in the grackle repository (e.g. something likemy-buildis fine, but choices like../my-grackle-buildandmy_builds/my-first-buildare problematic).
Note
For reference, the Classic build-system always links Grackle against the shared-library version of Grackle and requires that Grackle is fully installed in a location known by the system (either a standard system location OR a location specified by
LD_LIBRARY_PATH/DYLD_LIBRARY_PATH). In contrast, cmake automatically takes special-steps to try to ensure that each example-binary will link to the copy of the Grackle library (whether it is shared or static) that is in the<build-dir>; in fact, Grackle doesn’t even need to be installed to run the Grackle library.With that said, if you compile Grackle as a shared library in a cmake build, an example-binary might try to use a copy of a shared grackle library found in a directory specified by
LD_LIBRARY_PATH/DYLD_LIBRARY_PATHif one exists. The exact behavior may be platform dependent and also depends on whether CMake instructs the linker to use RPATH or RUNPATH (this is not specified by the cmake docs).
In order to verify that Grackle is fully functional, you can try running the test suite.
How to Specify Configuration Options
All configuration options can be specified when invoking cmake during configuration of the build.
Specifically you can specify the values by inserting an argument of the form -D<variable>=<value> to the list of arguments passed to cmake.
This is illustrated in the prior subsection where we pass -DCMAKE_INSTALL_PREFIX=/my/install/path... and -DBUILD_SHARED_LIBS=OFF.
Alternatively, you can replace the call to cmake during configuration with a call to ccmake to provide a TUI (text-based user interface) where you can manually configure options.
For example, a call to ccmake -B<build-dir> will bring up a TUI to configure a build in the specified directory.
CMake also provides a GUI (graphical user interface) for this purpose (it may not be available based on how exactly you installed CMake).
The CMake documentation provide more details about the GUI and how to more generally use cmake here.
A summary of all Grackle-specific configuration options and a subset of useful generic CMake configurations is provided in the next subsection.
The idiomatic way to control optimization/debugger flags is to specify a build-type via the standard CMAKE_BUILD_TYPE variable.
Choices include:
-DCMAKE_BUILD_TYPE=Release(typically-O3)-DCMAKE_BUILD_TYPE=RelWithDebInfo(typically-O2 -g)-DCMAKE_BUILD_TYPE=Debug(typically-O0 -g)
The first choice is generally fastest, while the second is a sensible choice during development (the compiler performs most optimizations and includes debugging information in the library).
Machine-specific compilation options can also be specified with host-files.
These host-files should generally not be necessary, but they may specify architecture-specific optimization flags.
This should be specified during the configuration stage with the -C flag followed by the path to the host-file.
For example, one might invoke:
~/grackle $ cmake \
-C config/tacc-frontera-intel.cmake \
-D CMAKE_INSTALL_PREFIX=<install-prefix> \
-D BUILD_SHARED_LIBS=ON \
-B <build-dir>
The order of -D and -C flags matters.
If they are both used to specify values for a given variable, the last one to appear “wins.”
More information about writing host-files are provided below.
Available Configuration Options
The compilation (and installation) of Grackle can be configured using various options. These options are described in the following 2 tables.
Many of these options are binary choices that accept a boolean value. [2]
This first table describes the Grackle-specific options to configure the build.
Name |
Description |
Default |
|---|---|---|
|
Turn off to build Grackle with single precision. |
|
|
Turn on to build Grackle with OpenMP |
|
|
Turn on to build tests of the core Grackle library (these are totally unrelated to the |
|
This second table highlights a subset of standardized CMake options that may also be useful.
Name |
Description |
Default |
|---|---|---|
|
When turned |
|
|
Specifies the desired build configuration (for single-configuration generators [4]).
Grackle currently supports the standard choices |
|
|
Specifies the path-prefix where Grackle will be installed when you invoke |
|
|
When cmake has trouble finding your hdf5 installation, you can set this variable equal to the path to the HDF5 installation to serve as a hint for cmake |
|
|
Set to |
|
|
Set of variables (where |
|
There are also additional standard options for BOTH configuring other aspects of the build and for finding the correct/preferred HDF5 library and configuring the correct openmp library.
Addtionally, CMake will also respect the values of certain environment variables.
For example, if you don’t manually specify the choice of compilers with the CMAKE_<LANG>_COMPILER flag, then CMake will use the values in the CC, FC, and CXX environment variables.
We strongly encourage users and developers to make use of the options described in this section. They exist to provide a curated/consistent experience in a variety of scenarios. Please let us know if you think we are missing a useful Grackle-specific option. You can also add the new option yourself (it may be useful to review the design philosophy for the CMake build-system).
With that said, we also recognize that the need may arise where a user/developer may want to specify arbitrary flags.
You can use the standardized CMAKE_<LANG>_FLAGS variables for that purpose (where <LANG> is C, CXX, Fortran).
For example, passing -DCMAKE_C_FLAGS="-Wall -Wpedantic -funroll-loops" will pass these flags to every invocation of the C compiler (for compiling Grackle itself as well as any examples or tests).
Technically, these flags are passed to every invocation of the C compiler-frontend (even during linking), but that usually isn’t a problem.
More About Host-Files
As noted above, we provide support for setting default value for particular machines by providing support for host-files. These files are provided mostly for convenience (and to provide parity with machine files provided by the classic build-system). They are most useful on HPC systems that provide multiple compiler toolchains. These are the *.cmake files in the config directory.
Importantly, the usage of host-files is optional (and usually not required). They usually aren’t needed on local systems (if you find that Grackle won’t compile without a host-file, please let us know – that may indicative of a bug). They should generally NOT be used when Grackle is embedded within another CMake project.
While there are a couple of ways to implement this concept, our current strategy draws some inspiration from here. Essentially, our strategy leverages cmake functionality to pre-load a script to populate some cache variables.
Usually, will specify the desired compilers.
If a HPC machine properly manages the CC, FC, and CXX environment variables this isn’t strictly necessary.
If the machine places HDF5 in an unusual location, you might also hardcode hints into the config-file.
The most important role is to specify cluster-specific optimization flags via the special Grackle-specific GRACKLE_OPTIMIZATION_FLIST_INIT variable.
These flags will ONLY be used when compiling Grackle with the Release or RelWithDebInfo build-types.
Here are 2 illustrative examples:
First we show that in order to pass multiple flags, the flags need to be specified by a semicolon delimited list. If you stored
"-xCORE-AVX512;-funroll-loops"withinGRACKLE_OPTIMIZATION_FLIST_INIT, then all source files will be compiled with these options (they won’t be passed to the linker).Next we show that to properly pass “option groups” you may need to make use of CMake’s shell-like quoting with the
SHELL:prefix (this relates to option de-duplication performed by CMake). Thus, storing"SHELL:-option1 A;-Wall;SHELL:-option2 B"withinGRACKLE_OPTIMIZATION_FLIST_INITwould cause all compiler invocations for source files used in Grackle to be passed-option1 A -Wall -option2 B.
While embedded builds currently respect GRACKLE_OPTIMIZATION_FLIST_INIT, that is something we may stop supporting.
Note
If you want to pass language-specific optimization options, let us know. That is something we can easily support. Until then, this could be addressed by enclosing a given option (or option-group) within a language-specific generator expressions.
Note
In terms of modern, idiomatic CMake features, a host-file could be replaced by a combination of a toolchain-file and a preset-file.
toolchain files usually define compiler-toolchain related-options and are commonly used for cross-compiling. As a basic rule of thumb: you should be able to recycle toolchain-files between unrelated projects (i.e. they don’t include project-specific variables)
a preset file (
CMakePresets.jsonorCMakeUserPresets.json) is intended to be used to specify common project-specific compilation options. These can be read by IDEs.after we update the minimum required CMake version for compiling Grackle to at least 3.19, we may transition to using these features.
CMake Troubleshooting
This section discusses how to resolve some common issues that could arise while building Grackle with the CMake build-system.
“Could NOT find HDF5”
CMake could not find the hdf5 installation.
If you are on a local machine (not a cluster) consider the following scenarios:
Did you remember to install hdf5?
If you installed hdf5 with a package manager, did you make sure that the package includes files for development? (For example, apt commonly supports a
libhdf5-<vers>package that only contains a shared library and alibhdf5-devpackage that supports everything you need).
If you are confident that HDF5 is installed, you can provide a hint about its location with the HDF5_ROOT cmake-configuration variable (you can also use HDF5_DIR, but the semantics are a little different).
“Fatal Error: omp_lib.h: No such file or directory”
This error appears when configuring a CMake build using the Ninja backend[6] and with GRACKLE_USE_OPENMP=ON.
The most robust solution: “have CMake use the default (Makefile) backend.”
In more detail, this error seems related to a preprocessing step of Fortran source files that is related to Ninja. This preprocessing step may be for generating module dependency information (as described here).
Note
There are ongoing efforts to convert all internal source code from Fortran to C++. By the 3.5 release, the core library should no longer contain any Fortran code (and this problem will be moot)
Generic Fortran Compiler Errors
There are some well-known bugs between Fortran and CMake’s Ninja backend[6] (these seem more prominent if you are using a compiler other than gfortran). You could try to update CMake and Ninja to their latest versions. The most robust solution: “have CMake use the default (Makefile) backend.”
Note
There are ongoing efforts to convert all internal source code from Fortran to C++. By the 3.5 release, the core library should no longer contain any Fortran code (and this problem will be moot)
Building with Classic Build-System
Attention
This build system is deprecated since version 3.4 and is scheduled for removal in version 3.5.
Please use the CMake build system (it is far more robust and maintainable) and let us know if you encounter any problems.
The classic compilation process for grackle is very similar to that of Enzo. For more details on the Enzo build system, see the Enzo build documentation. To compile Grackle, complete the following procedure:
Initialize the build system.
~ $ cd grackle
~/grackle $ ./configure
Proceed to the source directory.
~/grackle $ cd src/clib
Configure the build system.
Note
As of version 2.1, Grackle uses libtool for building and installation.
As such, both shared and static libraries will be built automatically and
it is not necessary to add the -fPIC compiler flag.
Compile settings for different systems are stored in files starting with
“Make.mach” in the source directory. Grackle comes with three sample make
macros: Make.mach.darwin for Mac OSX, Make.mach.linux-gnu for
Linux systems, and an unformatted Make.mach.unknown. If you have a make
file prepared for an Enzo install, it cannot be used straight away, but is a
very good place to start.
Once you have chosen the make file to be used, a few variables should be set:
MACH_LIBTOOL- path tolibtoolexecutable. Note, on a Mac, this should point toglibtool, which can be installed with macports or homebrew.
LOCAL_HDF5_INSTALL- path to your hdf5 installation.
LOCAL_FC_INSTALL- path to Fortran compilers (not including the bin subdirectory).
MACH_INSTALL_PREFIX- path where grackle header and library files will be installed.
MACH_INSTALL_LIB_DIR- path where libgrackle will be installed (only set if different from MACH_INSTALL_PREFIX/lib).
MACH_INSTALL_INCLUDE_DIR- path where grackle header files will be installed (only set if different from MACH_INSTALL_PREFIX/include).
Once the proper variables are set, they are loaded into the build system by doing the following:
~/grackle/src/clib $ make machine-<system>
Where system refers to the make file you have chosen. For example, if you
chose Make.mach.darwin, type:
~/grackle/src/clib $ make machine-darwin
Custom make files can be saved and loaded from a .grackle directory in the home directory.
Compiler Settings
There are three compile options available for setting the precision of baryon fields, compiler optimization, and enabling OpenMP. To see them, type:
~/grackle/src/clib $ make show-config
MACHINE: Darwin (OSX)
MACHINE-NAME: darwin
CONFIG_PRECISION [precision-{32,64}] : 64
CONFIG_OPT [opt-{warn,debug,high,aggressive}] : high
CONFIG_OMP [omp-{on,off}] : off
For example, to change the optimization to high, type:
~/grackle/src/clib $ make opt-high
Warning
Compiling Grackle in single precision (with make precision-32) is not
recommended. Because of the high dynamic range involved in calculating many
chemistry and cooling rates, running Grackle in single precision can produce
unreliable results. This is especially true when running with
primordial_chemistry >= 1.
Custom settings can be saved for later use by typing:
~/grackle/src/clib $ make save-config-<keyword>
They will be saved in the .grackle directory in your home directory. To reload them, type:
~/grackle/src/clib $ make load-config-<keyword>
For a list of all available make commands, type:
~/grackle/src/clib $ make help
========================================================================
Grackle Makefile Help
========================================================================
make Compile and generate librackle
make install Copy the library somewhere
make help Display this help information
make clean Remove object files, executable, etc.
make dep Create make dependencies in DEPEND file
make show-version Display revision control system branch and revision
make show-diff Display local file modifications
make help-config Display detailed help on configuration make targets
make show-config Display the configuration settings
make show-flags Display specific compilation flags
make default Reset the configuration to the default values
Compile and Install
To build the code, type:
~/grackle/src/clib $ make
Updating DEPEND
Compiling calc_rates.F
Compiling cool1d_multi.F
....
Linking
Success!
Then, to install:
~/grackle/src/clib $ make install
Test your Installation
Once installed, you can test your installation with the provided example to
assure it is functioning correctly. If something goes wrong in this process,
check the out.compile file to see what went wrong during compilation,
or use ldd (otool -L on Mac) on your executable to determine what went
wrong during linking.
~/grackle/src/clib $ cd ../example
~/grackle/src/example $ make clean
~/grackle/src/example $ make
Compiling cxx_example.C
Linking
Success!
~/grackle/src/example $ ./cxx_example
The Grackle Version 2.2
Mercurial Branch default
Mercurial Revision b4650914153d
Initializing grackle data.
with_radiative_cooling: 1.
primordial_chemistry: 3.
metal_cooling: 1.
UVbackground: 1.
Initializing Cloudy cooling: Metals.
cloudy_table_file: ../../input/CloudyData_UVB=HM2012.h5.
Cloudy cooling grid rank: 3.
Cloudy cooling grid dimensions: 29 26 161.
Parameter1: -10 to 4 (29 steps).
Parameter2: 0 to 14.849 (26 steps).
Temperature: 1 to 9 (161 steps).
Reading Cloudy Cooling dataset.
Reading Cloudy Heating dataset.
Initializing UV background.
Reading UV background data from ../../input/CloudyData_UVB=HM2012.h5.
UV background information:
Haardt & Madau (2012, ApJ, 746, 125) [Galaxies & Quasars]
z_min = 0.000
z_max = 15.130
Setting UVbackground_redshift_on to 15.130000.
Setting UVbackground_redshift_off to 0.000000.
Cooling time = -1.434987e+13 s.
Temperature = 4.637034e+02 K.
Pressure = 3.345738e+34.
gamma = 1.666645e+00.
In order to verify that Grackle is fully functional, try running the test suite.
Compiler Toolchain Compatability
As a general rule of thumb, the easiest, most reliable thing to do is to ensure that Grackle is built with the same compiler toolchain (or a compatible one) as the
the downstream application itself (whether it’s a simulation code or pygrackle)
any other dependencies of the application (whether it’s other software libraries or other python extension-modules loaded at the same time).
This is only something you need to consider on platforms with multiple compiler toolchains present.
In practice, toolchain-compatibility generally ISN’T much of a concern for Grackle, when compiled without OpenMP.
In this scenario, you need to use Fortran compilers with consistent runtime libraries (e.g. you might encounter issues if you use gfortran to compile Grackle and ifort to compile a downstream simulation code).
If the downstream application doesn’t use any Fortran, then there generally aren’t any concerns at all.
Things are slightly more complex when compiling Grackle with OpenMP.
You need to make sure that your C compiler and Fortran compiler use a compatible OpenMP runtime.
Usually, your best bet is to try to use C and Fortran compilers from the same vendor (e.g. using gcc with gfortran will work or using icc with ifort will work).
You might be able to mix compilers from different vendors by passing special compiler and linker options, but this usually isn’t well documented.
If your downstream application is also compiled with OpenMP, you also need to ensure that the downstream application is compiled with a compatible runtime.
You don’t generally need to worry about OpenMP-compatability between Grackle and the rest of the software stack if Grackle is compiled without OpenMP or if it is the only part of the software stack that is compiled with OpenMP.
As Grackle continues to evolve, compiler toolchain compatability will become more of an issue. For example, adding GPU-support with the likes of CUDA or HIP would involve linking to a C++ runtime library.
Note
Mixing compiler toolchains may be more difficult for certain vendors. For example, some vendors may more aggressively link their OpenMP runtime library or C++ runtime-library libraries to the resulting binaries, which could easily cause problems. But generally, GNU-compilers and clang are pretty good about this.
Footnotes