How to Develop Grackle

Grackle is a community project!

We are very happy to accept patches, features, and bugfixes from any member of the community! Grackle is developed using Git, primarily because of how well it enables open-source, community contribution. We’re eager to hear from you.

Note

If you are already familiar with Git and GitHub, the best way to contribute is to fork the main Grackle repository, make your changes, push them to your fork, and issue a pull request. The rest of this document is just an explanation of how to do that.

Keep in touch, and happy hacking!

Open Issues

If you’re interested in participating in Grackle development, take a look at the issue tracker on GitHub. If you are encountering a bug that is not already tracked there, please open a new issue.

Contributing to Grackle with Git and Github

We provide a brief introduction to submitting changes here. We encourage contributions from any user. If you are new to Git and/or GitHub, there are excellent guides available at guides.github.com, specifically the Git Handbook, and the GitHub Hello World. We are also happy to provide guidance on the mailing list or in our slack channel.

Licensing

Grackle is under the Enzo public license, a BSD-like license.

All contributed code must be BSD-compatible. If you’d rather not license in this manner, but still want to contribute, please consider creating an external package, which we’ll happily link to in the Grackle documentation.

How To Get The Source Code For Editing

Grackle is hosted on GitHub, and you can see all of the Grackle repositories at https://github.com/grackle-project/. In order to modify the source code for Grackle, we ask that you make a “fork” of the main Grackle repository on GitHub. A fork is simply an exact copy of the main repository (along with its history) that you will now own and can make modifications as you please. You can create a personal fork by visiting the Grackle GitHub webpage at https://github.com/grackle-project/grackle/. After logging in, you should see an option near the top right labeled “fork”. You now have a forked copy of the Grackle repository for your own personal modification.

This forked copy exists on GitHub under your username, so in order to access it locally, follow the instructions at the top of that webpage for that forked repository:

$ git clone http://bitbucket.org/<USER>/<REPOSITORY_NAME>

This downloads that new forked repository to your local machine, so that you can access it, read it, make modifications, etc. It will put the repository in a local directory of the same name as the repository in the current working directory.

$ cd grackle

Verify that you are on the main branch of Grackle by running:

$ git branch

If you’re not on the main branch, you can get to it with:

$ git checkout main

You can see any past state of the code by using the git log command. For example, the following command would show you the last 5 revisions (modifications to the code) that were submitted to that repository.

$ git log -n 5

Using the revision specifier (the number or hash identifier next to each changeset), you can update the local repository to any past state of the code (a previous changeset or version) by executing the command:

$ git checkout revision_specifier

Making and Sharing Changes

The simplest way to submit changes to Grackle is to do the following:

  1. Fork the main repository.

  2. Clone your fork.

  3. Make some changes and commit them.

  4. Push the changesets to your fork.

  5. Issue a pull request.

Here’s a more detailed flowchart of how to submit changes.

  1. Fork Grackle on GitHub. (This step only has to be done once.) You can do this by clicking on the fork button in the top-right corner of the main repository.

  2. Create a new branch in which to make your changes by doing git checkout -b <new branch name>. This will make it easy to move back and forth between the main branch of the code and your changes.

  3. Edit the source file you are interested in and test your changes.

  4. Use git add <files> to stage files to be committed.

  5. Commit your changes with git commit. This will open a text editor so you can write a commit message. To add your message inline, do git commit -m "<commit message>". You can list specific file to be committed.

  6. Remember that this is a large development effort and to keep the code accessible to everyone, good documentation is a must. Add in source code comments for what you are doing. Add documentation to the appropriate section of the online docs so that people other than yourself know how to use your new code.

  7. If your changes include new functionality or cover an untested area of the code, add a test. Commit these changes as well.

  8. Push your changes to your new fork using the command:

    $ git push origin <branch name>

    Note

    Note that the above approach uses HTTPS as the transfer protocol between your machine and GitHub. If you prefer to use SSH - or perhaps you’re behind a proxy that doesn’t play well with SSL via HTTPS - you may want to set up an SSH key on GitHub. Then, you use the syntax ssh://git@github.com/<USER>/grackle, or equivalent, in place of https://github.com/<USER>/grackle in git commands. For consistency, all commands we list in this document will use the HTTPS protocol.

  9. Issue a pull request by going to the main repository and clicking on the green button that says Compare & pull request. This will open up a page that will allow you to enter a description of the changes to be merged. Once submitted, a series of automated tests will run and their status will be reported on the pull request page.

During the course of your pull request you may be asked to make changes. These changes may be related to style issues, correctness issues, or requesting tests. The process for responding to pull request code review is relatively straightforward.

  1. Make requested changes, or leave a comment on the pull request page on GitHub indicating why you don’t think they should be made.

  2. Commit those changes to your local repository.

  3. Push the changes to your fork:

    $ git push origin <branch name>

  4. Your pull request will be automatically updated.

Once your pull request has been accepted, you can safely delete your branch:

$ git branch --delete <branch name>

Updating Your Branch

If your branch or pull request has been open for some time, it may be useful to keep it up to date with the latest changes from the main repository. This can be done by rebasing your changes. Before doing this, you will need to be able to pull the latest changes from the main repository.

  1. Add the main repository as a remote:

    $ git remote add grackle https://github.com/grackle-project/grackle

    You can verify that it has been added by doing git remote -v. This only needs to be done once.

  2. Go back to the main branch and pull the changes:

    $ git checkout main
$ git pull grackle main

  3. Return to your branch and rebase your changes onto the head of the main branch:

    $ git checkout <branch name>
$ git rebase main

This should go smoothly unless changes have been made to the same lines in the source, in which case you will need to fix conflicts. After rebasing, you will get an error when trying to push your branch to your fork. This is because you have changed the order of commits and git does not like that. In this case, you will need to add “-f” to your push command to force the changes to be accepted.:

$ git push -f origin <branch name>

Have fun!

Adding a New Parameter

This section provides a short list of tasks to complete when a new field is introduced to the chemistry_data struct:

  1. Add a new entry to src/clib/grackle_chemistry_data_fields.def for this field. This file holds a central list of fields in chemistry_data. Doing this accomplishes the following three things:

    1. the field is initialized to the appropriate default value.

    2. the new field and its value are printed when grackle_verbose = 1

    3. the field can be accessed by the functions providing the dynamic access to members of chemistry_data (see Dynamic configuration of Chemistry Data)

    4. the new field is accessible from Pygrackle (because that interface uses the functions providing dynamic access)

  2. Update the fortran definition for the grackle_chemistry_data type (in src/clib/grackle_fortran_interface.def). This type must exactly match the definition of chemistry_data and is used to integrate Grackle into simulation codes written in Fortran.

  3. Add documentation in doc/source/Parameters.rst for your new parameter.

CMake Build-System Design Rationale

The design rationale for the CMake build-system tries to walk a fine line between 2 competing philosophies:

  1. We should provide a curated, convenient out-of-box experience for most people who are building Grackle.

  2. The internals of a modern CMake build-system should only specify the firm build requirements. All other choices of compilation options should be specified through one of the many standardized “hooks” or options that CMake provides for customization and overriding default behavior. This is important for making a project easily consumable (as either a standalone project or an embedded build).

The problem with the second philopsphy is that it assumes that the person building software from source is already well-versed in CMake (e.g. a developer, someone who will package and distribute your software, someone who wants to directly embed your software within their project), or they are a motivated developer/user who can be expected to learn CMake. It implicitly assumes that most users of a project won’t ever need to directly build your software from source; they will instead install prebuilt and packaged copies of the software that are distributed through other channels (e.g. through package managers like apt, dnf, homebrew OR downloadable precompiled binaries through a website OR some kind of installer). While this implicit assumption may be accurate for most CMake software, it obviously doesn’t apply to scientific software).

It may be tempting to dismiss the second philosophy for scientific software. In fact, it doesn’t provide substantial benefits in build-systems that simply build an application (like a simulation code). However, it provides substantial benefits for build-systems of libraries (like Grackle) that are used as components in downstream software. It does a lot to make the libraries easier to consume.

Consequently, our CMake build-system pursues a pragmatic compromise between these philosophies. We clearly specify all build requirements (this is compatible with both). We provide the ability to include commonly desired compilation options (some of these may be specified hostfiles). However, these extra (not strictly necessary) options must all be introduced in a manner that they are easily enabled/disabled, without breaking a CMake build. This compromise allows both regular users and CMake experts (including people who want to embed Grackle into their simulation code) to easily interact with the build system.

Supporting pkg-config

This section assumes you are already familiar with this part of the documentation. This section exists for posterity and to explain some design choices (especially since there appears to be a dearth of authoritative documentation on this topic).

Overview

Our main focus is pkg-config’s --static flag, in the case where CMake was used to create a Grackle installation that includes the shared version and the static version of Grackle (while we focus on Grackle, this discussion has implications for other libraries).

In this scenario, the man-pages (and online discussions) for pkg-config suggest that

  • pkg-config --cflags grackle and pkg-config --libs grackle respectively supply the compiler and linker flags for consuming grackle as a shared library

  • pkg-config --static --cflags grackle and pkg-config --static --libs grackle respectively supply the compiler and linker flags for consuming grackle as a static library

Everything is perfectly fine in the first pair of invocations. In the second pair of invocations, the compiler flags are fine (other than some unnecessary information leaking through), but there are fundamentally 2 issues associated with the linker flags:

  1. The primary issue relates to how pkg-config determines the static linker flags. For now, we ignore dependencies like hdf5 (more on that shortly). In more detail:

    • grackle.pc internally tracks the 2 set of linker flags. The first set is used in the absence of the --static flag; in this scenario, these would be used for linking grackle as a shared library. This currently might look something like -L<path/to/installed/grackle> -lgrackle. The file also separately tracks private linker-flags that are intended to be used for this scenario when you use Grackle as a static library. These are somewhat specific to the compiler toolchain and machine/OS and depend on how Grackle was compiled. At the time of running, they usually specify implicit dependencies. For concreteness, this might look something like -lgomp -lgfortran -lm on a Linux machine (hdf5 is deliberately omitted).

    • In this scenario, pkg-config --static --libs grackle determines the list of flags by ostensibly concatenating both sets of linker flags. Importantly, this means that -lgrackle is specified in both cases.

    • When a linker is provided -lgrackle, it will search for a file called libgrackle.so (or libgrackle.dynlib on macOS) OR libgrackle.a. If both files are present (as is the case in this scenario), most linkers will prefer the shared library version.

    • The approach to inform the linker of a preference to use the static-library that is most portable and easiest-to-express is to prepend a flag like -Bstatic to the start of the flags returned by pkg-config --static --libs grackle. [1] Consequently, the linker will prefer to try to link every other required library statically (we return to this shortly). MORE IMPORTANTLY, there isn’t any completely portable way to do this; some linkers (namely the one on macOS [2] ) don’t provide ANY convenient way to express this preference.

  2. Issues dealing with dependencies like hdf5:

    • There are actually 2 ways to deal with linker flags of private dependencies inside grackle.pc: (i) directly specify the private linker flags (as described in the preceeding bullets) or (ii) specify a “private requirement” on the dependency. The latter option is only possible if the dependency provides its own *.pc file (indeed, hdf5 usually comes with a hdf5.pc) file.

    • Recall from the above bullets that when using pkg-config’s --static flag, we effectively need to tell the linker to prefer linking ALL of Grackle’s dependencies statically. With that in mind, we really should prefer the “private requirement” approach for specifying hdf5 as a private dependency of Grackle. In slightly more detail, libhdf5.a has its own set of nontrivial private dependencies, and by using the “private requirement” approach, pkg-config should automatically handle those dependencies for us.

    • Unfortunately, common hdf5 installations don’t properly specify these private linker flags on commonly used platforms. For example, on a Debian or Ubuntu system, the hdf5.pc file installed alongside hdf5 with apt doesn’t actually specify ANY private linker flags (this seems to be a common occurence). Additionally, the hdf5.pc file installed with HDF5 version 1.14.3 by Homebrew on macOS includes invalid private linker flags [3] .

It’s for the above reasons that we have choosen not to support static-linking (via pkg-config) in installations where both shared and static libraries are provided. If people want to use static libraries via pkg-config, we instead encourage an installation with just a static-library. In an installation that just provides a static-library, the contents of grackle.pc are customized to dynamically link every dependency other than Grackle (and there won’t be any linker-issues differentiating between the shared and static libraries).

A potential workaround

A potential work-around for most of the above difficulties when you have a shared library and static library installed is that we create a new file called grackle_static.pc that gets shipped alongside grackle.pc. To get the flags for using grackle as a static library, you would invoke pkg-config --cflags grackle_static and pkg-config --libs grackle_static. Essentially, the contents of grackle_static.pc would look a lot like the contents of grackle.pc when only a static library is installed.

While this wouldn’t solve the issue of telling the linker to use libgrackle.a on macOS, it would solve the other issues. Even if we could overcome that remaining issue, this workaround is undesirable (unless there is strong user-demand) for a number of reasons:

  • the primary reason is we would have more to maintain, and this workaround is totally unnecessary outside of a somewhat pathological case (Users need to go somewhat out of their way to use the CMake build-system to create a single installation that features both the static and shared versions of Grackle). The much easier/more sensible/more idiomatic default behavior is to compile Grackle just as a shared library or just as a static library (in which case there isn’t any problem).

  • this is far less “composable” than the existing alternative of requiring the end-user to compile Grackle just as a shared library or just as a static library. Under the existing alternative, the build-system of a downstream application will invoke the same commands commands in either scenario. If we shipped grackle_static.pc, the build system of a downstream application would need to embed additional logic to proactively alter the arguments passed to pkg-config based on the preference for shared and static libraries.

  • by the principle of least surprise there would be a solid argument to also supply the grackle_static.pc file in the case where we just install grackle as a static-library.

  • the solution wouldn’t scale well. As development progresses, it is conceivable that we may want to easily support having multiple versions of grackle installed where we support different “backends.” For example, we might hypothetically support grackle.pc, grackle_openmp.pc and grackle_cuda.pc. Alternatively, one could also imagine hypothetically imagine supporting grackle_float.pc (for a version compiled in single-precision). If we ever go down any of these roads, we would also be somewhat obligated to manually manage variants of each file with/without the _static suffix.

Instructions for Making a Release

Before reading this section, please ensure you are familiar with our versioning policy.

To create a new release:

  1. Draft the GitHub Release (using the GitHub website) and draft the changes to the changelog.

    • It’s often easiest to start by drafting the GitHub release because GitHub provides the option to automatically generate a list of the titles and links to all pull-requests that have been merged since the previous release. That functionality provides a list of all first-time contributors.

      • We recommend looking to older release notes as a guide. With that said, our release generally consists of (i) a short summary about the release, (ii) a list of the changes, and (iii) a list of all contributors (that highlights first-time contributors).

      • The automatically generated list of pull requests is an excellent way to start producing the list of changes. But, the list entries may need to be slightly modified. For example, PRs that simply update the version number tracked inside the repository should be removed (if present), it may make logical sense to group a series series of closely related PRs into a single entry, or an entry may need to be slightly more descriptive. We also like to sort the list-entries into sections (e.g. New Features, Minor Enhancements, Bugfixes, Documentation Updates, etc.). Finally, it makes sense to highlight any functions in the public API that have been deprecated or removed.

    • After you draft the GitHub release, you can take advantage of GitHub’s feature to save the draft and circulate it to other developers for comments/recommendations.

    • It’s fairly straightforward to draft changes to the changelog based on the contents of the GitHub release.

  2. Create a new PR that will serve as the final PR of the release. The PR should:

    • update the changelog (the changelog is stored within the CHANGELOG file at the root of the repository).

    • update the version number of the c-library. This is currently tracked within the VERSION file (that can be found at the root level of the repository)

    • (if applicable) update the version number of the python module (stored internally in pyproject.toml)

  3. After the PR is merged, perform the release on GitHub. Make sure to associate the release with the proper target (it should include changes from the PR).

  4. Make a new PR composed of a single commit (this is the first commit of the next release). It should only update the version number of the c-library to specify the next development version

    • like before, this must be updated in the VERSION file (that can be found at the root level of the repository). Instructions for choosing development version numbers are provided here (note that this changed in the version 3.3 release).

    • do NOT touch anything else (even the python version number)

  5. Go to the Read the Docs project webpage and the new release’s tag to the list of “Active Versions.” This ensures that Read the Docs will maintain a version of the documentation for this particular release.

  6. Announce the release in an email to the Grackle mailing list. This can be adapted from the GitHub Release notes.

Footnotes