diff --git a/docs/adding_software/adding_development_software.md b/docs/adding_software/adding_development_software.md new file mode 100644 index 000000000..b3716b02b --- /dev/null +++ b/docs/adding_software/adding_development_software.md @@ -0,0 +1,155 @@ +# Adding software to `dev.eessi.io` + +!!! warning "`dev.eessi.io` is still in active development and focused on MultiXscale" + + The `dev.eessi.io` repository and functionality is still in its early stages. + The repository itself and build + deploy procedure for it are functional, but may change + often for the time being. + + Our focus is currently on including and supporting developers and applications in the + [MultiXscale CoE](https://multixscale.eu). + +## What is `dev.eessi.io`? + +`dev.eessi.io` is the [development repository of EESSI](../repositories/dev.eessi.io.md). + +## Adding software + +Using `dev.eessi.io` is similar to using EESSI's production repository `software.eessi.io`. +Software builds are triggered by a [bot](https://www.eessi.io/docs/bot/) listening to pull +requests in [GitHub repositories](https://github.com/search?q=org%3AEESSI+dev.eessi.io&type=repositories). +These builds require custom easyconfig and easystack files, which should be in specific directories. + +To see this in practice, refer to the [dev.eessi.io-example GitHub repository](https://github.com/EESSI/dev.eessi.io-example). + In this GitHub repository you will find templates for some software installations with the appropriate directory structure, that is: + +``` +dev.eessi.io-example +├── easyconfigs +└── easystacks +``` + +### Quick steps to build for `dev.eessi.io` + +- Obtain commit ID from GitHub or GitLab repository with source to build. +- Fork the project's `dev.eessi.io` repository on GitHub, or checkout to a new branch if you can do so. +- If needed, prepare an easyconfig template using `--software-commit` and add it to `easyconfigs/` +- Add an easystack file in `easystacks/` that with the easyconfig file above, add the +commit ID to `software-commit` under `options`. +- Open a PR from the fork or branch to the main branch of the application's `dev.eessi.io` GitHub repository. +- Instruct the bot to start a build by adding a comment with `bot: build`. +- Confirm the build worked correctly. If so, you can deploy the software by adding the label `bot:build` to the pull request. +- Once the staging PR is approved, the development build will become available on `dev.eessi.io` in a few minutes! + +### Installation details + +#### easyconfig files and `--software-commit` +The approach to build and install software is similar to that of `software.eessi.io`. +It requires one or more easyconfig files. Easyconfig files used for building for `dev.eessi.io` +do not need to be a part of an [EasyBuild release](https://github.com/easybuilders/easybuild-easyconfigs), unlike builds for +`software.eessi.io`. In this case, the development easyconfigs can be located under `easyconfigs/` in the `dev.eessi.io` repository being used. + +To allow for development builds, we leverage the `--software-commit` functionality (requires [EasyBuild](https://easybuild.io/) v4.9.3 or higher). This lets us build a given application from +a specific commit in repository. This can also be done from a fork, by changing the `github_account` field in the easyconfig file. +We've created a template for `ESPResSo` based on the standard eaasyconfig of the most recent version. The relevant fields are: + +``` python +easyblock = 'CMakeMake' + +name = 'ESPResSo' +version = 'devel' +versionsuffix = '-%(software_commit)s' + +homepage = 'https://espressomd.org/wordpress' +description = """A software package for performing and analyzing scientific Molecular Dynamics simulations.""" + +github_account = 'espressomd' +source_urls = ['https://github.com/%(github_account)s/%(name)s/archive/'] + +sources = ['%(software_commit)s.tar.gz'] +``` + +!!! warning "`--software-commit` disables `--robot`" + + Using `--software-commit` disables the use of `--robot`, so make sure that you explicitly include + new dependencies that might need to be installed. Otherwise, the easyconfig files won't be found. + +You can also make additional changes to the easyconfig file, for example, if the new functionality requires new build or +runtime dependencies, patches, configuration options, etc. It's a good idea to try installing from a specific commit locally first, +to at least see if everything is parsed correctly and confirm that the right sources are being downloaded. + + +While the process to build for `dev.eessi.io` is similar to the one for the [production repository](../repositories/software.eessi.io.md), there +are a few additional details to keep in mind. + +#### Software version + +Installations to the EESSI production repository refer to specific versions of applications. However, development builds can't follow the same +approach as they are most often not pegged to a release. Because of this, it is possible to include a descriptive "version" label to the `version` parameter +in the easyconfig file for a given (set of) installations. + +Note that some applications are built with custom easyblocks, which may +use the `version` parameter to determine how the installation is meant to work (for example, recent versions need to copy files from to a new directory). Make sure +that you account for this, otherwise you may install software differently than intended. If you encounter issues, you can open an issue in our +[support portal](https://gitlab.com/eessi/support#eessi-support-portal). + +#### Installing dependencies + +Installations in `dev.eessi.io` are done _on top_ of `software.eessi.io`. That means if your development build depends on some application that is +already installed in `software.eessi.io`, then that will simply be used. However, if you need to add a new dependency, then this must included as +part of the build. That means including an easyconfig file for it, and adding it to the right easystack file. + +#### Using commit IDs or tags for `--software-commit` + +Installing with `--software-commit` requires that you include either a commit ID or a tag. The installation procedure will use this to +obtain the sources for the build. Because tags can be changed to point to a different commit ID, we recommend you avoid using them and sticking to +the commit ID itself. You can then include this in the `versionsuffix` on your easyconfig file, to generate a unique (though "ugly") module name. + +#### Patch files + +If your specific development build requires patch files, you should add these to the `easyconfigs/` directory. If the necessary patch is part of an +EasyBuild release, then this may not be necessary, as these will be directly taken from EasyBuild. If it is a new patch that is not on an EasyBuild +release, then include it in the `easyconfigs/` directory. + +#### Checksums + +EasyBuild's easyconfig files typically contain [checksums](https://docs.easybuild.io/writing-easyconfig-files/?h=checksums#common_easyconfig_param_sources_checksums) + as their use is highly recommended. By default, EasyBuild will compute the checksums of sources and patch files it needs for + a given installation, and compare them with the values in the easyconfig file. Because builds for `dev.eessi.io` change much + more often, hard coded checksums become a problem, as they'd need to be updated with every new build. For this reason, we + recommend not including checksums in your development easyconfig files (unless you need to, for a specific reason). + +#### Easystack files +After an easyconfig file has been created and added to the `easyconfigs` subdirectory, an [easystack file](https://docs.easybuild.io/easystack-files) that picks it up +needs to be in place so that a build can be triggered. + +!!! note "Naming convention for easystack files" + + The easystack files must follow a naming convention and be named something + like: `software-eb-X.Y.Z-dev.yml`, where X.Y.Z correspond to the EasyBuild + version used to install the software. Following our example for + `ESPREsSo`, it would look like: + + ```yaml + easyconfigs: + - ESPResSo-devel-foss-2023a-software-commit.eb: + options: + software-commit: 2ba17de6096933275abec0550981d9122e4e5f28 # release 4.2.2 + ``` + +`ESPResSo-devel-foss-2023a-software-commit.eb` would be the name of the easyconfig file added in our example step above. +Note the option passing the `software-commit` for the development version that should be built. +For the sake of this example, the chosen commit actually corresponds to the [4.2.2 release of ESPResSo](https://github.com/espressomd/espresso/releases/tag/4.2.2). + +### Triggering builds + +We use the [EESSI build-test-deploy bot](../bot.md) to handle software builds. +All one needs to do is open a PR with the changes adding the easyconfig and easystack +files and commenting `bot: build`. This can only be done by previously authorized users. +The current build cluster for `dev.eessi.io` builds only for the `zen2` CPU microarchitecture, but this is likely to change. + +Once a build is complete and the `bot:deploy` label is added, a staging PR can be merged to deploy the +application to the `dev.eessi.io` cvmfs repository. On a system with `dev.eessi.io` mounted, then all +that is left is to `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all` and try out the software! + +There is currently no initialisation script or module for `dev.eessi.io`, but this feature is coming soon. \ No newline at end of file diff --git a/docs/repositories/dev.eessi.io.md b/docs/repositories/dev.eessi.io.md new file mode 100644 index 000000000..a748cb8fa --- /dev/null +++ b/docs/repositories/dev.eessi.io.md @@ -0,0 +1,19 @@ +# Development repository (`dev.eessi.io`) + +## What is `dev.eessi.io`? + +`dev.eessi.io` is the development repository of EESSI. With it, developers can deploy pre-release builds of their software to EESSI. +This way, development versions of software can easily be tested on systems where the `dev.eessi.io` CernVM-FS repository is available. + +On a system with `dev.eessi.io` mounted access is possible with `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all`. Then, all that is left is +try out the development software! + +## Question or problems + +If you have any questions regarding EESSI, or if you experience a problem in accessing or using it, +please [open a support request](../support.md). If you experience issues with the development repository, feel free to use the #dev.eessi.io channel +of the EESSI Slack. + +## Infrastructure status + +The status of the CernVM-FS infrastructure for the production repository is shown at [https://status.eessi.io](https://status.eessi.io/). \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 88d033eb3..26e5275c4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -28,6 +28,7 @@ nav: - Production: repositories/software.eessi.io.md - RISC-V: repositories/riscv.eessi.io.md - Pilot: repositories/pilot.md + - Development repository: repositories/dev.eessi.io.md - Installation and configuration: - Is EESSI already installed?: getting_access/is_eessi_accessible.md - Native: getting_access/native_installation.md @@ -63,6 +64,7 @@ nav: - Contribution policy: adding_software/contribution_policy.md - Open PR: adding_software/opening_pr.md - Debugging: adding_software/debugging_failed_builds.md + - Development repository: adding_software/adding_development_software.md - For maintainers: - BTD bot: bot.md - Building software: adding_software/building_software.md