From 9aea2d332d9c7636a3baa02b9d0a80720c3a05cd Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 5 Aug 2024 13:31:16 +0200 Subject: [PATCH 01/26] Updated install guide Signed-off-by: christopher-hakkaart --- .../usage/Getting_started/installation.md | 175 +++++++++++------- 1 file changed, 113 insertions(+), 62 deletions(-) diff --git a/sites/docs/src/content/docs/usage/Getting_started/installation.md b/sites/docs/src/content/docs/usage/Getting_started/installation.md index 8d0804b347..6830d9c940 100644 --- a/sites/docs/src/content/docs/usage/Getting_started/installation.md +++ b/sites/docs/src/content/docs/usage/Getting_started/installation.md @@ -7,41 +7,94 @@ weight: 2 ## Nextflow -All nf-core pipelines use Nextflow, so this must be installed on the system where you launch your analysis. +Nextflow must be installed on the system where you launch an nf-core pipeline. -We recommend using a personal installation of Nextflow where possible, instead of using a system-wide installation. -This makes it easier to update and control versions. +A personal installation of Nextflow is recommended to simplify updates and version control. -We generally recommend following the official installation instructions from the Nextflow documentation. -However on this page we provide '[quick start](#quick-start-installation)' version of these instructions, as well as instructions for [installing via Conda](#bioconda-instllation), and on [Windows operating systems](#installation-on-windows). - -If in doubt, see the [official Nextflow installation documentation](https://www.nextflow.io/docs/latest/getstarted.html#installation). -Any instructions on this page are provided for convinence, and may not be up-to-date. +The [Nextflow documentation](https://www.nextflow.io/docs/latest/install.html#installation) is the most accurate and up-to-date installation guide. +However, on this page, we provide a Nextflow [quick start](#quick-start-installation) guide, instructions for [installing Nextflow via Conda](#bioconda-instllation), and installing Nextflow on [Windows operating systems](#installation-on-windows). :::note -You don't need to install the `nf-core` command line tools to run nf-core pipelines, you only need Nextflow. -However, they offer a number of helpful functions and are essential for pipeline development using nf-core components. -See the [tools page](/tools) for more information. +You don't need to install the `nf-core` command line tools to run nf-core pipelines. However, `nf-core` tools offer a number of helpful commands for users and are essential for pipeline developers. See the [tools page](/tools) for more information. ::: -### Official Nextflow installation. +### Nextflow installation The Nextflow installation docs can be found [here](https://www.nextflow.io/docs/latest/getstarted.html#installation) for the latest instructions. -### Quick-start installation +### Requirements + +Nextflow can be used on any POSIX-compatible system (Linux, macOS, etc), and on Windows through [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux). It requires Bash 3.2 (or later) and [Java 11 (or later, up to 22)](https://www.oracle.com/java/technologies/downloads/?er=221886). -Nextflow runs on most POSIX systems (Linux, macOS, etc) and can typically be installed by running these commands: +You can see which version of Java you have installed using the following command: ```bash -java -version # Check that Java v11+ is installed -curl -s https://get.nextflow.io | bash # Download Nextflow -chmod +x nextflow # Make executable -mv nextflow ~/bin/ # Add to user's $PATH +java -version ``` +If you don’t have a compatible version of Java installed in your computer, it is recommended that you install it through [SDKMAN!](https://sdkman.io/), and that you use the latest LTS version of Temurin. See [this website](https://whichjdk.com/) for more information. + +To install Java with SDKMAN: + +1. Install SDKMAN: + + ```bash + curl -s https://get.sdkman.io | bash + ``` + +2. Open a new terminal. + +3. Install Java: + + ```bash + sdk install java 17.0.10-tem + ``` + +4. Confirm that Java is installed correctly: + + ```bash + java -version + ``` + +### Nextflow + +Nextflow is distributed as a self-installing package, in order to make the installation process as simple as possible: + +1. Install Nextflow: + + ```bash + curl -s https://get.nextflow.io | bash + ``` + +This will create the `nextflow` executable in the current directory. + +2. Make Nextflow executable: + + ``` + chmod +x nextflow + ``` + +3. Move Nextflow into an executable path. For example: + + ```bash + mkdir -p $HOME/.local/bin/ + mv nextflow $HOME/.local/bin/ + ``` + + :::tip + Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. If it is not, add it by setting `export PATH="$PATH:$HOME/.local/bin"`. Alternatively, you could move Nextflow to a directory already in your `PATH`. Be aware that Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. + ::: + +4. Confirm that Nextflow is installed correctly: + + ```bash + nextflow info + ``` + ### Bioconda installation -You can also install Nextflow using [Bioconda](https://bioconda.github.io/). +Nextflow can also be installed using [Bioconda](https://bioconda.github.io/). + First, set up Bioconda according to the [Bioconda documentation](https://bioconda.github.io/#usage), notably setting up channels: ```bash @@ -70,67 +123,75 @@ conda deactivate ### Installation on Windows -For Windows the installation procedure is more complex and is fully described on the [Nextflow website](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html). +The installation procedure for Windows computers is more complex. -The main steps will be the following: +The main steps include: -- Install Windows PowerShell -- Configure the Windows Subsystem for Linux (WSL2) -- Install a Linux distribution (on WSL2) +- Installing Windows PowerShell +- Configuring the Windows Subsystem for Linux (WSL2) +- Installing a Linux distribution (on WSL2) -The step to install Nextflow in itself will afterwards be the same as previously mentioned. +See the [guide for setting up a Nextflow environment on Windows 10](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html) for more information. + +:::warning +Some information in the [guide for setting up a Nextflow environment on Windows 10](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html) may be out of date. +::: ### Updating Nextflow -Updating nextflow is as simple as running: +With Nextflow installed in your environment, you can update to the latest version using the following command: ```bash nextflow self-update ``` -or `conda update nextflow`, depending on how it was installed. - -### Specific Nextflow versions - -You can install a specific version of Nextflow by using the `NXF_VER` environment version. -Nextflow will automatically install that version at the start of the next command -(be it `nextflow self-update` or `nextflow run`). - -You can `export` this bash env variable, which is good to do in analysis scripts for full reproducibility: +Specific versions of Nextflow can be installed using the `NXF_VER` environment version. +Nextflow will automatically install that version at the start of the next command. ```bash export NXF_VER=23.10.1 nextflow run hello-world ``` -Or even just prepend it to a Nextflow command: +:::note +Setting the `NXF_VER` variable in your `.bashrc` or `.zshrc` file will fix your version of Nextflow across sessions. +::: + +You can also temporarily switch to a specific version of Nextflow with the `NXF_VER` environment variable. For example: ```bash -NXF_VER=23.10.1 nextflow run hello-world +NXF_VER=23.10.0 nextflow run hello ``` +:::note +The `conda update nextflow` command can be used to update Nextflow Bioconda installations. +::: + ### Edge releases -Nextflow has two `stable` releases per year and monthly `edge` releases. -Some pipelines may require the use of Nextflow `edge` releases in order to exploit _cutting edge_ features. +A stable version of Nextflow is released every six months, in the 4th and 10th month of each year. Additionally, an edge version is released on a monthly basis. The edge releases can be used to access the latest updates and experimental features. -Nextflow installs the latest stable version by default. -You can get an edge release either by defining the exact version with `NXF_VER` -or by using the `NXF_EDGE` environment variable: +To use the latest edge release, set `NXF_EDGE=1` when updating: ```bash -NXF_VER=24.02.0-edge nextflow run hello-world NXF_EDGE=1 nextflow self-update ``` +You can also use `NXF_VER` to temporarily switch to any edge release. For example: + +```bash +NXF_VER=24.06.0-edge nextflow info +``` + ## Pipeline software -An analysis pipeline chains the execution of multiple tools together. -Historically, all tools would have to be manually installed — often a source of great frustration and a key step where reproducibility between analyses is lost. -nf-core pipelines utilise the built-in support for software packaging that Nextflow offers: all can work with Docker and Singularity, and most pipelines also support Conda. +Analysis pipelines often chain together the execution of multiple tools. +Historically, all tools would need to be installed manually - often a source of great frustration and a major source of irreproducibility. -To use any of the below, simply run your nf-core pipeline with `-profile `. -For example, `-profile docker` or `-profile conda`. +nf-core pipelines utilise the built-in support for software packaging that Nextflow offers. +Using profiles, software dependencies can be managed through various packaging (e.g., container runtimes). +To use any of the below, simply execute your nf-core pipeline with the `-profile` option. +For example, `-profile docker` or `-profile singularity`. - [Docker](https://docs.docker.com/install/) - Typically used locally, on single-user servers, and the cloud @@ -160,18 +221,8 @@ For example, `-profile docker` or `-profile conda`. - There can be changes in low-level package dependencies over time - The software still runs in your native operating system environment and so core system functions can differ - [Mamba](https://mamba.readthedocs.io/) - - A faster reimplementation of Conda - -## Pipeline code - -### Automatic + - A faster implementation of Conda -The pipeline needs no installation - Nextflow will automatically fetch it from GitHub if `nf-core/` is specified as the pipeline name. - -This method requires an internet connection. If you're running on a system that has no internet connection, please see [Running Offline](offline.md). - -### Development - -If you would like to make changes to the pipeline, fork the GitHub repository and then clone the files. Once cloned, you can run the pipeline with `nextflow run `. - -Note that you should _only_ do this if you intend to make significant changes to the pipeline. All configuration options can be changed without editing the pipeline code. Forking the pipeline repositories means that you cannot use stable releases and you will fall behind new updates. +:::note +The respective tooling for each profile (e.g., [Docker](https://docs.docker.com/install/)) must be installed prior to execution. +::: From f90edbc296e68296f2cc5c2695e31f7bd84e8540 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 5 Aug 2024 14:32:19 +0200 Subject: [PATCH 02/26] Update installation guide page --- .../usage/Getting_started/installation.md | 31 ++++++------------- 1 file changed, 10 insertions(+), 21 deletions(-) diff --git a/sites/docs/src/content/docs/usage/Getting_started/installation.md b/sites/docs/src/content/docs/usage/Getting_started/installation.md index 6830d9c940..55f1d17d87 100644 --- a/sites/docs/src/content/docs/usage/Getting_started/installation.md +++ b/sites/docs/src/content/docs/usage/Getting_started/installation.md @@ -11,17 +11,10 @@ Nextflow must be installed on the system where you launch an nf-core pipeline. A personal installation of Nextflow is recommended to simplify updates and version control. -The [Nextflow documentation](https://www.nextflow.io/docs/latest/install.html#installation) is the most accurate and up-to-date installation guide. -However, on this page, we provide a Nextflow [quick start](#quick-start-installation) guide, instructions for [installing Nextflow via Conda](#bioconda-instllation), and installing Nextflow on [Windows operating systems](#installation-on-windows). - :::note You don't need to install the `nf-core` command line tools to run nf-core pipelines. However, `nf-core` tools offer a number of helpful commands for users and are essential for pipeline developers. See the [tools page](/tools) for more information. ::: -### Nextflow installation - -The Nextflow installation docs can be found [here](https://www.nextflow.io/docs/latest/getstarted.html#installation) for the latest instructions. - ### Requirements Nextflow can be used on any POSIX-compatible system (Linux, macOS, etc), and on Windows through [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux). It requires Bash 3.2 (or later) and [Java 11 (or later, up to 22)](https://www.oracle.com/java/technologies/downloads/?er=221886). @@ -56,7 +49,7 @@ To install Java with SDKMAN: java -version ``` -### Nextflow +### Nextflow installation Nextflow is distributed as a self-installing package, in order to make the installation process as simple as possible: @@ -81,8 +74,10 @@ This will create the `nextflow` executable in the current directory. mv nextflow $HOME/.local/bin/ ``` - :::tip - Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. If it is not, add it by setting `export PATH="$PATH:$HOME/.local/bin"`. Alternatively, you could move Nextflow to a directory already in your `PATH`. Be aware that Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. + Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. If it is not, add it by setting `export PATH="$PATH:$HOME/.local/bin"`. Setting your `PATH` variable in your `.bashrc` or `.zshrc` file will fix your version of Nextflow across sessions. Alternatively, you could move Nextflow to a directory already in your `PATH`. + + :::warning + Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. ::: 4. Confirm that Nextflow is installed correctly: @@ -91,11 +86,11 @@ This will create the `nextflow` executable in the current directory. nextflow info ``` -### Bioconda installation +#### Bioconda installation Nextflow can also be installed using [Bioconda](https://bioconda.github.io/). -First, set up Bioconda according to the [Bioconda documentation](https://bioconda.github.io/#usage), notably setting up channels: +Set up Bioconda according to the [Bioconda documentation](https://bioconda.github.io/#usage), notably setting up channels: ```bash conda config --add channels bioconda @@ -121,7 +116,7 @@ To deactivate the conda environment, run: conda deactivate ``` -### Installation on Windows +#### Windows installation The installation procedure for Windows computers is more complex. @@ -153,9 +148,7 @@ export NXF_VER=23.10.1 nextflow run hello-world ``` -:::note Setting the `NXF_VER` variable in your `.bashrc` or `.zshrc` file will fix your version of Nextflow across sessions. -::: You can also temporarily switch to a specific version of Nextflow with the `NXF_VER` environment variable. For example: @@ -191,7 +184,7 @@ Historically, all tools would need to be installed manually - often a source of nf-core pipelines utilise the built-in support for software packaging that Nextflow offers. Using profiles, software dependencies can be managed through various packaging (e.g., container runtimes). To use any of the below, simply execute your nf-core pipeline with the `-profile` option. -For example, `-profile docker` or `-profile singularity`. +For example, `-profile docker` or `-profile singularity`. The respective tooling for each profile (e.g., [Docker](https://docs.docker.com/install/)) must be installed prior to execution. - [Docker](https://docs.docker.com/install/) - Typically used locally, on single-user servers, and the cloud @@ -204,7 +197,7 @@ For example, `-profile docker` or `-profile singularity`. - [Apptainer](https://apptainer.org/) - Open source version of Singularity (split from Singularity in 2021) - - :::warning + :::warning Currently, nf-core pipelines run with `-profile apptainer` will build using docker containers instead of using pre-built singularity containers. @@ -222,7 +215,3 @@ For example, `-profile docker` or `-profile singularity`. - The software still runs in your native operating system environment and so core system functions can differ - [Mamba](https://mamba.readthedocs.io/) - A faster implementation of Conda - -:::note -The respective tooling for each profile (e.g., [Docker](https://docs.docker.com/install/)) must be installed prior to execution. -::: From 7f3adaf9c011a2121270c134f9f01a2468ffe903 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 5 Aug 2024 14:57:43 +0200 Subject: [PATCH 03/26] Update installation guide page --- .../usage/Getting_started/installation.md | 64 +++++++++---------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/sites/docs/src/content/docs/usage/Getting_started/installation.md b/sites/docs/src/content/docs/usage/Getting_started/installation.md index 55f1d17d87..f698efd02b 100644 --- a/sites/docs/src/content/docs/usage/Getting_started/installation.md +++ b/sites/docs/src/content/docs/usage/Getting_started/installation.md @@ -5,19 +5,19 @@ shortTitle: Dependency installation weight: 2 --- -## Nextflow +## Installation requirements Nextflow must be installed on the system where you launch an nf-core pipeline. +Nextflow can be used on any POSIX-compatible system (Linux, macOS, etc), and on Windows through [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux). It requires Bash 3.2 (or later) and [Java 11 (or later, up to 22)](https://www.oracle.com/java/technologies/downloads/?er=221886). + A personal installation of Nextflow is recommended to simplify updates and version control. :::note You don't need to install the `nf-core` command line tools to run nf-core pipelines. However, `nf-core` tools offer a number of helpful commands for users and are essential for pipeline developers. See the [tools page](/tools) for more information. ::: -### Requirements - -Nextflow can be used on any POSIX-compatible system (Linux, macOS, etc), and on Windows through [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux). It requires Bash 3.2 (or later) and [Java 11 (or later, up to 22)](https://www.oracle.com/java/technologies/downloads/?er=221886). +### Java You can see which version of Java you have installed using the following command: @@ -49,7 +49,7 @@ To install Java with SDKMAN: java -version ``` -### Nextflow installation +### Nextflow Nextflow is distributed as a self-installing package, in order to make the installation process as simple as possible: @@ -86,37 +86,32 @@ This will create the `nextflow` executable in the current directory. nextflow info ``` -#### Bioconda installation - -Nextflow can also be installed using [Bioconda](https://bioconda.github.io/). +Alternatively, Nextflow can be installed using [Bioconda](https://bioconda.github.io/): -Set up Bioconda according to the [Bioconda documentation](https://bioconda.github.io/#usage), notably setting up channels: +1. Add conda channels: -```bash -conda config --add channels bioconda -conda config --add channels conda-forge -``` - -:::warning -This is important - if not done, dependencies (such as Java) will be installed from -the wrong channels and things may break in strange ways. -::: + ```bash + conda config --add channels bioconda + conda config --add channels conda-forge + ``` -A best practice with conda is to use a dedicated conda environment. -This can help to prevent version conflicts and keep everything clean: + :::warning + This is important - if not done, dependencies (such as Java) will be installed from + the wrong channels and things may break in strange ways. + ::: -```bash -conda create --name env_nf nextflow -conda activate env_nf -``` +2. Create and activate a dedicated Nextflow conda environment. -To deactivate the conda environment, run: + ```bash + conda create --name env_nf nextflow + conda activate env_nf + ``` -```bash -conda deactivate -``` + :::note + To deactivate the `env_nf` conda environment, run `conda deactivate`. + ::: -#### Windows installation +## Windows installation The installation procedure for Windows computers is more complex. @@ -132,7 +127,7 @@ See the [guide for setting up a Nextflow environment on Windows 10](https://next Some information in the [guide for setting up a Nextflow environment on Windows 10](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html) may be out of date. ::: -### Updating Nextflow +## Updating Nextflow With Nextflow installed in your environment, you can update to the latest version using the following command: @@ -160,7 +155,7 @@ NXF_VER=23.10.0 nextflow run hello The `conda update nextflow` command can be used to update Nextflow Bioconda installations. ::: -### Edge releases +## Edge releases A stable version of Nextflow is released every six months, in the 4th and 10th month of each year. Additionally, an edge version is released on a monthly basis. The edge releases can be used to access the latest updates and experimental features. @@ -176,7 +171,7 @@ You can also use `NXF_VER` to temporarily switch to any edge release. For exampl NXF_VER=24.06.0-edge nextflow info ``` -## Pipeline software +## Software dependencies Analysis pipelines often chain together the execution of multiple tools. Historically, all tools would need to be installed manually - often a source of great frustration and a major source of irreproducibility. @@ -184,7 +179,8 @@ Historically, all tools would need to be installed manually - often a source of nf-core pipelines utilise the built-in support for software packaging that Nextflow offers. Using profiles, software dependencies can be managed through various packaging (e.g., container runtimes). To use any of the below, simply execute your nf-core pipeline with the `-profile` option. -For example, `-profile docker` or `-profile singularity`. The respective tooling for each profile (e.g., [Docker](https://docs.docker.com/install/)) must be installed prior to execution. +For example, `-profile docker` or `-profile singularity`. +The respective tooling for each profile (e.g., [Docker](https://docs.docker.com/install/)) must be installed prior to execution. - [Docker](https://docs.docker.com/install/) - Typically used locally, on single-user servers, and the cloud @@ -206,7 +202,7 @@ For example, `-profile docker` or `-profile singularity`. The respective tooling to the `apptainer` command. ::: -- [Podman](https://podman.io/), [Charliecloud](https://hpc.github.io/charliecloud/) and [Shifter](https://www.nersc.gov/research-and-development/user-defined-images/) +- [Podman](https://podman.io/), [Charliecloud](https://hpc.github.io/charliecloud/), and [Shifter](https://www.nersc.gov/research-and-development/user-defined-images/) - All alternatives to Docker, often used on HPC systems - [Conda](https://conda.io/) - Packaging system that manages environments instead of running analysis in containers From a56cfcfb137d8d820bd5f7378a74eec451e475b8 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 5 Aug 2024 16:16:35 +0200 Subject: [PATCH 04/26] Update install directory --- .../docs/src/content/docs/usage/Getting_started/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/usage/Getting_started/installation.md b/sites/docs/src/content/docs/usage/Getting_started/installation.md index f698efd02b..48932c6618 100644 --- a/sites/docs/src/content/docs/usage/Getting_started/installation.md +++ b/sites/docs/src/content/docs/usage/Getting_started/installation.md @@ -74,7 +74,7 @@ This will create the `nextflow` executable in the current directory. mv nextflow $HOME/.local/bin/ ``` - Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. If it is not, add it by setting `export PATH="$PATH:$HOME/.local/bin"`. Setting your `PATH` variable in your `.bashrc` or `.zshrc` file will fix your version of Nextflow across sessions. Alternatively, you could move Nextflow to a directory already in your `PATH`. + Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. If it is not, you can add it temporarily using `export PATH="$PATH:$HOME/.local/bin"`. To permanently add it to your `PATH` this command can be added to your shell configuration file, such as `~/.bashrc` or `~/.zshrc`. Alternatively, you could move the `nextflow` executable to a directory already in your `PATH`. :::warning Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. From cf6a7e857a9005bb0ed3c65428cb8debabdf8155 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Fri, 9 Aug 2024 13:25:34 +0200 Subject: [PATCH 05/26] First reshuffle usage guide --- .../terminology.md | 11 +- .../usage/Getting_started/configuration.md | 346 ------------------ .../usage/Getting_started/introduction.md | 98 ----- .../docs/usage/Getting_started/offline.md | 72 ---- .../docs/usage/configuration/introduction.md | 181 +++++++++ .../docs/usage/configuration/modifytools.md | 117 ++++++ .../usage/configuration/runningoffline.md | 73 ++++ .../usage/configuration/systemrequirements.md | 123 +++++++ .../installation.md | 92 ++--- .../docs/usage/quick_start/introduction.md | 49 +++ .../usage/quick_start/yourfirstpipeline.md | 40 ++ 11 files changed, 637 insertions(+), 565 deletions(-) rename sites/docs/src/content/docs/{usage/Getting_started => guidelines}/terminology.md (81%) delete mode 100644 sites/docs/src/content/docs/usage/Getting_started/configuration.md delete mode 100644 sites/docs/src/content/docs/usage/Getting_started/introduction.md delete mode 100644 sites/docs/src/content/docs/usage/Getting_started/offline.md create mode 100644 sites/docs/src/content/docs/usage/configuration/introduction.md create mode 100644 sites/docs/src/content/docs/usage/configuration/modifytools.md create mode 100644 sites/docs/src/content/docs/usage/configuration/runningoffline.md create mode 100644 sites/docs/src/content/docs/usage/configuration/systemrequirements.md rename sites/docs/src/content/docs/usage/{Getting_started => quick_start}/installation.md (70%) create mode 100644 sites/docs/src/content/docs/usage/quick_start/introduction.md create mode 100644 sites/docs/src/content/docs/usage/quick_start/yourfirstpipeline.md diff --git a/sites/docs/src/content/docs/usage/Getting_started/terminology.md b/sites/docs/src/content/docs/guidelines/terminology.md similarity index 81% rename from sites/docs/src/content/docs/usage/Getting_started/terminology.md rename to sites/docs/src/content/docs/guidelines/terminology.md index c5ee0b2aca..95c4d0c0d5 100644 --- a/sites/docs/src/content/docs/usage/Getting_started/terminology.md +++ b/sites/docs/src/content/docs/guidelines/terminology.md @@ -1,12 +1,11 @@ --- -title: nf-core Terminology -subtitle: Specification of the terms used in the nf-core community -shortTitle: nf-core terminology +title: Terminology +subtitle: nf-core terminology --- -The features offered by Nextflow DSL2 can be used in various ways depending on the granularity with which you would like to write pipelines. Please see the listing below for the hierarchy and associated terminology we have decided to use when referring to DSL2 components. +## Introduction -## Terminology +The features offered by Nextflow DSL2 can be used in various ways depending on the granularity with which you would like to write pipelines. Please see the listing below for the hierarchy and associated terminology nf-core uses when referring to DSL2 components. ### Domain-Specific Language (DSL) @@ -27,3 +26,5 @@ Component is our umbrella term for both modules and subworkflows, because they s ### Workflow An end-to-end pipeline where one or more inputs produce a series of outputs. This can either be implemented using a large monolithic script or by using a combination of DSL2 modules and subworkflows. nf-core pipelines can have multiple workflows, such as processing different data types for the same ultimate purpose (such as in [nf-core/viralrecon](https://github.com/nf-core/viralrecon/tree/master/workflows)) + +- Specify `--hook_url YOUR-HOOK-URL` or set the `params.hook_url` in `nextflow.config` to receive notifications from your pipeline in Microsoft Teams or Slack. Learn how to set up an incoming webhook in [MS Teams](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook?tabs=dotnet) and in [Slack](https://api.slack.com/messaging/webhooks). diff --git a/sites/docs/src/content/docs/usage/Getting_started/configuration.md b/sites/docs/src/content/docs/usage/Getting_started/configuration.md deleted file mode 100644 index 3609d24309..0000000000 --- a/sites/docs/src/content/docs/usage/Getting_started/configuration.md +++ /dev/null @@ -1,346 +0,0 @@ ---- -title: Pipeline configuration -subtitle: How configure Nextflow to work on your system -weight: 3 ---- - -# Introduction - -One of the strongest features of Nextflow is that it can run on virtually any computational infrastructure. -It has built-in support for HPC execution schedulers (such as [Slurm](https://slurm.schedmd.com/quickstart.html), [SGE](https://docs.oracle.com/cd/E19680-01/html/821-1541/ciagcgha.html#scrolltoc), and [others](https://www.nextflow.io/docs/latest/executor.html)) and cloud compute infrastructure (such as [AWS Batch](https://aws.amazon.com/batch/), [Google Cloud](https://cloud.google.com/) and [others](https://www.nextflow.io/docs/latest/executor.html)). - -For dependency management, Nextflow supports container engines such as [Docker](https://www.docker.com/), [Singularity](https://sylabs.io/), and [others](https://www.nextflow.io/docs/latest/container.html), and dependency management systems ([Conda](https://docs.conda.io/en/latest/) and [Spack](https://spack.readthedocs.io/en/latest/)) to deploy the pipelines. - -To get nf-core pipelines to run properly on your system, you must install your dependency management software of choice (see [Installation](/docs/usage/getting_started/installation)) and configure Nextflow to run your workflows. - -# Different config locations - -**You do not need to edit the pipeline code to configure nf-core pipelines.** -If you edit the pipeline defaults then you cannot update to more recent versions of the pipeline without overwriting your changes. -You also run the risk of moving away from the canonical pipeline and losing reproducibility. - -There are three main types of pipeline configuration that you can use: - -1. [Basic pipeline configuration profiles](#basic-configuration-profiles) -2. [Shared _nf-core/configs_ configuration profiles](#shared-nf-coreconfigs) -3. [Custom configuration files](#custom-configuration-files) - -## Basic configuration profiles - -Each nf-core pipeline comes with a set of "sensible defaults" for the resource requirements of each of the steps in the workflow, found in `config/base.config`. -These are always loaded and overwritten as needed by subsequent layers of configuration. - -In addition to this base config, pipelines have configuration "profiles" that can be enabled with the command line flag `-profile`. Multiple profiles can be specified in a comma-separated list (e.g. `-profile test,docker`). The order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. Alternatively, you can create your own configuration profiles and supply these to the pipeline when running. - -nf-core offers a range of basic profiles for configuration of container engines: - -- `docker` - - A generic configuration profile to be used with [Docker](http://docker.com/) - - Pulls software from quay.io -- `apptainer` - - A generic configuration profile to be used with [Singularity](https://apptainer.org/) - - Pulls software from quay.io -- `podman` - - A generic configuration profile to be used with [Podman](https://podman.io/) -- `shifter` - - A generic configuration profile to be used with [Shifter](https://www.nersc.gov/research-and-development/user-defined-images/) -- `charliecloud` - - A generic configuration profile to be used with [Charliecloud](https://hpc.github.io/charliecloud/) -- `conda` - - A generic configuration profile to be used with [Conda](https://conda.io/docs/) - - Pulls most software from [Bioconda](https://bioconda.github.io/) - -:::note -Please only use Conda as a last resort, i.e., when it's not possible to run the pipeline with Docker or Singularity. -::: - -If `-profile` is not specified, the pipeline will run locally and expect all software to be installed and available on the `PATH`. **This is not recommended**. - -Finally, each pipeline comes with a config profile called `test` and `test_full`. These are used for automated pipeline CI tests and will run the workflow with a minimal / full-size public dataset, respectively. They can also be used for performing test(s) run of nf-core pipeline on your infrastructure, before using your own data. - -## Shared nf-core/configs - -If you use a shared system with other people (such as a HPC or institutional server), it is best to use a configuration profile from [nf-core/configs](https://github.com/nf-core/configs). -These are shared config profiles loaded by all nf-core pipelines at run time. - -You may find that your system already has a shared profile available here (see [https://github.com/nf-core/configs](https://github.com/nf-core/configs)). -If not, please follow the instructions in the repository README and/or the [tutorial](https://nf-co.re/docs/usage/tutorials/step_by_step_institutional_profile) to add your cluster. - -## Custom configuration files - -If you are the only person to be running this pipeline, you can create a local config file and use this. -Nextflow looks for these files in three locations: - -1. User's home directory: `~/.nextflow/config` -2. Analysis working directory: `nextflow.config` -3. Custom path specified on the command line: `-c path/to/config` (multiple can be given) - -Configuration parameters are loaded one after another and overwrite previous values. -Hardcoded pipeline defaults are first, then the user's home directory, then the work directory, -then every `-c` file in the order supplied, and finally command line `--` options. - -:::warning -For Nextflow DSL2 nf-core pipelines - parameters defined in the parameter block in `custom.config` files **WILL NOT** override defaults in `nextflow.config`! Please use `-params-file` in `yaml` or `json` format in these cases: - -```json title="nf-params.json" -{ - "": 1, - "": "" -} -``` - -You can also generate such a JSON via each pipelines 'launch' button on the [nf-co.re website](https://nf-co.re/launch). -::: - -:::warning -When tuning your pipeline configuration resources, if you want to use the `check_max()` function in your custom config, you must copy the [function](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) in the link above to the bottom of your custom config -::: - -See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more information about configuration syntax and available parameters. - -# Running Nextflow on your system - -## Executor - -Nextflow pipelines default to running in "local" mode unless told otherwise - that is, running jobs on the same system where Nextflow is running. -Most users will need to specify an "executor", telling Nextflow how to submit jobs to a job scheduler (_e.g._ SGE, LSF, SLURM, PBS, AWS Batch _etc._). - -This can be done either within the shared configuration profiles (see [above](#shared-nf-coreconfigs)) or in custom configuration files. -For more information on how to specify an executor, please refer to the [Nextflow documentation](https://www.nextflow.io/docs/latest/executor.html#executor-page). - -## Max resources - -In addition to the executor, you may find that pipeline runs occasionally fail due to a particular step of the pipeline requesting more resources than you have on your system. - -To avoid these failures, all nf-core pipelines [check](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) pipeline-step resource requests against parameters called `--max_cpus`, `--max_memory` and `--max_time`. These should represent the maximum possible resources of a machine or node. - -These parameters only act as a _cap_, to prevent Nextflow submitting a single job requesting resources more than what is possible on your system. - -:::warning -Increasing these values from the defaults will not _increase_ the resources available to the pipeline tasks! See [Tuning workflow resources](#tuning-workflow-resources) for this. -::: - -Most pipelines will attempt to automatically restart jobs that fail due to lack of resources with double-requests, these caps keep those requests from getting out of hand and crashing the entire pipeline run. If a particular job exceeds the process-specific default resources and is retried, only resource requests (cpu, memory, or time) that have not yet reached the value set with `--max_` will be increased during the retry. - -:::warning -Setting the `--max_` parameters do not represent the total sum of resource usage of the pipeline at a given time - only a single pipeline job! -::: - -## Tuning workflow resources - -The base config of nf-core pipelines defines the default resources allocated to each different step in the workflow (e.g. in a [`base.config`](https://github.com/nf-core/rnaseq/blob/master/conf/base.config) file). - -These values are deliberately generous due to the wide variety of workloads done by different users. -As a result, you may find that the jobs are given more resources than they need and your system is not used as efficiently as possible. -At the other end of the scale, you may want to _increase_ the resources given to a specific task to make it run as fast as possible. You may wish to do this if you get a pipeline reporting a step failing with an `Command exit status` of e.g., `137`. - -Where possible we try to get tools to make use of the resources available, for example with a tool parameter like `-p ${task.cpus}`, where `${task.cpus}` is dynamically set according to what has been made specified in the pipeline configuration files. However, this is not possible with all tools, in which case we try to make a best guess. - -To tune workflow resources to better match your requirements, we can tweak these through [custom configuration files](#custom-configuration-files) or [shared nf-core/configs](#shared-nf-coreconfigs). - -By default, most process resources are specified using process _labels_, for example with the following base config: - -```groovy -process { - withLabel:process_low { - cpus = { check_max( 2 * task.attempt, 'cpus' ) } - memory = { check_max( 14.GB * task.attempt, 'memory' ) } - time = { check_max( 6.h * task.attempt, 'time' ) } - } - withLabel:process_medium { - cpus = { check_max( 6 * task.attempt, 'cpus' ) } - memory = { check_max( 42.GB * task.attempt, 'memory' ) } - time = { check_max( 8.h * task.attempt, 'time' ) } - } - withLabel:process_high { - cpus = { check_max( 12 * task.attempt, 'cpus' ) } - memory = { check_max( 84.GB * task.attempt, 'memory' ) } - time = { check_max( 10.h * task.attempt, 'time' ) } - } -} -``` - -- The [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) function applies the thresholds set in `--max_cpus`, `--max_memory` and `--max_time`. - - If you want to use `check_max()` in a **custom config** file, you must copy the function to the end of your config _outside_ of any configuration scopes! It will _not_ be inherited from `base.config`. -- The `* task.attempt` means that these values are doubled if a process is automatically retried after failing with an exit code that corresponds to a lack of resources. - -:::warning -If you want to use the `check_max()` function in your custom config, you must copy the function in the link above to the bottom of your custom config -::: - -:::warning -You don't need to copy all of the labels into your own custom config file, only overwrite the things you wish to change -::: - -If you want to give more memory to _all_ large tasks across most nf-core pipelines, would would specify in a custom config file: - -```groovy -process { - withLabel:process_high { - memory = 200.GB - } -} -``` - -You can be more specific than this by targeting a given process name instead of it's label using `withName`. You can see the process names in your console log when the pipeline is running For example: - -```groovy -process { - withName: STAR_ALIGN { - cpus = 32 - } -} -``` - -In some cases, a pipeline may use a tool multiple times in the workflow. In this case you will want to specify the whole execution 'path' of the module. - -```groovy -process { - withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { - memory = 100.GB - } -} -``` - -:::info -If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly. -::: - -See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#process-selectors) for more information. - -Once you've written the [config file](#custom-configuration-files), and you can give this to your pipeline command with `-c`. - -:::warning -Be careful with your syntax - if you set the memory to be `200` then it will get 200 _bytes_ of memory.
-Memory should be in quotes with a space or without quotes using a dot: `"200 GB"` or `200.GB`.
-See the Nextflow docs for [memory](https://www.nextflow.io/docs/latest/process.html#memory), -[cpus](https://www.nextflow.io/docs/latest/process.html#cpus) (int) and -[time](https://www.nextflow.io/docs/latest/process.html#time). -::: - -If you think that the defaults in the pipeline are way off, please the pipeline developers know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository! Then we can adjust the defaults to the benefit of all pipeline users. - -## Docker registries - -By default, the pipelines use `quay.io` as the default docker registry for Docker and Podman images. When specifying a docker container, it will pull the image from quay.io unless you specify a full URI. For example, if the process container is: - -- `biocontainers/fastqc:0.11.7--4` - -By default, the image will be pulled from quay.io, resulting in a full URI of: - -- `quay.io/biocontainers/fastqc:0.11.7--4` - -If the `docker.registry` is specified, this will be used first. E.g. if the config value `docker.registry = 'public.ecr.aws'` is added the image will be pulled from: - -- `public.ecr.aws/biocontainers/fastqc:0.11.7--4` - -Alternatively, if you specify a full URI in the container specification, you can ignore the `docker.registry` setting altogether. For example, this image will be pulled exactly as specified: - -- `docker.io/biocontainers/fastqc:v0.11.9_cv8` - -## Updating tool versions - -The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of nf-core pipelines uses one container or conda environment per process which makes it much easier to maintain and update software dependencies. - -If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the `process` name and override the Nextflow `container` or `conda` definition for that process using the `withName` declaration. - -For example, the [nf-core/viralrecon](https://nf-co.re/viralrecon) pipeline uses a tool called Pangolin that updates an internal database of COVID-19 lineages quite frequently. It doesn't make sense to re-release the nf-core/viralrecon every time a new version of Pangolin has been released. - -In this case, a user can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via `-c custom.config`. - -1. Check the default version used by the pipeline in the module file for the tool under `modules/nf-core/` directory of the pipeline. E.g. for [Pangolin](https://github.com/nf-core/viralrecon/blob/a85d5969f9025409e3618d6c280ef15ce417df65/modules/nf-core/software/pangolin/main.nf#L14-L19) -2. Find the latest version of the Biocontainer available on [Quay.io](https://quay.io/repository/biocontainers/pangolin?tag=latest&tab=tags) for Docker or [Galaxy Project](https://depot.galaxyproject.org/singularity/) for Singularity - - Note the container version tag is identical for both container systems, but must include the 'build' ID (e.g.`--pyhdfd78af_1`) -3. Create the custom config accordingly: - - - For Docker: - - ```groovy - process { - withName: PANGOLIN { - container = 'quay.io/biocontainers/pangolin:3.1.17--pyhdfd78af_1' - } - } - ``` - - - For Singularity: - - ```groovy - process { - withName: PANGOLIN { - container = 'https://depot.galaxyproject.org/singularity/pangolin:3.1.17--pyhdfd78af_1' - } - } - ``` - - - For Conda (note you must check against e.g. [bioconda](https://bioconda.github.io), and this does not contain the build tag): - - ```groovy - process { - withName: PANGOLIN { - conda = 'bioconda::pangolin=3.1.17' - } - } - ``` - -:::warning -It is important to note that updating containers comes with no warranty by the pipeline developers! If the update tool in the container has a major changes, this may break the pipeline. -::: - -:::warning -Sometimes tool developers change how tool versions are reported between updates. Updating containers may break version reporting within the pipeline and result in missing values in MultiQC version tables. -::: - -## Understanding and Modifying Tool Arguments - -In some cases you may wish to understand which tool arguments or options a pipeline uses, or even update or change these for your own analyses. - -You can sometimes find out what parameters are used in a tool in by checking the longer 'help' description of different pipeline parameters, e.g. by pressing the 'help' button next to [this parameter](https://nf-co.re/funcscan/1.0.1/parameters#annotation_bakta_mincontig) in [nf-core/funcscan](https://nf-co.re/funcscan). - -### Finding already used arguments - -However if this is not listed, there are two main places that a tool can have a tool argument specified. - -Most arguments (both mandatory or optional) are defined in the `conf/modules.conf` file in the pipeline code under the `ext.args` entry. For example, you can see the default arguments used by the `SORTMERNA` step of the nf-core/rnaseq pipeline [here](https://github.com/nf-core/rnaseq/blob/6e1e448f535ccf34d11cc691bb241cfd6e60a647/conf/modules.config#LL299). - -:::info -Arguments specified in `ext.args` are then inserted into the module itself via the `$args` variable in the module's bash code -::: - -In some cases _some_ modules have mandatory information for a tool for it to be executed, and these normally equate to 'mandatory' arguments. You can see the argument is used in the pipeline itself looking in the `script` section given module code itself, as in the pipeline's GitHub repository under `modules///main.nf`. - -### Customising tool arguments - -If you want to modify which parameters are used by a given tool, you can do this by specifying them in the the `ext.args` entry of a process in a [custom config](#custom-configuration-files). - -For example, lets say a pipeline does not yet support the `-n` parameter to a `BOWTIE_BUILD` step. You can add this in a custom config file like so: - -```groovy -process { - withName: BOWTIE2_ALIGN { - ext.args = "-n 0.1" - } -} -``` - -In some cases, a pipeline may use a tool multiple times in the workflow. In this case you will want to specify the whole execution 'path' of the module. - -```groovy -process { - withName: 'NFCORE_TAXPROFILER:TAXPROFILER:HOST_REMOVAL:BOWTIE2_BUILD' { - ext.args = "-n 0.1" - } -} -``` - -:::warning -It is recommended to copy and paste existing parameters in a pipelines `conf/modules.config` file, to ensure the pipeline can function as expected -::: - -:::warning -It is important to note that updating tool parameters or changing `ext.args` comes with no warranty by the pipeline developers! -::: - -# Debugging - -If you have any problems configuring your profile, please see relevant sections in the [Troubleshooting documentation](/docs/usage/troubleshooting/overview) diff --git a/sites/docs/src/content/docs/usage/Getting_started/introduction.md b/sites/docs/src/content/docs/usage/Getting_started/introduction.md deleted file mode 100644 index 2b709c4b1f..0000000000 --- a/sites/docs/src/content/docs/usage/Getting_started/introduction.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: Getting started -subtitle: What is nf-core and how to run a pipeline -weight: 1 -parentWeight: 10 ---- - -## What is nf-core? - -nf-core is a community effort to collect a curated set of analysis pipelines built with [Nextflow](https://www.nextflow.io/docs/latest/index.html). - -nf-core has three target audiences: facilities, single users, and developers. -For facilities, it provides highly automated and optimised pipelines that guarantee reproducibility of results for users. -Single users benefit from portable, documented, and easy-to-use workflows. -Developers can write Nextflow pipelines with nf-core's ready-made templates and tools. - -## What is Nextflow? - -Nextflow is a _workflow manager_. -It has been developed specifically to ease the creation and execution of bioinformatics pipelines. -The benefits of creating your pipeline with Nextflow include: - -- Built-in GitHub support. -- Compatibility with virtually all computational infrastructures, including all major cluster job schedulers. -- Integrated software dependency management (Docker, Singularity, Conda). -- Portability to run your pipelines anywhere: laptop, cluster, or cloud. -- Reproducibility of analyses, independent of time and computing platform. - -Whether your pipeline is a simple BLAST execution or a complex genome annotation pipeline, you can build it with Nextflow. - -## How to run a pipeline - -Nextflow works best with an active internet connection, as it is able to fetch all pipeline requirements. See [Running offline](offline.md) if you need to run Nextflow pipelines without an internet connection. - -To run a pipeline: - -1. All software dependencies must be installed (Nextflow + Docker / Singularity / Conda). See [Dependency installation](installation.md) for more information. - - - Run Nextflow's "hello world" example to confirm all tools are installed and operational: - - ```bash - nextflow run hello - ``` - -2. Choose a pipeline to run. See the available [nf-core pipelines](https://nf-co.re/pipelines). If you have [nf-core tools](https://nf-co.re/tools) installed, run `nf-core list`. - -3. Configure Nextflow to run on your system: - - - The simplest way to run is with `-profile docker` (or `singularity`). This instructs Nextflow to execute jobs locally, with Docker (or Singularity) to fulfill software dependencies. - - - Conda is also supported with `-profile conda`. However, this option is not recommended, as reproducibility of results can't be guaranteed without containerization. - - - If you are a member of one of the [listed institutions](https://github.com/nf-core/configs#documentation), use the [institutional config file](https://github.com/nf-core/configs/tree/master/conf) created for your institution. - - - See [Nextflow configuration](https://nf-co.re/docs/usage/getting_started/configuration) for advanced Nextflow configuration options. - -4. Run the tests for your pipeline in the terminal to confirm everything is working: - - ```bash - nextflow run nf-core/ -profile test,docker --outdir - ``` - - - Replace `` with the name of an nf-core pipeline. - - - If you don't have Docker installed, replace `docker` with either `singularity` or `conda`. - - - Nextflow will pull the code from the GitHub repository and fetch the software requirements automatically, so there is no need to download anything first. - - - If the pipeline fails, see [Troubleshooting](/docs/usage/troubleshooting/overview) or ask for help on the [nf-core Slack channel](https://nf-co.re/join) for your pipeline. - -5. Read the pipeline documentation to see which command-line parameters are required. This will be specific to your data type and usage. - -6. To launch the pipeline with real data, omit the `test` config profile and provide the required pipeline-specific parameters. For example, to run the `methylseq` pipeline, your command will be similar to this: - - ```bash - nextflow run nf-core/methylseq -profile docker --input 'input_data/*.fastq.gz' --outdir myproj/results --genome GRCh38 - ``` - -7. Once complete, check the pipeline execution and quality control reports (such as `multiqc_report.html` files for [MultiQC](https://multiqc.info/docs/usage/pipelines/#nextflow) reports). Each pipeline's documentation describes the outputs to expect. - -## Tips and tricks - -- Hyphens matter! Core Nextflow command-line options use one (`-`), whereas pipeline-specific parameters use two (`--`). -- Specify `--email your@email.com` to receive emails when your pipeline completes (requires Nextflow [mail and notification configuration](https://www.nextflow.io/docs/latest/mail.html#mail-configuration)). -- Specify `--hook_url YOUR-HOOK-URL` or set the `params.hook_url` in `nextflow.config` to receive notifications from your pipeline in Microsoft Teams or Slack. Learn how to set up an incoming webhook in [MS Teams](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook?tabs=dotnet) and in [Slack](https://api.slack.com/messaging/webhooks). -- Include `-r ` to specify a release version explicitly. This guarantees the same run command will give the same results in future runs. -- Use `-resume` to restart pipelines that did not complete. This uses cached results for successful tasks from the previous run, instead of executing all tasks from scratch. -- Use `nextflow log` to find the names of all previous runs in your directory, then use `nextflow run -resume ` to restart a specific run. -- Use multiple Nextflow configuration locations to your benefit. For example, use `-profile` for your cluster configuration, `~/.nextflow/config` for your personal configuration (with `params.email`, for example), and a working directory `nextflow.config` file for reproducible run-specific configuration. -- If you use Singularity, we recommend that you specify a cache directory with the [Nextflow environment variable](https://www.nextflow.io/docs/latest/config.html#environment-variables) `NXF_SINGULARITY_CACHEDIR` in your `~./bash_profile` or `~/.bashrc` during the installation. This will store all your container images in one place, instead of downloading an image each time you run a pipeline. Specify only the base directory — Nextflow handles the folders and file names for you. - -## Helper tools - -nf-core includes command-line tools to help you manage your nf-core pipelines and discover updates. -These allow you to list all available pipelines and versions, with information about the versions you're running locally. -You can also download pipelines for offline use. - -See [nf-core/tools](/docs/nf-core-tools) for more information. diff --git a/sites/docs/src/content/docs/usage/Getting_started/offline.md b/sites/docs/src/content/docs/usage/Getting_started/offline.md deleted file mode 100644 index 784ee6090b..0000000000 --- a/sites/docs/src/content/docs/usage/Getting_started/offline.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -title: Running offline -subtitle: How to use nf-core pipelines without an internet connection. -weight: 5 ---- - -# Introduction - -Nextflow supports fetching nearly everything it needs to run a pipeline over the web automatically: pipeline code, software requirements, reference genomes, and even remote data sources. - -Nextflow can run your analysis on a system that has no internet connection. -There are just a few extra steps required to get everything you need available locally. - -You will need to fetch everything you need on a system that _does_ have an internet connection (typically your personal computer). -Then, transfer these to your offline system using the methods you have available. - -You will need three things: a working version of [Nextflow](#nextflow), the [pipeline assets](#pipeline-code), and any required [reference genomes](#reference-genomes). - -## Nextflow - -First of all, you need to have Nextflow installed on your system. -We do this by installing it locally on a machine that _does_ have an internet connection, and then transferring to the offline system. - -- Start by [installing Nextflow locally](https://nextflow.io/docs/latest/getstarted.html#installation). - :::warning - do _not_ use the `-all` package, as this does not allow the use of custom plugins. - ::: -- Kick off a pipeline locally so that Nextflow fetches the required plugins. It does not need to run to completion. -- Copy the Nextflow binary and `$HOME/.nextflow` folder to your offline environment. -- In your Nextflow configuration file, specify each plugin that you downloaded, both name and version, including default plugins. This will prevent Nextflow from trying to download newer versions of plugins. -- Add the following environment variable in your `~/.bashrc` file: - ```bash title=".bashrc" - export NXF_OFFLINE='true' - ``` - -## Pipeline code - -To run a pipeline offline, you need the pipeline code, the software dependencies, and the shared nf-core/configs configuration profiles. -We have created a helper tool as part of the _nf-core_ package to automate this for you. - -On a computer with an internet connection, run `nf-core download ` to download the pipeline and config profiles. -Add the argument `--container singularity` to also fetch the singularity container(s). - -The pipeline and requirements will be downloaded, configured with their relative paths, and packaged into a `.tar.gz` file by default. -This can then be transferred to your offline system and unpacked. - -Inside, you will see directories called `workflow` (the pipeline files), `config` (a copy of [nf-core/configs](https://github.com/nf-core/configs)), and (if you used `--container singularity`) a directory called `singularity`. -The pipeline code is adjusted by the download tool to expect these relative paths, so as long as you keep them together it should work as is. - -To run the pipeline, simply use `nextflow run /workflow [pipeline flags]`. - -### Shared storage - -If you are downloading _directly_ to the offline storage (eg. a head node with internet access whilst compute nodes are offline), you can use the `--singularity-cache-only` option for `nf-core download` and set the `$NXF_SINGULARITY_CACHEDIR` environment variable. -This downloads the singularity images to the `$NXF_SINGULARITY_CACHEDIR` folder and does not copy them into the target downloaded pipeline folder. -This reduces total disk space usage and is faster. - -For more information, see the [documentation for `nf-core download`](/docs/nf-core-tools/pipelines/download). - -## Reference genomes - -Some pipelines require reference genomes and have built-in integration with AWS-iGenomes. -If you wish to use these references, you must download and transfer them to your offline cluster. -Once transferred, follow the [reference genomes documentation](/docs/usage/reference_genomes/reference_genomes.md) to configure the base path for the references. - -## Bytesize talk - -Here is a bytesize talk explaining the necessary steps to run pipelines offline. - - - - diff --git a/sites/docs/src/content/docs/usage/configuration/introduction.md b/sites/docs/src/content/docs/usage/configuration/introduction.md new file mode 100644 index 0000000000..eae5604a6e --- /dev/null +++ b/sites/docs/src/content/docs/usage/configuration/introduction.md @@ -0,0 +1,181 @@ +--- +title: Configuration +subtitle: How configure nf-core pipelines +shortTitle: Configuration options +weight: 1 +parentWeight: 20 +--- + +## Configure nf-core pipelines + +Each nf-core pipeline comes with a set of “sensible defaults”. +While the defaults are a great place to start, you will certainly want to modify these to fit your own purposes and system requirements. + +When a pipeline is launched, Nextflow will look for config files in several locations. +As each source can contain conflicting settings, the sources are ranked to decide which settings to apply. + +nf-core pipelines may utilize any of these configuration files so it is worthwhile to understand how they fit together. + +Configuration sources are reported below and listed in order of priority: + +1. Parameters specified on the command line (`--parameter`) +2. Parameters that are provided using the `-params-file` option +3. Config file that are provided using the `-c` option +4. The config file named `nextflow.config` in the current directory +5. The config file named `nextflow.config` in the pipeline project directory +6. The config file `$HOME/.nextflow/config` +7. Values defined within the pipeline script itself (e.g., `main.nf`) + +While some of these files are already included in the nf-core pipeline repository (e.g., the `nextflow.config` file in the nf-core pipeline repository), some are automatically identified on your local system (e.g., the `nextflow.config` in the launch directory), and others are only included if they are specified using run options (e.g., `-params-file`, and `-c`). + +If you clone and manually edit an nf-core pipeline then you cannot update to more recent versions of the pipeline without overwriting your changes. +You also risk moving away from the canonical pipeline and losing reproducibility. + +:::tip +Configuration locations can be used to your benefit. For example, use `-profile` for your cluster configuration, `~/.nextflow/config` for your personal configuration, and `nextflow.config` in your working directory for run-specific configuration. +::: + +:::warning +If you clone and manually edit an nf-core pipeline then you cannot update to more recent versions of the pipeline without overwriting your changes. You also risk moving away from the canonical pipeline and losing reproducibility. +::: + +### Parameters + +Parameters are pipeline specific settings that can be used to customize the execution of a pipeline. + +At the highest level, parameters can be customized using the command line. Any parameter can be configured on the command line by prefixing the parameter name with a double dash (--): + +```bash +-- +``` + +Depending on the parameter type, you may be required to add additional information after your parameter flag. +For example, you would add string parameter after the parameter flag. + +```bash +nextflow nf-core/rnaseq --input +``` + +Every nf-core pipeline has a full list of parameters on the nf-core website. You will be shown a description and the type of the parameter when viewing these parameters. Some parameters will also have additional text to help you understand how a parameter should be used. For example, the [parameters page of the nf-core rnaseq pipeline](https://nf-co.re/rnaseq/3.14.0/parameters/). + +### Default configuration files + +All parameters have a default configuration that is defined using the `nextflow.config` file in the root of the pipeline directory. Many parameters are set to `null` or `false` by default and are only activated by a profile or config file. + +There are also several `includeConfig` statements in the `nextflow.config` file that are used to include additional config files from the `conf/` folder. Each additional `.config` file contains categorized configuration information for your pipeline execution, some of which can be optionally included as profiles: + +- `base.config` + - Included by the pipeline by default + - Generous resource allocations using labels + - Does not specify any method for software management and expects software to be available (or specified elsewhere) +- `igenomes.config` + - Included by the pipeline by default + - Default configuration to access reference files stored on AWS iGenomes +- `modules.config` + - Included by the pipeline by default + - Module-specific configuration options (both mandatory and optional) +- `test.config` + - Only included if specified as a profile + - A configuration profile to test the pipeline with a small test dataset +- `test_full.config` + - Only included if specified as a profile + - A configuration profile to test the pipeline with a full-size test dataset + +:::note +Some config files contain the definition of profiles that can be flexibly applied. For example, the `docker`, `singularity`, and `conda` profiles are defined in the `nextflow.config` file in the pipeline project directory. +::: + +Profiles are sets of configuration options that can be flexibly applied to a pipeline. +They are also commonly defined in the `nextflow.config` file in the root of the pipeline directory. + +Profiles used by nf-core pipelines can be broadly categorized into two groups: + +- Software management profiles + - Profiles for the management of software using software management tools, for example, `docker`, `singularity`, and `conda`. +- Test profiles + - Profiles to execute the pipeline with a standardized set of test data and parameters, for example, `test` and `test_full`. + +nf-core pipelines are required to define software containers and environments that can be activated using profiles. Although it is possible to run the pipelines with software installed by other methods (e.g., environment modules or manual installation), using Docker or Singularity is more sharable, convenient, and reproducible. + +### Shared configuration files + +An `includeConfig` statement in the `nextflow.config` file is also used to include custom institutional profiles that have been submitted to the [nf-core config repository](https://github.com/nf-core/configs). At run time, nf-core pipelines will fetch these configuration profiles from the [nf-core config repository](https://github.com/nf-core/configs) and make them available. + +For shared resources such as an HPC cluster, you may consider developing a shared institutional profile. + +Follow [this tutorial](https://nf-co.re/docs/usage/tutorials/step_by_step_institutional_profile) to set up your own institutional profile. + +### Custom parameter and configuration files + +Nextflow will also look for files that are external to the pipeline project directory. These files include: + +- The config file `$HOME/.nextflow/config` +- A config file named `nextflow.config` in your current directory +- Custom configuration files specified using the command line + - A parameter file that is provided using the `-params-file` option + - A config file that are provided using the `-c` option + +**Parameter files** + +Parameter files are `.json` files that can contain an unlimited number of parameters: + +```json title="nf-params.json" +{ + "": 1, + "": "", + "": true +} +``` + +You can override default parameters by creating a `.json` file and passing it as a command-line argument using the `-param-file` option. + +```bash +nextflow run nf-core/rnaseq -profile docker --input -param-file +``` + +**Configuration files** + +Configuration files are `.config` files that can contain various pipeline properties and can be passed to Nextflow using the `-c` option in your execution command: + +```bash +nextflow run nf-core/rnaseq -profile docker --input --outdir -c +``` + +Custom configuration files are the same format as the configuration file included in the pipeline directory. + +Configuration properties are organized into scopes by dot prefixing the property names with a scope identifier or grouping the properties in the same scope using the curly brackets notation. For example: + +```groovy +alpha.x = 1 +alpha.y = 'string value' +``` + +Is equivalent to: + +```groovy +alpha { + x = 1 + y = 'string value' +} +``` + +Scopes allow you to quickly configure settings required to deploy a pipeline on different infrastructure using different software management. + +A common scenario is for users to write a custom configuration file specific to running a pipeline on their infrastructure. + +:::warning +Do not use `-c ` to specify parameters as this will result in errors. Custom config files specified with `-c` must only be used for tuning process resource specifications, other infrastructural tweaks (such as output directories), or module arguments (`args`). +::: + +Multiple scopes can be included in the same `.config` file using a mix of dot prefixes and curly brackets. + +```groovy +executor.name = "sge" + +singularity { + enabled = true + autoMounts = true +} +``` + +See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#config-scopes) for a full list of scopes. diff --git a/sites/docs/src/content/docs/usage/configuration/modifytools.md b/sites/docs/src/content/docs/usage/configuration/modifytools.md new file mode 100644 index 0000000000..c99612648f --- /dev/null +++ b/sites/docs/src/content/docs/usage/configuration/modifytools.md @@ -0,0 +1,117 @@ +--- +title: Tool arguments +subtitle: Configure tool containers and arguments +shortTitle: Tool arguments +weight: 3 +--- + +## Modifying tools + +Each tool in an nf-core pipeline come preconfigured with a set arguments for an average user. +The arguments are a great place to start and have been tested as a part of the development process. +However, you may want to modify these to fit your own purposes. + +**It is unlikely you will need to edit the pipeline code to configure a tool.** + +### Docker registries + +nf-core pipelines use `quay.io` as the default docker registry for Docker and Podman images. +When specifying a Docker container, it will pull the image from `quay.io` unless a full URI is specified. For example, if the process container is: + +- `biocontainers/fastqc:0.11.7--4` + +The image will be pulled from quay.io by default, resulting in a full URI of: + +- `quay.io/biocontainers/fastqc:0.11.7--4` + +If `docker.registry` is specified, it will be used first. For example, if the config value `docker.registry = 'public.ecr.aws'` is specified the image will be pulled from: + +- `public.ecr.aws/biocontainers/fastqc:0.11.7--4` + +However, the `docker.registry` setting will be ignored if you specify a full URI: + +- `docker.io/biocontainers/fastqc:v0.11.9_cv8` + +### Tool versions + +The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of nf-core pipelines uses one container or conda environment per process. + +If you need to use a different version of a tool within a pipeline you need to identify the `process` name and override the Nextflow `container` or `conda` definition using the `withName` declaration. + +For example, the [nf-core/viralrecon](https://nf-co.re/viralrecon) pipeline uses a tool called Pangolin that updates an internal database of COVID-19 lineages quite frequently. It doesn't make sense to re-release the `nf-core/viralrecon` pipeline every time a new version of Pangolin is released. + +A user can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via a custom configuration file. + +1. Check the default version used by the pipeline in the module file for the tool under `modules/nf-core/` directory of the pipeline. For example, for [Pangolin](https://github.com/nf-core/viralrecon/blob/a85d5969f9025409e3618d6c280ef15ce417df65/modules/nf-core/software/pangolin/main.nf#L14-L19) +2. Find the latest version of the Biocontainer available on [quay.io](https://quay.io/repository/biocontainers/pangolin?tag=latest&tab=tags) for Docker or [Galaxy Project](https://depot.galaxyproject.org/singularity/) for Singularity + - Note the container version tag is identical for both container systems, but must include the 'build' ID (e.g.`--pyhdfd78af_1`) +3. Create the custom config accordingly: + + - For Docker: + + ```groovy + process { + withName: PANGOLIN { + container = 'quay.io/biocontainers/pangolin:3.1.17--pyhdfd78af_1' + } + } + ``` + + - For Singularity: + + ```groovy + process { + withName: PANGOLIN { + container = 'https://depot.galaxyproject.org/singularity/pangolin:3.1.17--pyhdfd78af_1' + } + } + ``` + + - For Conda: + + ```groovy + process { + withName: PANGOLIN { + conda = 'bioconda::pangolin=3.1.17' + } + } + ``` + + + +:::warning +Updated tools may come with major changes and may break a pipeline. +::: + +:::warning +Updating containers may break version reporting within the pipeline and result in missing values in MultiQC version tables. +::: + +### Tool arguments + +You may wish to understand or update which tool arguments a pipeline uses. + +You can sometimes find out what parameters are used in a tool in by checking the longer 'help' description of different pipeline parameters, e.g., by pressing the 'help' button next to [this parameter](https://nf-co.re/funcscan/1.0.1/parameters#annotation_bakta_mincontig) in [nf-core/funcscan](https://nf-co.re/funcscan). + +There are two main places that a tool can have a tool argument specified: + +- The process `script` block +- The `conf/modules.conf` file + +Most arguments (both mandatory or optional) are defined in the `conf/modules.conf` file under the `ext.args` entry. Arguments that are defined in the `conf/modules.conf` file can be flexible modified using custom configuration files. + +Arguments specified in `ext.args` are then inserted into the module itself via the `$args` variable in the module's bash code + +For example, the `-n` parameter could be added to the `BOWTIE_BUILD` process: + +```groovy +process { + withName: BOWTIE_BUILD { + ext.args = "-n 0.1" + } +} +``` + +:::warning +It is recommended to copy and paste existing arguments in a pipelines `conf/modules.config` file to ensure the pipeline can function as expected. +::: diff --git a/sites/docs/src/content/docs/usage/configuration/runningoffline.md b/sites/docs/src/content/docs/usage/configuration/runningoffline.md new file mode 100644 index 0000000000..239d9b4f55 --- /dev/null +++ b/sites/docs/src/content/docs/usage/configuration/runningoffline.md @@ -0,0 +1,73 @@ +--- +title: Running offline +subtitle: Run nf-core pipelines offline +shortTitle: Running offline +weight: 4 +--- + +## Running offline + +When Nextflow is connected to the internet it will fetch nearly everything it needs to run a pipeline. Nextflow can also run analysis on an offline system that has no internet connection. However, there are a few extra steps that are required to get everything you will need locally. + +To run a pipeline offline you will need three things: + +- [Nextflow](#nextflow) +- [Pipeline assets](#pipeline-assets) +- [Reference genomes](#reference-genomes) (if required) + +These will first need to be fetched on a system that _does_ have an internet connection and transferred to your offline system. + +### Nextflow + +You need to have Nextflow installed on your local system. +You can do this by installing Nextflow on a machine that _does_ have an internet connection and transferring to the offline system: + +1. Install Nextflow locally + :::warning + do _not_ use the `-all` package, as this does not allow the use of custom plugins. + ::: +2. Run a Nextflow pipeline locally so that Nextflow fetches the required plugins. + - It does not need to run to completion. +3. Copy the Nextflow executable and your `$HOME/.nextflow` folder to your offline environment +4. Specify the name and version each plugin that you downloaded in your local Nextflow configuration file + - This will prevent Nextflow from trying to download newer versions of plugins. +5. Set `export NXF_OFFLINE='true'` + - Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently + + +### Pipeline assets + +To run a pipeline offline, you need the pipeline code, the software dependencies, and the shared nf-core/configs profiles. +We have created a helper tool as part of the _nf-core_ package to automate this for you. + +On a computer with an internet connection, run `nf-core download ` to download the pipeline and config profiles. +Add the argument `--container singularity` to also fetch the singularity container(s). + +The pipeline and requirements will be downloaded, configured with their relative paths, and packaged into a `.tar.gz` file by default. +This can then be transferred to your offline system and unpacked. + +Inside, you will see directories called `workflow` (the pipeline files), `config` (a copy of [nf-core/configs](https://github.com/nf-core/configs)), and (if you used `--container singularity`) a directory called `singularity`. +The pipeline code is adjusted by the download tool to expect these relative paths, so as long as you keep them together it should work as is. + +### Shared storage + +If you are downloading _directly_ to the offline storage (e.g., a head node with internet access whilst compute nodes are offline), you can use the `--singularity-cache-only` option for `nf-core download` and set the `$NXF_SINGULARITY_CACHEDIR` environment variable. +This downloads the singularity images to the `$NXF_SINGULARITY_CACHEDIR` folder and does not copy them into the target downloaded pipeline folder. +This reduces total disk space usage and is faster. + +See the [documentation for `nf-core download`](/docs/nf-core-tools/pipelines/download) for more information. + +### Reference genomes + +Some pipelines require reference genomes and have built-in integration with AWS-iGenomes. +If you wish to use these references, you must also download and transfer them to your offline system. + +Follow the [reference genomes documentation](/docs/usage/reference_genomes/reference_genomes.md) to configure the base path for the references. + +### Bytesize talk + +Here is a recent bytesize talk explaining the necessary steps to run pipelines offline. + + + + diff --git a/sites/docs/src/content/docs/usage/configuration/systemrequirements.md b/sites/docs/src/content/docs/usage/configuration/systemrequirements.md new file mode 100644 index 0000000000..8a7b2e0179 --- /dev/null +++ b/sites/docs/src/content/docs/usage/configuration/systemrequirements.md @@ -0,0 +1,123 @@ +--- +title: System requirements +subtitle: Configure your system requirements +shortTitle: System requirements +weight: 3 +--- + +## System requirements + +nf-core pipelines are reproducible, portable, and scalable. nf-core pipelines are designed to work "out of the box". However, it is likely you will need to configure each pipeline to suit your data and system requirements. + +The following sections are common considerations for nf-core pipeline users. + +### Executors + +Nextflow pipelines will run locally by default. +Most users will need to specify an "executor" to submit jobs to a scheduler (e.g. SGE, LSF, SLURM, PBS, and AWS Batch). + +Schedulers can be specified using [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files). +See the [Nextflow executor documentation](https://www.nextflow.io/docs/latest/executor.html#executor-page) for more information about specific schedulers. + +### Max resources + +Pipeline runs occasionally fail when a process requests more resources than you have available on your system. + +To avoid these failures, all nf-core pipelines [check](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) pipeline-step resource requests against parameters called `--max_cpus`, `--max_memory` and `--max_time`. These parameters can be set as the maximum possible resources of a machine or node and act as a _cap_ that prevents Nextflow submitting a single job requesting resources more than what is possible. + +:::warning +Increasing these values from the defaults will not _increase_ the resources available to the pipeline tasks! See [Tuning resources](#tuning-resources) for this. +::: + +Most pipelines will attempt to automatically restart jobs that fail due to lack of resources with double-requests. These `--max_` caps keep requests from getting out of hand and crashing the entire pipeline run. If a particular job exceeds the process-specific default resources and is retried, only resource requests (cpu, memory, or time) that have not yet reached the value set with `--max_` will be increased during the retry. + +:::warning +The `--max_` parameters do not represent the total sum of resource usage of the pipeline at a given time - only a single pipeline job! +::: + +### Tuning resources + +The base config of nf-core pipelines defines the default resources allocated to each different step in the workflow (e.g., in a [`base.config`](https://github.com/nf-core/rnaseq/blob/master/conf/base.config) file). + +These values are deliberately generous due to the wide variety of workloads done by different users. +As a result, you may find that the jobs are given more resources than they need and your system is not used efficiently. +At the other end of the scale, you may want to increase the resources given to a specific task to make it run faster. +You may wish to do this if you get a pipeline reporting a step failing with an `Command exit status`, such as `137`. + +Where possible tools are tuned to make use of the resources available, for example with a tool parameter (e.g., `-p ${task.cpus}`, where `${task.cpus}` is dynamically set according to what has been specified in the pipeline configuration files). However, this is not possible with all tools, in which case required resources are estimated for an average user. + +Workflow resources can modified through [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files) to better match your requirements. + +Most process resources are specified using process _labels_ by default. For example: + +```groovy +process { + withLabel:process_low { + cpus = { check_max( 2 * task.attempt, 'cpus' ) } + memory = { check_max( 14.GB * task.attempt, 'memory' ) } + time = { check_max( 6.h * task.attempt, 'time' ) } + } + withLabel:process_medium { + cpus = { check_max( 6 * task.attempt, 'cpus' ) } + memory = { check_max( 42.GB * task.attempt, 'memory' ) } + time = { check_max( 8.h * task.attempt, 'time' ) } + } + withLabel:process_high { + cpus = { check_max( 12 * task.attempt, 'cpus' ) } + memory = { check_max( 84.GB * task.attempt, 'memory' ) } + time = { check_max( 10.h * task.attempt, 'time' ) } + } +} +``` + +:::note +The [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) function applies the thresholds set in `--max_cpus`, `--max_memory` and `--max_time`. The `* task.attempt` means that these values are doubled and automatically retried after failing with an exit code that corresponds to a lack of resources. +::: + +:::warning +If you want to use the `check_max()` function in a custom configuration file, you must copy the [check_max function](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) to the bottom of your custom config +::: + +To modify the memory to all processes with the `process_high` label you would use the `withLabel` process selector. For example: + +```groovy +process { + withLabel:process_high { + memory = 200.GB + } +} +``` + +To modify the memory of a specific process with the name `STAR_ALIGN` you would use the `withName` process selector. For example: + +```groovy +process { + withName: STAR_ALIGN { + memory = 200.GB + } +} +``` + +If a pipeline uses a tool multiple times you will need to specify the whole execution path of the module. For example: + +```groovy +process { + withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { + memory = 100.GB + } +} +``` + +See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#process-selectors) for more information. + +:::warning +Be careful with your syntax - if you set the memory to be `200` then it will get 200 _bytes_ of memory.
+Memory should be in quotes with a space or without quotes using a dot: `"200 GB"` or `200.GB`.
+See the Nextflow docs for [memory](https://www.nextflow.io/docs/latest/process.html#memory), +[cpus](https://www.nextflow.io/docs/latest/process.html#cpus) (int) and +[time](https://www.nextflow.io/docs/latest/process.html#time) for more information. +::: + +:::note +If you think that the default resources for a pipeline are drastically too high or low please contact the developers know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository. +::: diff --git a/sites/docs/src/content/docs/usage/Getting_started/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md similarity index 70% rename from sites/docs/src/content/docs/usage/Getting_started/installation.md rename to sites/docs/src/content/docs/usage/quick_start/installation.md index 48932c6618..0b7645e62a 100644 --- a/sites/docs/src/content/docs/usage/Getting_started/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -1,31 +1,29 @@ --- -title: Installation of nf-core dependencies +title: Installation subtitle: Install the software requirements needed to run nf-core pipelines. -shortTitle: Dependency installation +shortTitle: Installation weight: 2 --- -## Installation requirements +## Installation Nextflow must be installed on the system where you launch an nf-core pipeline. Nextflow can be used on any POSIX-compatible system (Linux, macOS, etc), and on Windows through [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux). It requires Bash 3.2 (or later) and [Java 11 (or later, up to 22)](https://www.oracle.com/java/technologies/downloads/?er=221886). -A personal installation of Nextflow is recommended to simplify updates and version control. - :::note -You don't need to install the `nf-core` command line tools to run nf-core pipelines. However, `nf-core` tools offer a number of helpful commands for users and are essential for pipeline developers. See the [tools page](/tools) for more information. +A personal installation of Nextflow is recommended to simplify updates and version control. ::: -### Java +### Install Java -You can see which version of Java you have installed using the following command: +See which version of [Java](https://www.oracle.com/java/technologies/downloads/?er=221886) you have installed using the following command: ```bash java -version ``` -If you don’t have a compatible version of Java installed in your computer, it is recommended that you install it through [SDKMAN!](https://sdkman.io/), and that you use the latest LTS version of Temurin. See [this website](https://whichjdk.com/) for more information. +If you don’t have a compatible version of Java installed, it is recommended that you install it through [SDKMAN!](https://sdkman.io/), and that you use the latest LTS version of Temurin. See [this website](https://whichjdk.com/) for more information. To install Java with SDKMAN: @@ -49,9 +47,9 @@ To install Java with SDKMAN: java -version ``` -### Nextflow +### Install Nextflow -Nextflow is distributed as a self-installing package, in order to make the installation process as simple as possible: +Nextflow is distributed as a self-installing package. It can be installed using a few easy to follow steps: 1. Install Nextflow: @@ -74,7 +72,7 @@ This will create the `nextflow` executable in the current directory. mv nextflow $HOME/.local/bin/ ``` - Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. If it is not, you can add it temporarily using `export PATH="$PATH:$HOME/.local/bin"`. To permanently add it to your `PATH` this command can be added to your shell configuration file, such as `~/.bashrc` or `~/.zshrc`. Alternatively, you could move the `nextflow` executable to a directory already in your `PATH`. + Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. Execute `export PATH="$PATH:$HOME/.local/bin"` to add this directory to `PATH` temporarily . Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently. :::warning Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. @@ -96,8 +94,8 @@ Alternatively, Nextflow can be installed using [Bioconda](https://bioconda.githu ``` :::warning - This is important - if not done, dependencies (such as Java) will be installed from - the wrong channels and things may break in strange ways. + Conda channels are the locations where packages are stored. If the channels are not configured accurately your dependencies may be installed from + the wrong channels and be incompatible. ::: 2. Create and activate a dedicated Nextflow conda environment. @@ -108,28 +106,12 @@ Alternatively, Nextflow can be installed using [Bioconda](https://bioconda.githu ``` :::note - To deactivate the `env_nf` conda environment, run `conda deactivate`. + Run `conda deactivate` to deactivate the `env_nf` conda environment. ::: -## Windows installation - -The installation procedure for Windows computers is more complex. - -The main steps include: +### Update Nextflow -- Installing Windows PowerShell -- Configuring the Windows Subsystem for Linux (WSL2) -- Installing a Linux distribution (on WSL2) - -See the [guide for setting up a Nextflow environment on Windows 10](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html) for more information. - -:::warning -Some information in the [guide for setting up a Nextflow environment on Windows 10](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html) may be out of date. -::: - -## Updating Nextflow - -With Nextflow installed in your environment, you can update to the latest version using the following command: +To update your version of Nextflow installed in your environment, you can update to the latest version using the following command: ```bash nextflow self-update @@ -143,7 +125,9 @@ export NXF_VER=23.10.1 nextflow run hello-world ``` -Setting the `NXF_VER` variable in your `.bashrc` or `.zshrc` file will fix your version of Nextflow across sessions. +:::note +Set the `NXF_VER` variable in your `.bashrc` or `.zshrc` to fix your version of Nextflow. +::: You can also temporarily switch to a specific version of Nextflow with the `NXF_VER` environment variable. For example: @@ -151,11 +135,7 @@ You can also temporarily switch to a specific version of Nextflow with the `NXF_ NXF_VER=23.10.0 nextflow run hello ``` -:::note -The `conda update nextflow` command can be used to update Nextflow Bioconda installations. -::: - -## Edge releases +### Edge releases A stable version of Nextflow is released every six months, in the 4th and 10th month of each year. Additionally, an edge version is released on a monthly basis. The edge releases can be used to access the latest updates and experimental features. @@ -171,16 +151,24 @@ You can also use `NXF_VER` to temporarily switch to any edge release. For exampl NXF_VER=24.06.0-edge nextflow info ``` +## Install nf-core tools + +You don't need to install nf-core tools to run nf-core pipelines. However, nf-core tools offer a number of helpful commands for users and are essential for pipeline developers. + +nf-core tools includes commands tools to list, launch, configure, and download nf-core pipelines. + +See [nf-core tools](/docs/nf-core-tools) for more information. + + ## Software dependencies Analysis pipelines often chain together the execution of multiple tools. -Historically, all tools would need to be installed manually - often a source of great frustration and a major source of irreproducibility. +Historically, all tools would need to be installed manually and was often a source of great frustration and irreproducibility. nf-core pipelines utilise the built-in support for software packaging that Nextflow offers. -Using profiles, software dependencies can be managed through various packaging (e.g., container runtimes). -To use any of the below, simply execute your nf-core pipeline with the `-profile` option. -For example, `-profile docker` or `-profile singularity`. -The respective tooling for each profile (e.g., [Docker](https://docs.docker.com/install/)) must be installed prior to execution. +Using profiles, software dependencies can be managed through various packaging (e.g., Docker, Singularity, and Conda). + +The respective tooling for a profile must be installed prior to pipeline execution. Follow the links below to install the required profile tooling: - [Docker](https://docs.docker.com/install/) - Typically used locally, on single-user servers, and the cloud @@ -211,3 +199,19 @@ The respective tooling for each profile (e.g., [Docker](https://docs.docker.com/ - The software still runs in your native operating system environment and so core system functions can differ - [Mamba](https://mamba.readthedocs.io/) - A faster implementation of Conda + +## Windows installation + +The installation procedure for Windows computers is more complex. + +The main steps include: + +- Installing Windows PowerShell +- Configuring the Windows Subsystem for Linux (WSL2) +- Installing a Linux distribution (on WSL2) + +See the [guide for setting up a Nextflow environment on Windows 10](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html) for more information. + +:::warning +Some information in the [guide for setting up a Nextflow environment on Windows 10](https://nextflow.io/blog/2021/setup-nextflow-on-windows.html) may be out of date. +::: diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/usage/quick_start/introduction.md new file mode 100644 index 0000000000..029f64c625 --- /dev/null +++ b/sites/docs/src/content/docs/usage/quick_start/introduction.md @@ -0,0 +1,49 @@ +--- +title: Quick start +subtitle: Get started with Nextflow and nf-core +shortTitle: Introduction +weight: 1 +parentWeight: 10 +--- + +## What is nf-core? + +nf-core is a community effort to collect a curated set of analysis pipelines built using [Nextflow](https://www.nextflow.io/docs/latest/index.html). + +nf-core provides a standardised set of best practices, guidelines, and templates for building and sharing bioinformatics pipelines. Pipelines are designed to be modular, scalable, and portable, allowing researchers to easily adapt and execute them using their own data and compute resources. + +One of the key benefits of nf-core is that it promotes open development, testing, and peer review, ensuring that the pipelines are robust, well-documented, and validated against real-world datasets. This helps to increase the reliability and reproducibility of bioinformatics analyses and ultimately enables researchers to accelerate their scientific discoveries. + +### Key Features + +- **Documentation** + - nf-core pipelines have extensive documentation covering installation, usage, and description of output files to ensure that you won't be left in the dark. +- **CI Testing** + - Every time a change is made to the pipeline code, nf-core pipelines use continuous-integration testing to ensure that nothing has broken. +- **Stable Releases** + - nf-core pipelines use GitHub releases to tag stable versions of the code and software, making pipeline runs totally reproducible. +- **Packaged software** + - Pipeline dependencies are automatically downloaded and handled using Docker, Singularity, Conda, or other software management tools. There is no need for any software installations. +- **Portable and reproducible** + - nf-core pipelines follow best practices to ensure maximum portability and reproducibility. The large community makes the pipelines exceptionally well-tested and easy to execute. +- **Cloud-ready** + - nf-core pipelines are tested on AWS after every major release. You can even browse results live on the website and use outputs for your own benchmarking. + +nf-core is published in [Nature Biotechnology: Nat Biotechnol 38, 276–278 (2020)](https://www.nature.com/articles/s41587-020-0439-x). Nature Biotechnology. An updated preprint is available at [bioRxiv](https://www.biorxiv.org/content/10.1101/2024.05.10.592912v1). + +## Pipelines + +There are currently >110 nf-core pipelines at various stages of development. + +Each released pipeline has a dedicated page that includes 6 documentation sections: + +- **Introduction**: An introduction and overview of the pipeline +- **Usage**: Descriptions of how to execute the pipeline +- **Parameters**: Grouped pipeline parameters with descriptions +- **Output**: Descriptions and examples of the expected output files +- **Results**: Example output files generated from the full test dataset +- **Releases & Statistics**: pipeline version history and statistics + +Each section should be explored to understand what the pipeline does and how it can be configured. + +Follow on to the next sections to learn how to install Java, Nextflow, and nf-core, and launch your first pipelines! diff --git a/sites/docs/src/content/docs/usage/quick_start/yourfirstpipeline.md b/sites/docs/src/content/docs/usage/quick_start/yourfirstpipeline.md new file mode 100644 index 0000000000..94c1562eee --- /dev/null +++ b/sites/docs/src/content/docs/usage/quick_start/yourfirstpipeline.md @@ -0,0 +1,40 @@ +--- +title: Your first pipeline +subtitle: Rune your first Nextflow and nf-core pipelines. +shortTitle: Your first pipeline +weight: 3 +--- + +## Your first pipeline + +Nextflow works best with an active internet connection, as it is able to fetch all pipeline requirements. See [Running offline](/docs/usage/configuration/runningoffline) if you need to run Nextflow pipelines without an internet connection. + +### Hello world! + +Run Nextflow’s [Hello world!](https://github.com/nextflow-io/hello) pipeline example to confirm Nextflow is installed and that you have an active internet connection: + +```bash +nextflow run hello +``` + +If everything has been installed properly you should see the pipeline launch and run to completion. + +### nf-core pipelines + +Each nf-core pipeline is different and will have different run requirements. + +Before running a pipeline with your own data it is best to test it is working using a small dataset that is known to work. + +nf-core pipelines come packed with a `test` profile that uses small data files from the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. The `test` profile is extensively tested and should run without extensive configuration. nf-core pipelines also come packed with directives for containers and environments that can be flexibly enabled using software dependency profiles (e.g., `docker`, `singularity`, and `conda`). By using a profile for software dependencies an nf-core pipeline should run with all required tools. + +Run the `nf-core/rnaseq` pipeline using the `test` profile and a software profile (e.g., `docker`) to confirm Nextflow is installed and that you have an active internet connection: + +```bash +nextflow run nf-core/rnaseq -profile test,docker --outdir results +``` + +If everything has been installed properly you should see the pipeline launch and run to completion. + +You are now ready to start adding your own data and configure the pipeline to better suit your requirements. + +See the [Configuration](/docs/usage/configuration/introduction) section for more information about how you can customize the execution of a pipeline. From f6794d863f6e439324e59b7f3ea0a3e719a6eb4a Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Fri, 9 Aug 2024 14:28:07 +0200 Subject: [PATCH 06/26] Fix snake_case --- sites/docs/src/content/docs/guidelines/terminology.md | 2 -- .../usage/configuration/{modifytools.md => modify_tools.md} | 0 .../configuration/{runningoffline.md => running_offline.md} | 0 .../{systemrequirements.md => system_requirements.md} | 0 .../{yourfirstpipeline.md => your_first_pipeline.md} | 0 5 files changed, 2 deletions(-) rename sites/docs/src/content/docs/usage/configuration/{modifytools.md => modify_tools.md} (100%) rename sites/docs/src/content/docs/usage/configuration/{runningoffline.md => running_offline.md} (100%) rename sites/docs/src/content/docs/usage/configuration/{systemrequirements.md => system_requirements.md} (100%) rename sites/docs/src/content/docs/usage/quick_start/{yourfirstpipeline.md => your_first_pipeline.md} (100%) diff --git a/sites/docs/src/content/docs/guidelines/terminology.md b/sites/docs/src/content/docs/guidelines/terminology.md index 95c4d0c0d5..506b8de5f1 100644 --- a/sites/docs/src/content/docs/guidelines/terminology.md +++ b/sites/docs/src/content/docs/guidelines/terminology.md @@ -26,5 +26,3 @@ Component is our umbrella term for both modules and subworkflows, because they s ### Workflow An end-to-end pipeline where one or more inputs produce a series of outputs. This can either be implemented using a large monolithic script or by using a combination of DSL2 modules and subworkflows. nf-core pipelines can have multiple workflows, such as processing different data types for the same ultimate purpose (such as in [nf-core/viralrecon](https://github.com/nf-core/viralrecon/tree/master/workflows)) - -- Specify `--hook_url YOUR-HOOK-URL` or set the `params.hook_url` in `nextflow.config` to receive notifications from your pipeline in Microsoft Teams or Slack. Learn how to set up an incoming webhook in [MS Teams](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook?tabs=dotnet) and in [Slack](https://api.slack.com/messaging/webhooks). diff --git a/sites/docs/src/content/docs/usage/configuration/modifytools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md similarity index 100% rename from sites/docs/src/content/docs/usage/configuration/modifytools.md rename to sites/docs/src/content/docs/usage/configuration/modify_tools.md diff --git a/sites/docs/src/content/docs/usage/configuration/runningoffline.md b/sites/docs/src/content/docs/usage/configuration/running_offline.md similarity index 100% rename from sites/docs/src/content/docs/usage/configuration/runningoffline.md rename to sites/docs/src/content/docs/usage/configuration/running_offline.md diff --git a/sites/docs/src/content/docs/usage/configuration/systemrequirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md similarity index 100% rename from sites/docs/src/content/docs/usage/configuration/systemrequirements.md rename to sites/docs/src/content/docs/usage/configuration/system_requirements.md diff --git a/sites/docs/src/content/docs/usage/quick_start/yourfirstpipeline.md b/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md similarity index 100% rename from sites/docs/src/content/docs/usage/quick_start/yourfirstpipeline.md rename to sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md From 352db78f8c78a45340887b151de6f4d086103780 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Fri, 9 Aug 2024 16:22:44 +0200 Subject: [PATCH 07/26] Remove note box that feels intrusive --- .../src/content/docs/usage/configuration/system_requirements.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/system_requirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md index 8a7b2e0179..0913f01cfe 100644 --- a/sites/docs/src/content/docs/usage/configuration/system_requirements.md +++ b/sites/docs/src/content/docs/usage/configuration/system_requirements.md @@ -70,9 +70,7 @@ process { } ``` -:::note The [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) function applies the thresholds set in `--max_cpus`, `--max_memory` and `--max_time`. The `* task.attempt` means that these values are doubled and automatically retried after failing with an exit code that corresponds to a lack of resources. -::: :::warning If you want to use the `check_max()` function in a custom configuration file, you must copy the [check_max function](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) to the bottom of your custom config From 17a9f801649ccd173ee1b89dba72cf275ca5e31d Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Fri, 9 Aug 2024 16:23:46 +0200 Subject: [PATCH 08/26] Remove note box that feels intrusive --- .../docs/usage/configuration/system_requirements.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/system_requirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md index 0913f01cfe..ec3345c4ca 100644 --- a/sites/docs/src/content/docs/usage/configuration/system_requirements.md +++ b/sites/docs/src/content/docs/usage/configuration/system_requirements.md @@ -106,7 +106,9 @@ process { } ``` -See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#process-selectors) for more information. +If you think that the default resources for a pipeline are drastically too high or low please contact the developers know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository. + +See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#process-selectors) for more information about process selectors. :::warning Be careful with your syntax - if you set the memory to be `200` then it will get 200 _bytes_ of memory.
@@ -115,7 +117,3 @@ See the Nextflow docs for [memory](https://www.nextflow.io/docs/latest/process.h [cpus](https://www.nextflow.io/docs/latest/process.html#cpus) (int) and [time](https://www.nextflow.io/docs/latest/process.html#time) for more information. ::: - -:::note -If you think that the default resources for a pipeline are drastically too high or low please contact the developers know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository. -::: From 5b7416a9789de92cfe4f56b8b7af46cc940984ec Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Fri, 9 Aug 2024 16:25:31 +0200 Subject: [PATCH 09/26] Remove note box that feels intrusive --- .../src/content/docs/usage/configuration/modify_tools.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/modify_tools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md index c99612648f..e1ad1614f2 100644 --- a/sites/docs/src/content/docs/usage/configuration/modify_tools.md +++ b/sites/docs/src/content/docs/usage/configuration/modify_tools.md @@ -80,11 +80,7 @@ A user can override the default container used by the pipeline by creating a cus :::warning -Updated tools may come with major changes and may break a pipeline. -::: - -:::warning -Updating containers may break version reporting within the pipeline and result in missing values in MultiQC version tables. +Updated tools may come with major changes and may break a pipeline and/or create missing values in MultiQC version tables. ::: ### Tool arguments From 6f0084414bdeb9a3b3bcf868173b5520af76df76 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 12 Aug 2024 09:04:10 +0200 Subject: [PATCH 10/26] Apply suggestions --- .../docs/usage/configuration/introduction.md | 31 +++++++++---------- .../docs/usage/configuration/modify_tools.md | 2 +- .../configuration/system_requirements.md | 22 +++++++------ .../docs/usage/quick_start/installation.md | 14 ++++----- .../docs/usage/quick_start/introduction.md | 31 ++++++++++++++++++- .../usage/quick_start/your_first_pipeline.md | 18 +++++------ 6 files changed, 73 insertions(+), 45 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/introduction.md b/sites/docs/src/content/docs/usage/configuration/introduction.md index eae5604a6e..17f13b70e0 100644 --- a/sites/docs/src/content/docs/usage/configuration/introduction.md +++ b/sites/docs/src/content/docs/usage/configuration/introduction.md @@ -14,7 +14,7 @@ While the defaults are a great place to start, you will certainly want to modify When a pipeline is launched, Nextflow will look for config files in several locations. As each source can contain conflicting settings, the sources are ranked to decide which settings to apply. -nf-core pipelines may utilize any of these configuration files so it is worthwhile to understand how they fit together. +nf-core pipelines may utilize any of these configuration files. Configuration sources are reported below and listed in order of priority: @@ -28,17 +28,12 @@ Configuration sources are reported below and listed in order of priority: While some of these files are already included in the nf-core pipeline repository (e.g., the `nextflow.config` file in the nf-core pipeline repository), some are automatically identified on your local system (e.g., the `nextflow.config` in the launch directory), and others are only included if they are specified using run options (e.g., `-params-file`, and `-c`). -If you clone and manually edit an nf-core pipeline then you cannot update to more recent versions of the pipeline without overwriting your changes. -You also risk moving away from the canonical pipeline and losing reproducibility. +If you clone and manually edit an nf-core pipeline then you cannot update to more recent versions of the pipeline without overwriting your changes. You also risk moving away from the canonical pipeline and losing reproducibility. :::tip Configuration locations can be used to your benefit. For example, use `-profile` for your cluster configuration, `~/.nextflow/config` for your personal configuration, and `nextflow.config` in your working directory for run-specific configuration. ::: -:::warning -If you clone and manually edit an nf-core pipeline then you cannot update to more recent versions of the pipeline without overwriting your changes. You also risk moving away from the canonical pipeline and losing reproducibility. -::: - ### Parameters Parameters are pipeline specific settings that can be used to customize the execution of a pipeline. @@ -50,13 +45,13 @@ At the highest level, parameters can be customized using the command line. Any p ``` Depending on the parameter type, you may be required to add additional information after your parameter flag. -For example, you would add string parameter after the parameter flag. +For example, you would add string parameter after the parameter flag for the `nf-core/rnaseq` `--input` and `--output` parameters. ```bash -nextflow nf-core/rnaseq --input +nextflow nf-core/rnaseq --input --outdir ``` -Every nf-core pipeline has a full list of parameters on the nf-core website. You will be shown a description and the type of the parameter when viewing these parameters. Some parameters will also have additional text to help you understand how a parameter should be used. For example, the [parameters page of the nf-core rnaseq pipeline](https://nf-co.re/rnaseq/3.14.0/parameters/). +Every nf-core pipeline has a full list of parameters on the nf-core website. You will be shown a description and the type of the parameter when viewing these parameters. Some parameters will also have additional text to help you understand how a parameter should be used. See the [parameters page of the nf-core rnaseq pipeline](https://nf-co.re/rnaseq/3.14.0/parameters/). ### Default configuration files @@ -67,7 +62,7 @@ There are also several `includeConfig` statements in the `nextflow.config` file - `base.config` - Included by the pipeline by default - Generous resource allocations using labels - - Does not specify any method for software management and expects software to be available (or specified elsewhere) + - Does not specify any method for software dependencies and expects software to be available (or specified elsewhere) - `igenomes.config` - Included by the pipeline by default - Default configuration to access reference files stored on AWS iGenomes @@ -82,7 +77,7 @@ There are also several `includeConfig` statements in the `nextflow.config` file - A configuration profile to test the pipeline with a full-size test dataset :::note -Some config files contain the definition of profiles that can be flexibly applied. For example, the `docker`, `singularity`, and `conda` profiles are defined in the `nextflow.config` file in the pipeline project directory. +Some configuration files contain the definition of profiles that can be flexibly applied. For example, the `docker`, `singularity`, and `conda` profiles are defined in the `nextflow.config` file in the pipeline project directory. ::: Profiles are sets of configuration options that can be flexibly applied to a pipeline. @@ -91,11 +86,11 @@ They are also commonly defined in the `nextflow.config` file in the root of the Profiles used by nf-core pipelines can be broadly categorized into two groups: - Software management profiles - - Profiles for the management of software using software management tools, for example, `docker`, `singularity`, and `conda`. + - Profiles for the management of software dependencies using container or environment management tools, for example, `docker`, `singularity`, and `conda`. - Test profiles - Profiles to execute the pipeline with a standardized set of test data and parameters, for example, `test` and `test_full`. -nf-core pipelines are required to define software containers and environments that can be activated using profiles. Although it is possible to run the pipelines with software installed by other methods (e.g., environment modules or manual installation), using Docker or Singularity is more sharable, convenient, and reproducible. +nf-core pipelines are required to define software containers and environments that can be activated using profiles. Although it is possible to run the pipelines with software installed by other methods (e.g., environment modules or manual installation), using container technology is more sharable, convenient, and reproducible. ### Shared configuration files @@ -107,7 +102,7 @@ Follow [this tutorial](https://nf-co.re/docs/usage/tutorials/step_by_step_instit ### Custom parameter and configuration files -Nextflow will also look for files that are external to the pipeline project directory. These files include: +Nextflow will look for files that are external to the pipeline project directory. These files include: - The config file `$HOME/.nextflow/config` - A config file named `nextflow.config` in your current directory @@ -115,7 +110,7 @@ Nextflow will also look for files that are external to the pipeline project dire - A parameter file that is provided using the `-params-file` option - A config file that are provided using the `-c` option -**Parameter files** +**Parameter file format** Parameter files are `.json` files that can contain an unlimited number of parameters: @@ -133,7 +128,7 @@ You can override default parameters by creating a `.json` file and passing it as nextflow run nf-core/rnaseq -profile docker --input -param-file ``` -**Configuration files** +**Configuration file format** Configuration files are `.config` files that can contain various pipeline properties and can be passed to Nextflow using the `-c` option in your execution command: @@ -159,6 +154,8 @@ alpha { } ``` +### Scopes + Scopes allow you to quickly configure settings required to deploy a pipeline on different infrastructure using different software management. A common scenario is for users to write a custom configuration file specific to running a pipeline on their infrastructure. diff --git a/sites/docs/src/content/docs/usage/configuration/modify_tools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md index e1ad1614f2..a9ffe14aa0 100644 --- a/sites/docs/src/content/docs/usage/configuration/modify_tools.md +++ b/sites/docs/src/content/docs/usage/configuration/modify_tools.md @@ -11,7 +11,7 @@ Each tool in an nf-core pipeline come preconfigured with a set arguments for an The arguments are a great place to start and have been tested as a part of the development process. However, you may want to modify these to fit your own purposes. -**It is unlikely you will need to edit the pipeline code to configure a tool.** +**It is unlikely that you will need to edit the pipeline code to configure a tool.** ### Docker registries diff --git a/sites/docs/src/content/docs/usage/configuration/system_requirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md index ec3345c4ca..3472fca837 100644 --- a/sites/docs/src/content/docs/usage/configuration/system_requirements.md +++ b/sites/docs/src/content/docs/usage/configuration/system_requirements.md @@ -7,7 +7,7 @@ weight: 3 ## System requirements -nf-core pipelines are reproducible, portable, and scalable. nf-core pipelines are designed to work "out of the box". However, it is likely you will need to configure each pipeline to suit your data and system requirements. +nf-core pipelines are reproducible, portable, and scalable, and are designed to work "out of the box". However, it is likely you will need to configure each pipeline to suit your data and system requirements. The following sections are common considerations for nf-core pipeline users. @@ -16,17 +16,17 @@ The following sections are common considerations for nf-core pipeline users. Nextflow pipelines will run locally by default. Most users will need to specify an "executor" to submit jobs to a scheduler (e.g. SGE, LSF, SLURM, PBS, and AWS Batch). -Schedulers can be specified using [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files). +Schedulers can be configured using scopes settings that can be specified using [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files). See the [Nextflow executor documentation](https://www.nextflow.io/docs/latest/executor.html#executor-page) for more information about specific schedulers. ### Max resources Pipeline runs occasionally fail when a process requests more resources than you have available on your system. -To avoid these failures, all nf-core pipelines [check](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) pipeline-step resource requests against parameters called `--max_cpus`, `--max_memory` and `--max_time`. These parameters can be set as the maximum possible resources of a machine or node and act as a _cap_ that prevents Nextflow submitting a single job requesting resources more than what is possible. +To avoid these failures, all nf-core pipelines [check](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) pipeline-step resource requests against parameters called `--max_cpus`, `--max_memory` and `--max_time`. These parameters can be set as the maximum possible resources of a machine or node and act as a cap that prevents Nextflow submitting a single job requesting resources more than what is possible. :::warning -Increasing these values from the defaults will not _increase_ the resources available to the pipeline tasks! See [Tuning resources](#tuning-resources) for this. +Increasing these values from the defaults will not increase the resources available to the pipeline tasks! See [Tuning resources](#tuning-resources) for this. ::: Most pipelines will attempt to automatically restart jobs that fail due to lack of resources with double-requests. These `--max_` caps keep requests from getting out of hand and crashing the entire pipeline run. If a particular job exceeds the process-specific default resources and is retried, only resource requests (cpu, memory, or time) that have not yet reached the value set with `--max_` will be increased during the retry. @@ -42,13 +42,13 @@ The base config of nf-core pipelines defines the default resources allocated to These values are deliberately generous due to the wide variety of workloads done by different users. As a result, you may find that the jobs are given more resources than they need and your system is not used efficiently. At the other end of the scale, you may want to increase the resources given to a specific task to make it run faster. -You may wish to do this if you get a pipeline reporting a step failing with an `Command exit status`, such as `137`. +You may wish to increase resources if you get a pipeline reporting a step failing with an `Command exit status`, such as `137`. -Where possible tools are tuned to make use of the resources available, for example with a tool parameter (e.g., `-p ${task.cpus}`, where `${task.cpus}` is dynamically set according to what has been specified in the pipeline configuration files). However, this is not possible with all tools, in which case required resources are estimated for an average user. +Where possible, tools are tuned to make use of the resources available, for example with a tool parameter (e.g., `-p ${task.cpus}`, where `${task.cpus}` is dynamically set according to what has been specified in the pipeline configuration files). However, this is not possible with all tools, in which case required resources are estimated for an average user. -Workflow resources can modified through [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files) to better match your requirements. +Workflow resources can modified through [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files) to match your requirements. -Most process resources are specified using process _labels_ by default. For example: +Most process resources are specified using process labels by default. For example: ```groovy process { @@ -106,10 +106,12 @@ process { } ``` -If you think that the default resources for a pipeline are drastically too high or low please contact the developers know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository. - See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#process-selectors) for more information about process selectors. +:::note +If you think that the default resources for a pipeline are drastically too high or low please contact the developers know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository. +::: + :::warning Be careful with your syntax - if you set the memory to be `200` then it will get 200 _bytes_ of memory.
Memory should be in quotes with a space or without quotes using a dot: `"200 GB"` or `200.GB`.
diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md index 0b7645e62a..fe633f578d 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -65,7 +65,7 @@ This will create the `nextflow` executable in the current directory. chmod +x nextflow ``` -3. Move Nextflow into an executable path. For example: +3. Move `nextflow` to an executable path. For example: ```bash mkdir -p $HOME/.local/bin/ @@ -106,19 +106,19 @@ Alternatively, Nextflow can be installed using [Bioconda](https://bioconda.githu ``` :::note - Run `conda deactivate` to deactivate the `env_nf` conda environment. + To deactivate the `env_nf` conda environment you can use the `conda deactivate` command. ::: ### Update Nextflow -To update your version of Nextflow installed in your environment, you can update to the latest version using the following command: +You can update to the latest version of Nextflow using the self update command: ```bash nextflow self-update ``` -Specific versions of Nextflow can be installed using the `NXF_VER` environment version. -Nextflow will automatically install that version at the start of the next command. +Specific versions of Nextflow can be installed using the `NXF_VER` environment variable. +Nextflow will automatically install the specified version at the start of the next command. ```bash export NXF_VER=23.10.1 @@ -126,7 +126,7 @@ nextflow run hello-world ``` :::note -Set the `NXF_VER` variable in your `.bashrc` or `.zshrc` to fix your version of Nextflow. +Set the `NXF_VER` variable in your `.bashrc` or `.zshrc` to fix your version of Nextflow for all sessions. ::: You can also temporarily switch to a specific version of Nextflow with the `NXF_VER` environment variable. For example: @@ -145,7 +145,7 @@ To use the latest edge release, set `NXF_EDGE=1` when updating: NXF_EDGE=1 nextflow self-update ``` -You can also use `NXF_VER` to temporarily switch to any edge release. For example: +You can also use `NXF_VER` to temporarily switch to any specific edge release. For example: ```bash NXF_VER=24.06.0-edge nextflow info diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/usage/quick_start/introduction.md index 029f64c625..8e3b00a4d2 100644 --- a/sites/docs/src/content/docs/usage/quick_start/introduction.md +++ b/sites/docs/src/content/docs/usage/quick_start/introduction.md @@ -46,4 +46,33 @@ Each released pipeline has a dedicated page that includes 6 documentation sectio Each section should be explored to understand what the pipeline does and how it can be configured. -Follow on to the next sections to learn how to install Java, Nextflow, and nf-core, and launch your first pipelines! +### Join the nf-core community! + +The nf-core Slack is the place to get help, share expertise, and connect with other community members. + +

+ Join the nf-core Slack +

+ +We hope to see you there! + +**Important to know** + +- **Community conduct**: It’s important that this community is an open, collaborative, and welcoming environment. Read our nf-core [Code of Conduct](https://nf-co.re/code_of_conduct). +- **Channels**: The nf-core Slack has a channel for just about everything! Common topics and each pipeline has its own channel. When you join, you will automatically be signed up to some of our favorite channels. You can leave and join groups at any time. +- **Github invitations**: Post your GitHub username on the [`#github-invitations`](https://nfcore.slack.com/archives/CEB982K2T) channel and we will send you an invitation to join the nf-core GitHub organization. +- **Your Slack profile**: Don’t forget to fill in your profile (especially your GitHub username) so your accounts are linked. +- **Nextflow forum**: There is also a Nextflow Community Forum for asking Nextflow questions. Follow [this link](https://community.seqera.io/) to join. +- **Say hey!**: Introduce yourself in [`#say-hello`](https://nfcore.slack.com/archives/C04F9UNM46M) If you need inspiration, community members often share a few fun notes about their backgrounds, their goals, and some of their favorite things. + +There are also several groups that promote diversity and inclusion, e.g., `#nf-women-nb`, please join any that apply to you. + +:::note +You’ll probably end up joining lot of channels, so it can be a good idea to organize your sidebar with custom sections. +::: + +:::warning +Avoid using `@here` and `@channel` unless absolutely necessary. The `@here` command will notify all currently active channel members. The `@channel` command will send a message to all channel members, regardless of their activity status. If you are responding to a question or comment, try to respond in a thread to help keep conversations organised and together, especially if the channel is busy with other posts. +::: diff --git a/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md b/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md index 94c1562eee..5726d0e895 100644 --- a/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md +++ b/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md @@ -7,11 +7,11 @@ weight: 3 ## Your first pipeline -Nextflow works best with an active internet connection, as it is able to fetch all pipeline requirements. See [Running offline](/docs/usage/configuration/runningoffline) if you need to run Nextflow pipelines without an internet connection. +Nextflow works best with an active internet connection, as it is able to fetch all pipeline requirements. Running Nextflow without an active internet connection requires extra steps to move tooling, pipelines, and other assets offline. See [Running offline](/docs/usage/configuration/runningoffline) for more information. ### Hello world! -Run Nextflow’s [Hello world!](https://github.com/nextflow-io/hello) pipeline example to confirm Nextflow is installed and that you have an active internet connection: +Run Nextflow’s example [Hello world!](https://github.com/nextflow-io/hello) pipeline to confirm Nextflow is installed and that you have an active internet connection: ```bash nextflow run hello @@ -19,15 +19,17 @@ nextflow run hello If everything has been installed properly you should see the pipeline launch and run to completion. +Congratulations! You have just run your first Nextflow pipeline! + ### nf-core pipelines -Each nf-core pipeline is different and will have different run requirements. +Each nf-core pipeline will have different run requirements. -Before running a pipeline with your own data it is best to test it is working using a small dataset that is known to work. +It is best to test it is working using a small dataset that is known to work before running a pipeline with your own data. -nf-core pipelines come packed with a `test` profile that uses small data files from the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. The `test` profile is extensively tested and should run without extensive configuration. nf-core pipelines also come packed with directives for containers and environments that can be flexibly enabled using software dependency profiles (e.g., `docker`, `singularity`, and `conda`). By using a profile for software dependencies an nf-core pipeline should run with all required tools. +nf-core pipelines come packed with a `test` profile that uses small files from the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. The `test` profile is extensively tested and should run without extra configuration. nf-core pipelines come packed with directives for containers and environments that can be flexibly enabled using software dependency profiles (e.g., `docker`, `singularity`, and `conda`). By using a profile for software dependencies an nf-core pipeline should run with all required tools. -Run the `nf-core/rnaseq` pipeline using the `test` profile and a software profile (e.g., `docker`) to confirm Nextflow is installed and that you have an active internet connection: +Run the `nf-core/rnaseq` pipeline using the `test` profile and a software dependencies profile (e.g., `docker`) to confirm Nextflow is installed and that you have an active internet connection: ```bash nextflow run nf-core/rnaseq -profile test,docker --outdir results @@ -35,6 +37,4 @@ nextflow run nf-core/rnaseq -profile test,docker --outdir results If everything has been installed properly you should see the pipeline launch and run to completion. -You are now ready to start adding your own data and configure the pipeline to better suit your requirements. - -See the [Configuration](/docs/usage/configuration/introduction) section for more information about how you can customize the execution of a pipeline. +Congratulations! You have just run your first nf-core pipeline! You are now ready to start adding your own data and configure the pipeline to better suit your requirements. From 8eb3ac351ffb9f9b9358f36107bba79358ed272c Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 12 Aug 2024 16:32:20 +0200 Subject: [PATCH 11/26] Prettier --- .../docs/usage/configuration/introduction.md | 36 ++++---- .../docs/usage/configuration/modify_tools.md | 2 - .../usage/configuration/running_offline.md | 13 ++- .../docs/usage/quick_start/installation.md | 83 +++++++++---------- .../docs/usage/quick_start/introduction.md | 12 +-- 5 files changed, 71 insertions(+), 75 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/introduction.md b/sites/docs/src/content/docs/usage/configuration/introduction.md index 17f13b70e0..538ecba653 100644 --- a/sites/docs/src/content/docs/usage/configuration/introduction.md +++ b/sites/docs/src/content/docs/usage/configuration/introduction.md @@ -60,21 +60,21 @@ All parameters have a default configuration that is defined using the `nextflow. There are also several `includeConfig` statements in the `nextflow.config` file that are used to include additional config files from the `conf/` folder. Each additional `.config` file contains categorized configuration information for your pipeline execution, some of which can be optionally included as profiles: - `base.config` - - Included by the pipeline by default - - Generous resource allocations using labels - - Does not specify any method for software dependencies and expects software to be available (or specified elsewhere) + - Included by the pipeline by default + - Generous resource allocations using labels + - Does not specify any method for software dependencies and expects software to be available (or specified elsewhere) - `igenomes.config` - - Included by the pipeline by default - - Default configuration to access reference files stored on AWS iGenomes + - Included by the pipeline by default + - Default configuration to access reference files stored on AWS iGenomes - `modules.config` - - Included by the pipeline by default - - Module-specific configuration options (both mandatory and optional) + - Included by the pipeline by default + - Module-specific configuration options (both mandatory and optional) - `test.config` - - Only included if specified as a profile - - A configuration profile to test the pipeline with a small test dataset + - Only included if specified as a profile + - A configuration profile to test the pipeline with a small test dataset - `test_full.config` - - Only included if specified as a profile - - A configuration profile to test the pipeline with a full-size test dataset + - Only included if specified as a profile + - A configuration profile to test the pipeline with a full-size test dataset :::note Some configuration files contain the definition of profiles that can be flexibly applied. For example, the `docker`, `singularity`, and `conda` profiles are defined in the `nextflow.config` file in the pipeline project directory. @@ -86,9 +86,9 @@ They are also commonly defined in the `nextflow.config` file in the root of the Profiles used by nf-core pipelines can be broadly categorized into two groups: - Software management profiles - - Profiles for the management of software dependencies using container or environment management tools, for example, `docker`, `singularity`, and `conda`. + - Profiles for the management of software dependencies using container or environment management tools, for example, `docker`, `singularity`, and `conda`. - Test profiles - - Profiles to execute the pipeline with a standardized set of test data and parameters, for example, `test` and `test_full`. + - Profiles to execute the pipeline with a standardized set of test data and parameters, for example, `test` and `test_full`. nf-core pipelines are required to define software containers and environments that can be activated using profiles. Although it is possible to run the pipelines with software installed by other methods (e.g., environment modules or manual installation), using container technology is more sharable, convenient, and reproducible. @@ -107,8 +107,8 @@ Nextflow will look for files that are external to the pipeline project directory - The config file `$HOME/.nextflow/config` - A config file named `nextflow.config` in your current directory - Custom configuration files specified using the command line - - A parameter file that is provided using the `-params-file` option - - A config file that are provided using the `-c` option + - A parameter file that is provided using the `-params-file` option + - A config file that are provided using the `-c` option **Parameter file format** @@ -116,9 +116,9 @@ Parameter files are `.json` files that can contain an unlimited number of parame ```json title="nf-params.json" { - "": 1, - "": "", - "": true + "": 1, + "": "", + "": true } ``` diff --git a/sites/docs/src/content/docs/usage/configuration/modify_tools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md index a9ffe14aa0..530d3b144b 100644 --- a/sites/docs/src/content/docs/usage/configuration/modify_tools.md +++ b/sites/docs/src/content/docs/usage/configuration/modify_tools.md @@ -77,8 +77,6 @@ A user can override the default container used by the pipeline by creating a cus } ``` - - :::warning Updated tools may come with major changes and may break a pipeline and/or create missing values in MultiQC version tables. ::: diff --git a/sites/docs/src/content/docs/usage/configuration/running_offline.md b/sites/docs/src/content/docs/usage/configuration/running_offline.md index 239d9b4f55..1f7e625cb0 100644 --- a/sites/docs/src/content/docs/usage/configuration/running_offline.md +++ b/sites/docs/src/content/docs/usage/configuration/running_offline.md @@ -23,17 +23,16 @@ You need to have Nextflow installed on your local system. You can do this by installing Nextflow on a machine that _does_ have an internet connection and transferring to the offline system: 1. Install Nextflow locally - :::warning - do _not_ use the `-all` package, as this does not allow the use of custom plugins. - ::: + :::warning + do _not_ use the `-all` package, as this does not allow the use of custom plugins. + ::: 2. Run a Nextflow pipeline locally so that Nextflow fetches the required plugins. - - It does not need to run to completion. + - It does not need to run to completion. 3. Copy the Nextflow executable and your `$HOME/.nextflow` folder to your offline environment 4. Specify the name and version each plugin that you downloaded in your local Nextflow configuration file - - This will prevent Nextflow from trying to download newer versions of plugins. + - This will prevent Nextflow from trying to download newer versions of plugins. 5. Set `export NXF_OFFLINE='true'` - - Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently - + - Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently ### Pipeline assets diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md index fe633f578d..24d44a8a00 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -29,23 +29,23 @@ To install Java with SDKMAN: 1. Install SDKMAN: - ```bash - curl -s https://get.sdkman.io | bash - ``` +```bash +curl -s https://get.sdkman.io | bash +``` 2. Open a new terminal. 3. Install Java: - ```bash - sdk install java 17.0.10-tem - ``` + ```bash + sdk install java 17.0.10-tem + ``` 4. Confirm that Java is installed correctly: - ```bash - java -version - ``` + ```bash + java -version + ``` ### Install Nextflow @@ -53,61 +53,61 @@ Nextflow is distributed as a self-installing package. It can be installed using 1. Install Nextflow: - ```bash - curl -s https://get.nextflow.io | bash - ``` + ```bash + curl -s https://get.nextflow.io | bash + ``` This will create the `nextflow` executable in the current directory. 2. Make Nextflow executable: - ``` - chmod +x nextflow - ``` + ``` + chmod +x nextflow + ``` 3. Move `nextflow` to an executable path. For example: - ```bash - mkdir -p $HOME/.local/bin/ - mv nextflow $HOME/.local/bin/ - ``` + ```bash + mkdir -p $HOME/.local/bin/ + mv nextflow $HOME/.local/bin/ + ``` - Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. Execute `export PATH="$PATH:$HOME/.local/bin"` to add this directory to `PATH` temporarily . Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently. + Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. Execute `export PATH="$PATH:$HOME/.local/bin"` to add this directory to `PATH` temporarily . Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently. - :::warning - Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. - ::: + :::warning + Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. + ::: 4. Confirm that Nextflow is installed correctly: - ```bash - nextflow info - ``` + ```bash + nextflow info + ``` Alternatively, Nextflow can be installed using [Bioconda](https://bioconda.github.io/): 1. Add conda channels: - ```bash - conda config --add channels bioconda - conda config --add channels conda-forge - ``` + ```bash + conda config --add channels bioconda + conda config --add channels conda-forge + ``` - :::warning - Conda channels are the locations where packages are stored. If the channels are not configured accurately your dependencies may be installed from - the wrong channels and be incompatible. - ::: + :::warning + Conda channels are the locations where packages are stored. If the channels are not configured accurately your dependencies may be installed from + the wrong channels and be incompatible. + ::: 2. Create and activate a dedicated Nextflow conda environment. - ```bash - conda create --name env_nf nextflow - conda activate env_nf - ``` + ```bash + conda create --name env_nf nextflow + conda activate env_nf + ``` - :::note - To deactivate the `env_nf` conda environment you can use the `conda deactivate` command. - ::: + :::note + To deactivate the `env_nf` conda environment you can use the `conda deactivate` command. + ::: ### Update Nextflow @@ -159,7 +159,6 @@ nf-core tools includes commands tools to list, launch, configure, and download n See [nf-core tools](/docs/nf-core-tools) for more information. - ## Software dependencies Analysis pipelines often chain together the execution of multiple tools. diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/usage/quick_start/introduction.md index 8e3b00a4d2..e1b1447d6b 100644 --- a/sites/docs/src/content/docs/usage/quick_start/introduction.md +++ b/sites/docs/src/content/docs/usage/quick_start/introduction.md @@ -17,17 +17,17 @@ One of the key benefits of nf-core is that it promotes open development, testing ### Key Features - **Documentation** - - nf-core pipelines have extensive documentation covering installation, usage, and description of output files to ensure that you won't be left in the dark. + - nf-core pipelines have extensive documentation covering installation, usage, and description of output files to ensure that you won't be left in the dark. - **CI Testing** - - Every time a change is made to the pipeline code, nf-core pipelines use continuous-integration testing to ensure that nothing has broken. + - Every time a change is made to the pipeline code, nf-core pipelines use continuous-integration testing to ensure that nothing has broken. - **Stable Releases** - - nf-core pipelines use GitHub releases to tag stable versions of the code and software, making pipeline runs totally reproducible. + - nf-core pipelines use GitHub releases to tag stable versions of the code and software, making pipeline runs totally reproducible. - **Packaged software** - - Pipeline dependencies are automatically downloaded and handled using Docker, Singularity, Conda, or other software management tools. There is no need for any software installations. + - Pipeline dependencies are automatically downloaded and handled using Docker, Singularity, Conda, or other software management tools. There is no need for any software installations. - **Portable and reproducible** - - nf-core pipelines follow best practices to ensure maximum portability and reproducibility. The large community makes the pipelines exceptionally well-tested and easy to execute. + - nf-core pipelines follow best practices to ensure maximum portability and reproducibility. The large community makes the pipelines exceptionally well-tested and easy to execute. - **Cloud-ready** - - nf-core pipelines are tested on AWS after every major release. You can even browse results live on the website and use outputs for your own benchmarking. + - nf-core pipelines are tested on AWS after every major release. You can even browse results live on the website and use outputs for your own benchmarking. nf-core is published in [Nature Biotechnology: Nat Biotechnol 38, 276–278 (2020)](https://www.nature.com/articles/s41587-020-0439-x). Nature Biotechnology. An updated preprint is available at [bioRxiv](https://www.biorxiv.org/content/10.1101/2024.05.10.592912v1). From 2e2d7bcc2d4ee68a954a642ad8f398b5c2eef8a9 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 12 Aug 2024 16:37:50 +0200 Subject: [PATCH 12/26] Prettier x2 --- .../special-interest-groups/regulatory/meetings.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/sites/main-site/src/content/special-interest-groups/regulatory/meetings.md b/sites/main-site/src/content/special-interest-groups/regulatory/meetings.md index 4ee1f46be8..702ad53930 100644 --- a/sites/main-site/src/content/special-interest-groups/regulatory/meetings.md +++ b/sites/main-site/src/content/special-interest-groups/regulatory/meetings.md @@ -10,8 +10,7 @@ The Regulatory special interest group is going to Kick off on July 1st, 2024 at After the kick off, we will have regular meetups with a to-be-defined schedule that will be posted here, on LinkedIn and in the `#regulatory` Slack channel. -| Date | Speaker | Event | -| ----: | :---------------------------------------------- | ------------------------------------------------------- | -| Jul 1 | Alex Peltzer (Boehringer Ingelheim Pharma GmbH) | [Kick off meeting](/events/2024/SIG_regulatory_kickoff) | -| Sep 9 | Regulatory SIG | [Where do we stand and where do we want to go?](/events/2024/SIG_regulatory_2024-09-09.md) | - +| Date | Speaker | Event | +| ----: | :---------------------------------------------- | ------------------------------------------------------------------------------------------ | +| Jul 1 | Alex Peltzer (Boehringer Ingelheim Pharma GmbH) | [Kick off meeting](/events/2024/SIG_regulatory_kickoff) | +| Sep 9 | Regulatory SIG | [Where do we stand and where do we want to go?](/events/2024/SIG_regulatory_2024-09-09.md) | From 5891a9ab996894d409e403dccfc54c6fc7bd6f05 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Tue, 13 Aug 2024 11:36:04 +0200 Subject: [PATCH 13/26] Fix header --- sites/docs/src/content/docs/usage/quick_start/introduction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/usage/quick_start/introduction.md index e1b1447d6b..18cede1cf4 100644 --- a/sites/docs/src/content/docs/usage/quick_start/introduction.md +++ b/sites/docs/src/content/docs/usage/quick_start/introduction.md @@ -46,7 +46,7 @@ Each released pipeline has a dedicated page that includes 6 documentation sectio Each section should be explored to understand what the pipeline does and how it can be configured. -### Join the nf-core community! +## Join the nf-core community! The nf-core Slack is the place to get help, share expertise, and connect with other community members. From d80eec74fc960cef1733be984e789396a68ed1d3 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Tue, 13 Aug 2024 11:38:42 +0200 Subject: [PATCH 14/26] Small improvemnts --- .../src/content/docs/usage/configuration/running_offline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/usage/configuration/running_offline.md b/sites/docs/src/content/docs/usage/configuration/running_offline.md index 1f7e625cb0..594d5e3839 100644 --- a/sites/docs/src/content/docs/usage/configuration/running_offline.md +++ b/sites/docs/src/content/docs/usage/configuration/running_offline.md @@ -13,7 +13,7 @@ To run a pipeline offline you will need three things: - [Nextflow](#nextflow) - [Pipeline assets](#pipeline-assets) -- [Reference genomes](#reference-genomes) (if required) +- [Reference genomes](#reference-genomes) _(if required)_ These will first need to be fetched on a system that _does_ have an internet connection and transferred to your offline system. From 5c85cc9e6aaf7191ab778d19d78c3ebe552f0b2c Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Tue, 13 Aug 2024 11:50:24 +0200 Subject: [PATCH 15/26] Fix code blocks --- .../docs/usage/configuration/modify_tools.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/modify_tools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md index 530d3b144b..6e5877bbd8 100644 --- a/sites/docs/src/content/docs/usage/configuration/modify_tools.md +++ b/sites/docs/src/content/docs/usage/configuration/modify_tools.md @@ -18,19 +18,27 @@ However, you may want to modify these to fit your own purposes. nf-core pipelines use `quay.io` as the default docker registry for Docker and Podman images. When specifying a Docker container, it will pull the image from `quay.io` unless a full URI is specified. For example, if the process container is: -- `biocontainers/fastqc:0.11.7--4` +`````bash +biocontainers/fastqc:0.11.7--4 +``` The image will be pulled from quay.io by default, resulting in a full URI of: -- `quay.io/biocontainers/fastqc:0.11.7--4` +````bash +quay.io/biocontainers/fastqc:0.11.7--4 +``` If `docker.registry` is specified, it will be used first. For example, if the config value `docker.registry = 'public.ecr.aws'` is specified the image will be pulled from: -- `public.ecr.aws/biocontainers/fastqc:0.11.7--4` +````bash +public.ecr.aws/biocontainers/fastqc:0.11.7--4 +``` However, the `docker.registry` setting will be ignored if you specify a full URI: -- `docker.io/biocontainers/fastqc:v0.11.9_cv8` +````bash +docker.io/biocontainers/fastqc:v0.11.9_cv8 +``` ### Tool versions @@ -109,3 +117,4 @@ process { :::warning It is recommended to copy and paste existing arguments in a pipelines `conf/modules.config` file to ensure the pipeline can function as expected. ::: +````` From b3aace6b90a7211f640590aba9dcf07a96669949 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 3 Sep 2024 08:07:29 +0000 Subject: [PATCH 16/26] Fix rendering issue --- .../src/content/docs/usage/configuration/modify_tools.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/modify_tools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md index 6e5877bbd8..3de9a2edf3 100644 --- a/sites/docs/src/content/docs/usage/configuration/modify_tools.md +++ b/sites/docs/src/content/docs/usage/configuration/modify_tools.md @@ -18,25 +18,25 @@ However, you may want to modify these to fit your own purposes. nf-core pipelines use `quay.io` as the default docker registry for Docker and Podman images. When specifying a Docker container, it will pull the image from `quay.io` unless a full URI is specified. For example, if the process container is: -`````bash +```bash biocontainers/fastqc:0.11.7--4 ``` The image will be pulled from quay.io by default, resulting in a full URI of: -````bash +```bash quay.io/biocontainers/fastqc:0.11.7--4 ``` If `docker.registry` is specified, it will be used first. For example, if the config value `docker.registry = 'public.ecr.aws'` is specified the image will be pulled from: -````bash +```bash public.ecr.aws/biocontainers/fastqc:0.11.7--4 ``` However, the `docker.registry` setting will be ignored if you specify a full URI: -````bash +```bash docker.io/biocontainers/fastqc:v0.11.9_cv8 ``` @@ -117,4 +117,3 @@ process { :::warning It is recommended to copy and paste existing arguments in a pipelines `conf/modules.config` file to ensure the pipeline can function as expected. ::: -````` From 6316c3124db4e2cd11094389b9e07b2514c436b9 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Wed, 4 Sep 2024 09:24:24 +0200 Subject: [PATCH 17/26] Update sites/docs/src/content/docs/usage/quick_start/introduction.md Co-authored-by: James A. Fellows Yates --- sites/docs/src/content/docs/usage/quick_start/introduction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/usage/quick_start/introduction.md index 18cede1cf4..6e9f0373da 100644 --- a/sites/docs/src/content/docs/usage/quick_start/introduction.md +++ b/sites/docs/src/content/docs/usage/quick_start/introduction.md @@ -10,7 +10,7 @@ parentWeight: 10 nf-core is a community effort to collect a curated set of analysis pipelines built using [Nextflow](https://www.nextflow.io/docs/latest/index.html). -nf-core provides a standardised set of best practices, guidelines, and templates for building and sharing bioinformatics pipelines. Pipelines are designed to be modular, scalable, and portable, allowing researchers to easily adapt and execute them using their own data and compute resources. +nf-core provides a standardised set of best practices, guidelines, and templates for building and sharing bioinformatics pipelines. Pipelines are designed to be modular, scalable, efficient and portable, allowing researchers to easily adapt and execute them using their own data and compute resources. One of the key benefits of nf-core is that it promotes open development, testing, and peer review, ensuring that the pipelines are robust, well-documented, and validated against real-world datasets. This helps to increase the reliability and reproducibility of bioinformatics analyses and ultimately enables researchers to accelerate their scientific discoveries. From 1d5360126507027c2709f819aa846bcb13d203dd Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Wed, 4 Sep 2024 09:46:53 +0200 Subject: [PATCH 18/26] Apply suggestions from code review Merge suggestions from James Co-authored-by: James A. Fellows Yates --- .../content/docs/guidelines/terminology.md | 2 +- .../docs/usage/configuration/introduction.md | 8 ++--- .../docs/usage/configuration/modify_tools.md | 18 ++++++---- .../usage/configuration/running_offline.md | 12 +++---- .../configuration/system_requirements.md | 34 ++++++++++++++----- .../docs/usage/quick_start/installation.md | 32 ++++++++++++----- .../docs/usage/quick_start/introduction.md | 14 ++++---- 7 files changed, 78 insertions(+), 42 deletions(-) diff --git a/sites/docs/src/content/docs/guidelines/terminology.md b/sites/docs/src/content/docs/guidelines/terminology.md index 506b8de5f1..b82118fd17 100644 --- a/sites/docs/src/content/docs/guidelines/terminology.md +++ b/sites/docs/src/content/docs/guidelines/terminology.md @@ -5,7 +5,7 @@ subtitle: nf-core terminology ## Introduction -The features offered by Nextflow DSL2 can be used in various ways depending on the granularity with which you would like to write pipelines. Please see the listing below for the hierarchy and associated terminology nf-core uses when referring to DSL2 components. +The features offered by Nextflow [DSL2](#domain-specific-language-dsl) can be used in various ways depending on the granularity with which you would like to write pipelines. Please see the listing below for the hierarchy and associated terminology nf-core uses when referring to DSL2 components. ### Domain-Specific Language (DSL) diff --git a/sites/docs/src/content/docs/usage/configuration/introduction.md b/sites/docs/src/content/docs/usage/configuration/introduction.md index 538ecba653..ce2a654bbe 100644 --- a/sites/docs/src/content/docs/usage/configuration/introduction.md +++ b/sites/docs/src/content/docs/usage/configuration/introduction.md @@ -48,7 +48,7 @@ Depending on the parameter type, you may be required to add additional informati For example, you would add string parameter after the parameter flag for the `nf-core/rnaseq` `--input` and `--output` parameters. ```bash -nextflow nf-core/rnaseq --input --outdir +nextflow nf-core/rnaseq --input --outdir ``` Every nf-core pipeline has a full list of parameters on the nf-core website. You will be shown a description and the type of the parameter when viewing these parameters. Some parameters will also have additional text to help you understand how a parameter should be used. See the [parameters page of the nf-core rnaseq pipeline](https://nf-co.re/rnaseq/3.14.0/parameters/). @@ -57,7 +57,7 @@ Every nf-core pipeline has a full list of parameters on the nf-core website. You All parameters have a default configuration that is defined using the `nextflow.config` file in the root of the pipeline directory. Many parameters are set to `null` or `false` by default and are only activated by a profile or config file. -There are also several `includeConfig` statements in the `nextflow.config` file that are used to include additional config files from the `conf/` folder. Each additional `.config` file contains categorized configuration information for your pipeline execution, some of which can be optionally included as profiles: +nf-core pipelines also include additional config files from the `conf/` folder of a pipeline repository. Each additional `.config` file contains categorized configuration information for your pipeline execution, some of which can be optionally included as profiles: - `base.config` - Included by the pipeline by default @@ -83,7 +83,7 @@ Some configuration files contain the definition of profiles that can be flexibly Profiles are sets of configuration options that can be flexibly applied to a pipeline. They are also commonly defined in the `nextflow.config` file in the root of the pipeline directory. -Profiles used by nf-core pipelines can be broadly categorized into two groups: +Profiles that come with nf-core pipelines can be broadly categorized into two groups: - Software management profiles - Profiles for the management of software dependencies using container or environment management tools, for example, `docker`, `singularity`, and `conda`. @@ -94,7 +94,7 @@ nf-core pipelines are required to define software containers and environments th ### Shared configuration files -An `includeConfig` statement in the `nextflow.config` file is also used to include custom institutional profiles that have been submitted to the [nf-core config repository](https://github.com/nf-core/configs). At run time, nf-core pipelines will fetch these configuration profiles from the [nf-core config repository](https://github.com/nf-core/configs) and make them available. +nf-core pipelines can also load custom institutional profiles that have been submitted to the [nf-core config repository](https://github.com/nf-core/configs). At run time, nf-core pipelines will fetch these configuration profiles from the [nf-core config repository](https://github.com/nf-core/configs) and make them available. For shared resources such as an HPC cluster, you may consider developing a shared institutional profile. diff --git a/sites/docs/src/content/docs/usage/configuration/modify_tools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md index 3de9a2edf3..b3b6defe9a 100644 --- a/sites/docs/src/content/docs/usage/configuration/modify_tools.md +++ b/sites/docs/src/content/docs/usage/configuration/modify_tools.md @@ -9,9 +9,10 @@ weight: 3 Each tool in an nf-core pipeline come preconfigured with a set arguments for an average user. The arguments are a great place to start and have been tested as a part of the development process. +You normally can change the default settings using parameters using the double dash notation, e.g. `--input` or `--run_bwa`. However, you may want to modify these to fit your own purposes. -**It is unlikely that you will need to edit the pipeline code to configure a tool.** +**It is very unlikely that you will need to edit the pipeline code to configure a tool.** ### Docker registries @@ -40,15 +41,16 @@ However, the `docker.registry` setting will be ignored if you specify a full URI docker.io/biocontainers/fastqc:v0.11.9_cv8 ``` -### Tool versions +### Changing tool versions -The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of nf-core pipelines uses one container or conda environment per process. -If you need to use a different version of a tool within a pipeline you need to identify the `process` name and override the Nextflow `container` or `conda` definition using the `withName` declaration. +You can tell the pipeline to use a different container image within a config file and the `process` scope. -For example, the [nf-core/viralrecon](https://nf-co.re/viralrecon) pipeline uses a tool called Pangolin that updates an internal database of COVID-19 lineages quite frequently. It doesn't make sense to re-release the `nf-core/viralrecon` pipeline every time a new version of Pangolin is released. +You then need to identify the `process` name and override the Nextflow `container` or `conda` definition using the `withName` declaration. -A user can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via a custom configuration file. +For example, the [nf-core/viralrecon](https://nf-co.re/viralrecon) pipeline uses a tool called Pangolin that updates an internal database of COVID-19 lineages quite frequently. + +To update the container specification, you can do the following steps: 1. Check the default version used by the pipeline in the module file for the tool under `modules/nf-core/` directory of the pipeline. For example, for [Pangolin](https://github.com/nf-core/viralrecon/blob/a85d5969f9025409e3618d6c280ef15ce417df65/modules/nf-core/software/pangolin/main.nf#L14-L19) 2. Find the latest version of the Biocontainer available on [quay.io](https://quay.io/repository/biocontainers/pangolin?tag=latest&tab=tags) for Docker or [Galaxy Project](https://depot.galaxyproject.org/singularity/) for Singularity @@ -87,11 +89,13 @@ A user can override the default container used by the pipeline by creating a cus :::warning Updated tools may come with major changes and may break a pipeline and/or create missing values in MultiQC version tables. + +Such changes come with no warranty or support by the the pipeline developers! ::: ### Tool arguments -You may wish to understand or update which tool arguments a pipeline uses. +You may wish to understand which tool arguments a pipeline uses, or update to add additional arguments not supported currently by the pipeline. You can sometimes find out what parameters are used in a tool in by checking the longer 'help' description of different pipeline parameters, e.g., by pressing the 'help' button next to [this parameter](https://nf-co.re/funcscan/1.0.1/parameters#annotation_bakta_mincontig) in [nf-core/funcscan](https://nf-co.re/funcscan). diff --git a/sites/docs/src/content/docs/usage/configuration/running_offline.md b/sites/docs/src/content/docs/usage/configuration/running_offline.md index 594d5e3839..7569d99771 100644 --- a/sites/docs/src/content/docs/usage/configuration/running_offline.md +++ b/sites/docs/src/content/docs/usage/configuration/running_offline.md @@ -24,23 +24,23 @@ You can do this by installing Nextflow on a machine that _does_ have an internet 1. Install Nextflow locally :::warning - do _not_ use the `-all` package, as this does not allow the use of custom plugins. + Do _not_ use the `-all` package, as this does not allow the use of custom plugins. ::: 2. Run a Nextflow pipeline locally so that Nextflow fetches the required plugins. - It does not need to run to completion. 3. Copy the Nextflow executable and your `$HOME/.nextflow` folder to your offline environment -4. Specify the name and version each plugin that you downloaded in your local Nextflow configuration file +4. Specify the name and version each plugin that you downloaded in a local Nextflow configuration file - This will prevent Nextflow from trying to download newer versions of plugins. -5. Set `export NXF_OFFLINE='true'` - - Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently +5. Set `export NXF_OFFLINE='true'` in your terminal + - To set this permanently, add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) ### Pipeline assets -To run a pipeline offline, you need the pipeline code, the software dependencies, and the shared nf-core/configs profiles. +To run a pipeline offline, you next need the pipeline code, the software dependencies, and the shared nf-core/configs profiles. We have created a helper tool as part of the _nf-core_ package to automate this for you. On a computer with an internet connection, run `nf-core download ` to download the pipeline and config profiles. -Add the argument `--container singularity` to also fetch the singularity container(s). +Add the argument `--container singularity` to also fetch the singularity container(s). Note that only singularity is supported. The pipeline and requirements will be downloaded, configured with their relative paths, and packaged into a `.tar.gz` file by default. This can then be transferred to your offline system and unpacked. diff --git a/sites/docs/src/content/docs/usage/configuration/system_requirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md index 3472fca837..e76116a93d 100644 --- a/sites/docs/src/content/docs/usage/configuration/system_requirements.md +++ b/sites/docs/src/content/docs/usage/configuration/system_requirements.md @@ -7,16 +7,27 @@ weight: 3 ## System requirements -nf-core pipelines are reproducible, portable, and scalable, and are designed to work "out of the box". However, it is likely you will need to configure each pipeline to suit your data and system requirements. +nf-core pipelines are reproducible, portable, and scalable, and are designed to work "out of the box". However to maximise efficiency and benefit from all the advantages of Nextflow, it is likely you will need to configure each pipeline to suit your data and system requirements. The following sections are common considerations for nf-core pipeline users. +All of the following options can be can be specified within a Nextflow config file. + ### Executors Nextflow pipelines will run locally by default. Most users will need to specify an "executor" to submit jobs to a scheduler (e.g. SGE, LSF, SLURM, PBS, and AWS Batch). -Schedulers can be configured using scopes settings that can be specified using [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files). +A simple example of a config file specifying this could be: + + ```nextflow + process { + executor = 'slurm' + queue = { task.time >= '24.d' ? 'long' : 'short' } + } + ``` + +For more complex examples, see the various institutional configs on the [nf-core/configs repository](https://github.com/nf-core/configs/tree/master/conf) See the [Nextflow executor documentation](https://www.nextflow.io/docs/latest/executor.html#executor-page) for more information about specific schedulers. ### Max resources @@ -44,11 +55,15 @@ As a result, you may find that the jobs are given more resources than they need At the other end of the scale, you may want to increase the resources given to a specific task to make it run faster. You may wish to increase resources if you get a pipeline reporting a step failing with an `Command exit status`, such as `137`. -Where possible, tools are tuned to make use of the resources available, for example with a tool parameter (e.g., `-p ${task.cpus}`, where `${task.cpus}` is dynamically set according to what has been specified in the pipeline configuration files). However, this is not possible with all tools, in which case required resources are estimated for an average user. +:::warning +You should not modify the `base.config` of the pipeline! But always modify resource requests in your own custom or institutioanl config file! +::: + +Where possible, pipeline steps are tuned to make use of the resources available. +For example, if a tool allows specification of the number of CPUs or threads to use with the parameter (e.g., `-p`), nf-core pipeline modules will use this parameter when executing the tool, with the number of CPUs being derived from either the `base.config` or your own custom config file. -Workflow resources can modified through [shared configuration profiles](#shared-configuration-files) or [custom configuration files](#custom-parameter-and-configuration-files) to match your requirements. -Most process resources are specified using process labels by default. For example: +By default, process resources are inherited by a label. For example: ```groovy process { @@ -76,7 +91,7 @@ The [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f5926687 If you want to use the `check_max()` function in a custom configuration file, you must copy the [check_max function](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) to the bottom of your custom config ::: -To modify the memory to all processes with the `process_high` label you would use the `withLabel` process selector. For example: +To modify the memory to all processes with the `process_high` label you can use the `withLabel` process selector in your config file. For example: ```groovy process { @@ -86,7 +101,8 @@ process { } ``` -To modify the memory of a specific process with the name `STAR_ALIGN` you would use the `withName` process selector. For example: +You can also modify the memory of a specific process by using the process' name. +For example, for the step of the pipeline with the name `STAR_ALIGN`, you would use the `withName` process selector. For example: ```groovy process { @@ -96,7 +112,7 @@ process { } ``` -If a pipeline uses a tool multiple times you will need to specify the whole execution path of the module. For example: +If a pipeline uses a tool multiple times you may need to specify the whole 'execution path' of the module. For example: ```groovy process { @@ -109,7 +125,7 @@ process { See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html#process-selectors) for more information about process selectors. :::note -If you think that the default resources for a pipeline are drastically too high or low please contact the developers know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository. +If you think that the default resources for a pipeline are drastically too high or low please contact the developers of the given pipeline know either on Slack in the channel for the pipeline or via a GitHub issue on the pipeline repository. ::: :::warning diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md index 24d44a8a00..59526ed489 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -65,14 +65,14 @@ This will create the `nextflow` executable in the current directory. chmod +x nextflow ``` -3. Move `nextflow` to an executable path. For example: +3. Move `nextflow` to a location on your systems `$PATH`. For example: ```bash mkdir -p $HOME/.local/bin/ mv nextflow $HOME/.local/bin/ ``` - Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. Execute `export PATH="$PATH:$HOME/.local/bin"` to add this directory to `PATH` temporarily . Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently. + Ensure the directory `$HOME/.local/bin/` is included in your `PATH` variable. Execute `export PATH="$PATH:$HOME/.local/bin"` to add this directory to `PATH` temporarily. Add this command to your shell configuration file (e.g., `~/.bashrc` or `~/.zshrc`) to add this directory to `PATH` permanently. :::warning Nextflow will update its executable during the self update process and it should not be placed in a directory with restricted permissions. @@ -94,7 +94,7 @@ Alternatively, Nextflow can be installed using [Bioconda](https://bioconda.githu ``` :::warning - Conda channels are the locations where packages are stored. If the channels are not configured accurately your dependencies may be installed from + Conda channels are the locations where packages are distributed from. If the channels are not configured accurately your dependencies may be installed from the wrong channels and be incompatible. ::: @@ -117,8 +117,8 @@ You can update to the latest version of Nextflow using the self update command: nextflow self-update ``` -Specific versions of Nextflow can be installed using the `NXF_VER` environment variable. -Nextflow will automatically install the specified version at the start of the next command. +Specific versions of Nextflow can be installed for a particular terminal session using the `NXF_VER` environment variable. +Nextflow will automatically install the specified version at the start of the next command, and re-use this version until you close your terminal session. ```bash export NXF_VER=23.10.1 @@ -129,7 +129,7 @@ nextflow run hello-world Set the `NXF_VER` variable in your `.bashrc` or `.zshrc` to fix your version of Nextflow for all sessions. ::: -You can also temporarily switch to a specific version of Nextflow with the `NXF_VER` environment variable. For example: +You can also temporarily switch to a specific version of Nextflow for a specific run with the `NXF_VER` environment variable. For example: ```bash NXF_VER=23.10.0 nextflow run hello @@ -155,7 +155,19 @@ NXF_VER=24.06.0-edge nextflow info You don't need to install nf-core tools to run nf-core pipelines. However, nf-core tools offer a number of helpful commands for users and are essential for pipeline developers. -nf-core tools includes commands tools to list, launch, configure, and download nf-core pipelines. +For users: + + - list nf-core pipelines + - download nf-core pipelines + - configure pipeline runs + - launch pipeline runs + +For developers: + + - Create nf-core components from nf-core templates + - Lint nf-core components + - Manage nf-core pipeline schemas + - and more! See [nf-core tools](/docs/nf-core-tools) for more information. @@ -166,6 +178,7 @@ Historically, all tools would need to be installed manually and was often a sour nf-core pipelines utilise the built-in support for software packaging that Nextflow offers. Using profiles, software dependencies can be managed through various packaging (e.g., Docker, Singularity, and Conda). +These ensure that you have the correct tool and version of the tool that each pipeline needs to correctly execute. The respective tooling for a profile must be installed prior to pipeline execution. Follow the links below to install the required profile tooling: @@ -173,10 +186,12 @@ The respective tooling for a profile must be installed prior to pipeline executi - Typically used locally, on single-user servers, and the cloud - Analysis runs in a _container_, which behaves like an isolated operating system - Typically requires system root access, though a _"rootless mode"_ is available + - Needs system administrator for install - [Singularity](https://www.sylabs.io/) - Often used as an alternative to Docker on HPC systems - Also runs _containers_, and can optionally create these from Docker images - Does not need root access or any daemon processes + - Needs system administrator for install - [Apptainer](https://apptainer.org/) - Open source version of Singularity (split from Singularity in 2021) @@ -193,7 +208,8 @@ The respective tooling for a profile must be installed prior to pipeline executi - All alternatives to Docker, often used on HPC systems - [Conda](https://conda.io/) - Packaging system that manages environments instead of running analysis in containers - - Poorer reproducibility than Docker / Singularity + - Can be installed by any user + - Less reproducible than Docker / Singularity - There can be changes in low-level package dependencies over time - The software still runs in your native operating system environment and so core system functions can differ - [Mamba](https://mamba.readthedocs.io/) diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/usage/quick_start/introduction.md index 6e9f0373da..5105bb90df 100644 --- a/sites/docs/src/content/docs/usage/quick_start/introduction.md +++ b/sites/docs/src/content/docs/usage/quick_start/introduction.md @@ -25,28 +25,28 @@ One of the key benefits of nf-core is that it promotes open development, testing - **Packaged software** - Pipeline dependencies are automatically downloaded and handled using Docker, Singularity, Conda, or other software management tools. There is no need for any software installations. - **Portable and reproducible** - - nf-core pipelines follow best practices to ensure maximum portability and reproducibility. The large community makes the pipelines exceptionally well-tested and easy to execute. + - nf-core pipelines follow best practices to ensure maximum portability and reproducibility. You can run nf-core pipelines on most local machines, HPCs and in the cloud with minimal configuration changes. - **Cloud-ready** - nf-core pipelines are tested on AWS after every major release. You can even browse results live on the website and use outputs for your own benchmarking. -nf-core is published in [Nature Biotechnology: Nat Biotechnol 38, 276–278 (2020)](https://www.nature.com/articles/s41587-020-0439-x). Nature Biotechnology. An updated preprint is available at [bioRxiv](https://www.biorxiv.org/content/10.1101/2024.05.10.592912v1). +nf-core is published in [Nature Biotechnology: Nat Biotechnol 38, 276–278 (2020)](https://www.nature.com/articles/s41587-020-0439-x). Nature Biotechnology. A new preprint describing the latest updates to nf-core is available at [bioRxiv](https://www.biorxiv.org/content/10.1101/2024.05.10.592912v1). ## Pipelines -There are currently >110 nf-core pipelines at various stages of development. +There are currently >110 [nf-core pipelines](https://nf-co.re/pipelines) at various stages of development. Each released pipeline has a dedicated page that includes 6 documentation sections: - **Introduction**: An introduction and overview of the pipeline - **Usage**: Descriptions of how to execute the pipeline -- **Parameters**: Grouped pipeline parameters with descriptions +- **Parameters**: Descriptions of each pipeline parameter with additional help information - **Output**: Descriptions and examples of the expected output files -- **Results**: Example output files generated from the full test dataset +- **Results**: Example output files generated from a realistic and full-sized dataset - **Releases & Statistics**: pipeline version history and statistics Each section should be explored to understand what the pipeline does and how it can be configured. -## Join the nf-core community! +## Any questions? Wanna help out? Join the nf-core community! The nf-core Slack is the place to get help, share expertise, and connect with other community members. @@ -62,7 +62,7 @@ We hope to see you there! - **Community conduct**: It’s important that this community is an open, collaborative, and welcoming environment. Read our nf-core [Code of Conduct](https://nf-co.re/code_of_conduct). - **Channels**: The nf-core Slack has a channel for just about everything! Common topics and each pipeline has its own channel. When you join, you will automatically be signed up to some of our favorite channels. You can leave and join groups at any time. -- **Github invitations**: Post your GitHub username on the [`#github-invitations`](https://nfcore.slack.com/archives/CEB982K2T) channel and we will send you an invitation to join the nf-core GitHub organization. +- **Github invitations**: If you want to get involved with development of any kind (even documentation!), post your GitHub username on the [`#github-invitations`](https://nfcore.slack.com/archives/CEB982K2T) channel and we will send you an invitation to join the nf-core GitHub organization. - **Your Slack profile**: Don’t forget to fill in your profile (especially your GitHub username) so your accounts are linked. - **Nextflow forum**: There is also a Nextflow Community Forum for asking Nextflow questions. Follow [this link](https://community.seqera.io/) to join. - **Say hey!**: Introduce yourself in [`#say-hello`](https://nfcore.slack.com/archives/C04F9UNM46M) If you need inspiration, community members often share a few fun notes about their backgrounds, their goals, and some of their favorite things. From 5f6e5b84bbb22ed2184d204e164fcfd0e2aa797d Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Thu, 5 Sep 2024 10:05:57 +0200 Subject: [PATCH 19/26] Apply suggestions from code review Co-authored-by: James A. Fellows Yates --- .../docs/usage/configuration/system_requirements.md | 11 ++++++++++- .../content/docs/usage/quick_start/installation.md | 2 ++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/sites/docs/src/content/docs/usage/configuration/system_requirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md index e76116a93d..09cf42934c 100644 --- a/sites/docs/src/content/docs/usage/configuration/system_requirements.md +++ b/sites/docs/src/content/docs/usage/configuration/system_requirements.md @@ -13,6 +13,8 @@ The following sections are common considerations for nf-core pipeline users. All of the following options can be can be specified within a Nextflow config file. +All of the following options can be can be specified within a Nextflow config file. + ### Executors Nextflow pipelines will run locally by default. @@ -59,6 +61,10 @@ You may wish to increase resources if you get a pipeline reporting a step failin You should not modify the `base.config` of the pipeline! But always modify resource requests in your own custom or institutioanl config file! ::: +:::warning +You should not modify the `base.config` of the pipeline! But always modify resource requests in your own custom or institutioanl config file! +::: + Where possible, pipeline steps are tuned to make use of the resources available. For example, if a tool allows specification of the number of CPUs or threads to use with the parameter (e.g., `-p`), nf-core pipeline modules will use this parameter when executing the tool, with the number of CPUs being derived from either the `base.config` or your own custom config file. @@ -85,7 +91,8 @@ process { } ``` -The [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) function applies the thresholds set in `--max_cpus`, `--max_memory` and `--max_time`. The `* task.attempt` means that these values are doubled and automatically retried after failing with an exit code that corresponds to a lack of resources. +A default [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) function will apply the thresholds set in `--max_cpus`, `--max_memory` and `--max_time` (or `params.max_cpus` etc. in a config file). +The `* task.attempt` means that these values are doubled and automatically retried after failing with an exit code that corresponds to a lack of resources. :::warning If you want to use the `check_max()` function in a custom configuration file, you must copy the [check_max function](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) to the bottom of your custom config @@ -112,6 +119,8 @@ process { } ``` +Using `withName` allows you to optimise the resource usage of the pipeline to a very fine grained level. + If a pipeline uses a tool multiple times you may need to specify the whole 'execution path' of the module. For example: ```groovy diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md index 59526ed489..dc57daf0ab 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -192,6 +192,7 @@ The respective tooling for a profile must be installed prior to pipeline executi - Also runs _containers_, and can optionally create these from Docker images - Does not need root access or any daemon processes - Needs system administrator for install + - Needs system administrator for install - [Apptainer](https://apptainer.org/) - Open source version of Singularity (split from Singularity in 2021) @@ -209,6 +210,7 @@ The respective tooling for a profile must be installed prior to pipeline executi - [Conda](https://conda.io/) - Packaging system that manages environments instead of running analysis in containers - Can be installed by any user + - Can be installed by any user - Less reproducible than Docker / Singularity - There can be changes in low-level package dependencies over time - The software still runs in your native operating system environment and so core system functions can differ From 6aeb60d837385b3014583f8c825bfbe747f9c16d Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Thu, 5 Sep 2024 10:07:30 +0200 Subject: [PATCH 20/26] Small edits --- .../src/content/docs/usage/quick_start/installation.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md index 24d44a8a00..3594eb85e9 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -84,7 +84,7 @@ This will create the `nextflow` executable in the current directory. nextflow info ``` -Alternatively, Nextflow can be installed using [Bioconda](https://bioconda.github.io/): +Alternatively, Nextflow can be installed using `conda` via [Bioconda](https://bioconda.github.io/): 1. Add conda channels: @@ -126,7 +126,7 @@ nextflow run hello-world ``` :::note -Set the `NXF_VER` variable in your `.bashrc` or `.zshrc` to fix your version of Nextflow for all sessions. +Set the `NXF_VER` variable in your `.bashrc` or `.zshrc` if you want to fix your version of Nextflow for all sessions. ::: You can also temporarily switch to a specific version of Nextflow with the `NXF_VER` environment variable. For example: @@ -139,6 +139,10 @@ NXF_VER=23.10.0 nextflow run hello A stable version of Nextflow is released every six months, in the 4th and 10th month of each year. Additionally, an edge version is released on a monthly basis. The edge releases can be used to access the latest updates and experimental features. +:::note +Edge versions are not stable and may be subject to change. +::: + To use the latest edge release, set `NXF_EDGE=1` when updating: ```bash @@ -166,6 +170,7 @@ Historically, all tools would need to be installed manually and was often a sour nf-core pipelines utilise the built-in support for software packaging that Nextflow offers. Using profiles, software dependencies can be managed through various packaging (e.g., Docker, Singularity, and Conda). +These ensure that you have the correct tool and version of the tool that each pipeline needs to correctly execute. The respective tooling for a profile must be installed prior to pipeline execution. Follow the links below to install the required profile tooling: From 405a13953d619f9f5bdb52881364ab5cc481f0fd Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Thu, 5 Sep 2024 11:01:17 +0200 Subject: [PATCH 21/26] Apply suggested changes from comments --- .../docs/usage/configuration/introduction.md | 12 ++- .../docs/usage/configuration/modify_tools.md | 81 ++++++++++--------- .../configuration/system_requirements.md | 8 +- .../docs/usage/quick_start/installation.md | 5 +- .../docs/usage/quick_start/introduction.md | 4 +- .../usage/quick_start/your_first_pipeline.md | 8 +- 6 files changed, 62 insertions(+), 56 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/introduction.md b/sites/docs/src/content/docs/usage/configuration/introduction.md index ce2a654bbe..3b79a7474b 100644 --- a/sites/docs/src/content/docs/usage/configuration/introduction.md +++ b/sites/docs/src/content/docs/usage/configuration/introduction.md @@ -8,8 +8,8 @@ parentWeight: 20 ## Configure nf-core pipelines -Each nf-core pipeline comes with a set of “sensible defaults”. -While the defaults are a great place to start, you will certainly want to modify these to fit your own purposes and system requirements. +Each nf-core pipeline comes with a set of “sensible defaults” for a typical analysis of full size data. +While the defaults are a great place to start, you will certainly want to modify these to fit your own data and system requirements. For example, modifying a tool flag of compute resources allocated for a process. When a pipeline is launched, Nextflow will look for config files in several locations. As each source can contain conflicting settings, the sources are ranked to decide which settings to apply. @@ -28,10 +28,10 @@ Configuration sources are reported below and listed in order of priority: While some of these files are already included in the nf-core pipeline repository (e.g., the `nextflow.config` file in the nf-core pipeline repository), some are automatically identified on your local system (e.g., the `nextflow.config` in the launch directory), and others are only included if they are specified using run options (e.g., `-params-file`, and `-c`). -If you clone and manually edit an nf-core pipeline then you cannot update to more recent versions of the pipeline without overwriting your changes. You also risk moving away from the canonical pipeline and losing reproducibility. +You should not clone and manually edit an nf-core pipeline. Manually edited nf-core pipelines cannot be updated to more recent versions of the pipeline without overwriting your changes. You also risk moving away from the canonical pipeline and losing reproducibility. :::tip -Configuration locations can be used to your benefit. For example, use `-profile` for your cluster configuration, `~/.nextflow/config` for your personal configuration, and `nextflow.config` in your working directory for run-specific configuration. +Configuration locations can be used to your benefit. For example, use `-profile` for your cluster configuration, `~/.nextflow/config` for your personal configuration, and `-c specific_run.config` for run-specific configuration. ::: ### Parameters @@ -77,7 +77,7 @@ nf-core pipelines also include additional config files from the `conf/` folder o - A configuration profile to test the pipeline with a full-size test dataset :::note -Some configuration files contain the definition of profiles that can be flexibly applied. For example, the `docker`, `singularity`, and `conda` profiles are defined in the `nextflow.config` file in the pipeline project directory. +Some configuration files contain the definition of profiles that can be flexibly applied. For example, the `docker`, `singularity`, and `conda` profiles are defined in the `nextflow.config` file in the pipeline project directory. You should not need to manually edit any of these configuration files. ::: Profiles are sets of configuration options that can be flexibly applied to a pipeline. @@ -154,8 +154,6 @@ alpha { } ``` -### Scopes - Scopes allow you to quickly configure settings required to deploy a pipeline on different infrastructure using different software management. A common scenario is for users to write a custom configuration file specific to running a pipeline on their infrastructure. diff --git a/sites/docs/src/content/docs/usage/configuration/modify_tools.md b/sites/docs/src/content/docs/usage/configuration/modify_tools.md index b3b6defe9a..ee9f853980 100644 --- a/sites/docs/src/content/docs/usage/configuration/modify_tools.md +++ b/sites/docs/src/content/docs/usage/configuration/modify_tools.md @@ -1,7 +1,7 @@ --- -title: Tool arguments +title: Modifying pipelines subtitle: Configure tool containers and arguments -shortTitle: Tool arguments +shortTitle: Modifying pipelines weight: 3 --- @@ -14,35 +14,38 @@ However, you may want to modify these to fit your own purposes. **It is very unlikely that you will need to edit the pipeline code to configure a tool.** -### Docker registries +### Tool arguments -nf-core pipelines use `quay.io` as the default docker registry for Docker and Podman images. -When specifying a Docker container, it will pull the image from `quay.io` unless a full URI is specified. For example, if the process container is: +You may wish to understand which tool arguments a pipeline uses, or update to add additional arguments not supported currently by the pipeline. -```bash -biocontainers/fastqc:0.11.7--4 -``` +You can sometimes find out what parameters are used in a tool in by checking the longer 'help' description of different pipeline parameters, e.g., by pressing the 'help' button next to [this parameter](https://nf-co.re/funcscan/1.0.1/parameters#annotation_bakta_mincontig) in [nf-core/funcscan](https://nf-co.re/funcscan). -The image will be pulled from quay.io by default, resulting in a full URI of: +There are two main places that a tool can have a tool argument specified: -```bash -quay.io/biocontainers/fastqc:0.11.7--4 -``` +- The process `script` block +- The `conf/modules.conf` file -If `docker.registry` is specified, it will be used first. For example, if the config value `docker.registry = 'public.ecr.aws'` is specified the image will be pulled from: +Most arguments (both mandatory or optional) are defined in the `conf/modules.conf` file under the `ext.args` entry. Arguments that are defined in the `conf/modules.conf` file can be flexible modified using custom configuration files. -```bash -public.ecr.aws/biocontainers/fastqc:0.11.7--4 -``` +Arguments specified in `ext.args` are then inserted into the module itself via the `$args` variable in the module's bash code -However, the `docker.registry` setting will be ignored if you specify a full URI: +For example, the `-n` parameter could be added to the `BOWTIE_BUILD` process: -```bash -docker.io/biocontainers/fastqc:v0.11.9_cv8 +```groovy +process { + withName: BOWTIE_BUILD { + ext.args = "-n 0.1" + } +Updated tools may come with major changes and may break a pipeline and/or create missing values in MultiQC version tables. + +Such changes come with no warranty or support by the the pipeline developers! ``` -### Changing tool versions +:::warning +It is recommended to copy and paste existing arguments in a pipelines `conf/modules.config` file to ensure the pipeline can function as expected. +::: +### Changing tool versions You can tell the pipeline to use a different container image within a config file and the `process` scope. @@ -90,34 +93,32 @@ To update the container specification, you can do the following steps: :::warning Updated tools may come with major changes and may break a pipeline and/or create missing values in MultiQC version tables. -Such changes come with no warranty or support by the the pipeline developers! +Such changes come with no warranty or support by the the pipeline developers! ::: -### Tool arguments +### Docker registries -You may wish to understand which tool arguments a pipeline uses, or update to add additional arguments not supported currently by the pipeline. +nf-core pipelines use `quay.io` as the default docker registry for Docker and Podman images. +When specifying a Docker container, it will pull the image from `quay.io` unless a full URI is specified. For example, if the process container is: -You can sometimes find out what parameters are used in a tool in by checking the longer 'help' description of different pipeline parameters, e.g., by pressing the 'help' button next to [this parameter](https://nf-co.re/funcscan/1.0.1/parameters#annotation_bakta_mincontig) in [nf-core/funcscan](https://nf-co.re/funcscan). +```bash +biocontainers/fastqc:0.11.7--4 +``` -There are two main places that a tool can have a tool argument specified: +The image will be pulled from quay.io by default, resulting in a full URI of: -- The process `script` block -- The `conf/modules.conf` file +```bash +quay.io/biocontainers/fastqc:0.11.7--4 +``` -Most arguments (both mandatory or optional) are defined in the `conf/modules.conf` file under the `ext.args` entry. Arguments that are defined in the `conf/modules.conf` file can be flexible modified using custom configuration files. +If `docker.registry` is specified, it will be used first. For example, if the config value `docker.registry = 'public.ecr.aws'` is specified the image will be pulled from: -Arguments specified in `ext.args` are then inserted into the module itself via the `$args` variable in the module's bash code +```bash +public.ecr.aws/biocontainers/fastqc:0.11.7--4 +``` -For example, the `-n` parameter could be added to the `BOWTIE_BUILD` process: +However, the `docker.registry` setting will be ignored if you specify a full URI: -```groovy -process { - withName: BOWTIE_BUILD { - ext.args = "-n 0.1" - } -} +```bash +docker.io/biocontainers/fastqc:v0.11.9_cv8 ``` - -:::warning -It is recommended to copy and paste existing arguments in a pipelines `conf/modules.config` file to ensure the pipeline can function as expected. -::: diff --git a/sites/docs/src/content/docs/usage/configuration/system_requirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md index 09cf42934c..3a9b2e0419 100644 --- a/sites/docs/src/content/docs/usage/configuration/system_requirements.md +++ b/sites/docs/src/content/docs/usage/configuration/system_requirements.md @@ -1,11 +1,11 @@ --- -title: System requirements +title: Configure your system subtitle: Configure your system requirements -shortTitle: System requirements +shortTitle: Configure your system weight: 3 --- -## System requirements +## Configure your system nf-core pipelines are reproducible, portable, and scalable, and are designed to work "out of the box". However to maximise efficiency and benefit from all the advantages of Nextflow, it is likely you will need to configure each pipeline to suit your data and system requirements. @@ -108,7 +108,7 @@ process { } ``` -You can also modify the memory of a specific process by using the process' name. +You can also modify the memory of a specific process by using the process' name. For example, for the step of the pipeline with the name `STAR_ALIGN`, you would use the `withName` process selector. For example: ```groovy diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md index d51c250d83..ea41e9426b 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -160,7 +160,7 @@ NXF_VER=24.06.0-edge nextflow info You don't need to install nf-core tools to run nf-core pipelines. However, nf-core tools offer a number of helpful commands for users and are essential for pipeline developers. For users: - + - list nf-core pipelines - download nf-core pipelines - configure pipeline runs @@ -185,7 +185,8 @@ Using profiles, software dependencies can be managed through various packaging ( These ensure that you have the correct tool and version of the tool that each pipeline needs to correctly execute. The respective tooling for a profile must be installed prior to pipeline execution. Follow the links below to install the required profile tooling: - + - Typically requires system root access, though a _"rootless mode"_ is available + - Needs system administrator for install - [Docker](https://docs.docker.com/install/) - Typically used locally, on single-user servers, and the cloud - Analysis runs in a _container_, which behaves like an isolated operating system diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/usage/quick_start/introduction.md index 5105bb90df..3bcee0d77f 100644 --- a/sites/docs/src/content/docs/usage/quick_start/introduction.md +++ b/sites/docs/src/content/docs/usage/quick_start/introduction.md @@ -19,11 +19,11 @@ One of the key benefits of nf-core is that it promotes open development, testing - **Documentation** - nf-core pipelines have extensive documentation covering installation, usage, and description of output files to ensure that you won't be left in the dark. - **CI Testing** - - Every time a change is made to the pipeline code, nf-core pipelines use continuous-integration testing to ensure that nothing has broken. + - Every time a change is made to the pipeline code, nf-core pipelines use continuous-integration testing to check that nothing has broken. - **Stable Releases** - nf-core pipelines use GitHub releases to tag stable versions of the code and software, making pipeline runs totally reproducible. - **Packaged software** - - Pipeline dependencies are automatically downloaded and handled using Docker, Singularity, Conda, or other software management tools. There is no need for any software installations. + - Tools used by pipelines can be downloaded and handled using Docker, Singularity, Conda, or other software management tools. - **Portable and reproducible** - nf-core pipelines follow best practices to ensure maximum portability and reproducibility. You can run nf-core pipelines on most local machines, HPCs and in the cloud with minimal configuration changes. - **Cloud-ready** diff --git a/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md b/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md index 5726d0e895..414d708ef1 100644 --- a/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md +++ b/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md @@ -29,6 +29,10 @@ It is best to test it is working using a small dataset that is known to work bef nf-core pipelines come packed with a `test` profile that uses small files from the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. The `test` profile is extensively tested and should run without extra configuration. nf-core pipelines come packed with directives for containers and environments that can be flexibly enabled using software dependency profiles (e.g., `docker`, `singularity`, and `conda`). By using a profile for software dependencies an nf-core pipeline should run with all required tools. +:::note +Profiles are not just for software management, but also deeper configuration on different computing infrastructure. Some institutes have profiles to help you run nf-core pipelines on your local infrastructure. See https://nf-co.re/configs to see if yours is listed! +::: + Run the `nf-core/rnaseq` pipeline using the `test` profile and a software dependencies profile (e.g., `docker`) to confirm Nextflow is installed and that you have an active internet connection: ```bash @@ -37,4 +41,6 @@ nextflow run nf-core/rnaseq -profile test,docker --outdir results If everything has been installed properly you should see the pipeline launch and run to completion. -Congratulations! You have just run your first nf-core pipeline! You are now ready to start adding your own data and configure the pipeline to better suit your requirements. +Congratulations! You have just run your first nf-core pipeline! + +What's next? Check out the [Configuration](/docs/usage/configuration/introduction) guide for more information about adding your own data and configuring a pipeline to better suit your needs. From 6dca224129d68ecc205f9e5d299636293fd89d61 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Thu, 5 Sep 2024 11:52:47 +0200 Subject: [PATCH 22/26] Prettier --- .../usage/configuration/running_offline.md | 2 +- .../configuration/system_requirements.md | 5 ++--- .../docs/usage/quick_start/installation.md | 21 ++++++++++--------- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/sites/docs/src/content/docs/usage/configuration/running_offline.md b/sites/docs/src/content/docs/usage/configuration/running_offline.md index 7569d99771..b0084b7b44 100644 --- a/sites/docs/src/content/docs/usage/configuration/running_offline.md +++ b/sites/docs/src/content/docs/usage/configuration/running_offline.md @@ -22,7 +22,7 @@ These will first need to be fetched on a system that _does_ have an internet con You need to have Nextflow installed on your local system. You can do this by installing Nextflow on a machine that _does_ have an internet connection and transferring to the offline system: -1. Install Nextflow locally +1. [Install Nextflow locally](/docs/usage/quick_start/installation.md) :::warning Do _not_ use the `-all` package, as this does not allow the use of custom plugins. ::: diff --git a/sites/docs/src/content/docs/usage/configuration/system_requirements.md b/sites/docs/src/content/docs/usage/configuration/system_requirements.md index 3a9b2e0419..eed0110619 100644 --- a/sites/docs/src/content/docs/usage/configuration/system_requirements.md +++ b/sites/docs/src/content/docs/usage/configuration/system_requirements.md @@ -68,7 +68,6 @@ You should not modify the `base.config` of the pipeline! But always modify resou Where possible, pipeline steps are tuned to make use of the resources available. For example, if a tool allows specification of the number of CPUs or threads to use with the parameter (e.g., `-p`), nf-core pipeline modules will use this parameter when executing the tool, with the number of CPUs being derived from either the `base.config` or your own custom config file. - By default, process resources are inherited by a label. For example: ```groovy @@ -91,7 +90,7 @@ process { } ``` -A default [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) function will apply the thresholds set in `--max_cpus`, `--max_memory` and `--max_time` (or `params.max_cpus` etc. in a config file). +A default [`check_max()`](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) function will apply the thresholds set in `--max_cpus`, `--max_memory` and `--max_time` (or `params.max_cpus` etc. in a config file). The `* task.attempt` means that these values are doubled and automatically retried after failing with an exit code that corresponds to a lack of resources. :::warning @@ -108,7 +107,7 @@ process { } ``` -You can also modify the memory of a specific process by using the process' name. +You can also modify the memory of a specific process by using the process' name. For example, for the step of the pipeline with the name `STAR_ALIGN`, you would use the `withName` process selector. For example: ```groovy diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/usage/quick_start/installation.md index ea41e9426b..c094e576e6 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/usage/quick_start/installation.md @@ -161,17 +161,17 @@ You don't need to install nf-core tools to run nf-core pipelines. However, nf-co For users: - - list nf-core pipelines - - download nf-core pipelines - - configure pipeline runs - - launch pipeline runs +- list nf-core pipelines +- download nf-core pipelines +- configure pipeline runs +- launch pipeline runs For developers: - - Create nf-core components from nf-core templates - - Lint nf-core components - - Manage nf-core pipeline schemas - - and more! +- Create nf-core components from nf-core templates +- Lint nf-core components +- Manage nf-core pipeline schemas +- and more! See [nf-core tools](/docs/nf-core-tools) for more information. @@ -185,8 +185,9 @@ Using profiles, software dependencies can be managed through various packaging ( These ensure that you have the correct tool and version of the tool that each pipeline needs to correctly execute. The respective tooling for a profile must be installed prior to pipeline execution. Follow the links below to install the required profile tooling: - - Typically requires system root access, though a _"rootless mode"_ is available - - Needs system administrator for install + +- Typically requires system root access, though a _"rootless mode"_ is available +- Needs system administrator for install - [Docker](https://docs.docker.com/install/) - Typically used locally, on single-user servers, and the cloud - Analysis runs in a _container_, which behaves like an isolated operating system From 967fe5d6e6a30602744f6bfa8eb6039d01756dd8 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 9 Sep 2024 10:54:06 +0200 Subject: [PATCH 23/26] Rename usage guides and typos --- .../configuration/introduction.md | 10 +++----- .../configuration/modify_tools.md | 25 ++++++++++--------- .../configuration/running_offline.md | 0 .../configuration/system_requirements.md | 23 ++++++++--------- .../quick_start/installation.md | 15 ++++++----- .../quick_start/introduction.md | 8 +++--- .../quick_start/your_first_pipeline.md | 21 ++++++++++------ .../{usage => guides}/reference_genomes.md | 0 .../troubleshooting/aquire_lock_error.md | 0 .../troubleshooting/basics.md | 0 .../troubleshooting/configuration.md | 0 .../troubleshooting/crash_halfway.md | 0 .../troubleshooting/docker_permissions.md | 0 .../troubleshooting/early_crash.md | 0 .../troubleshooting/input_output.md | 0 .../{usage => guides}/troubleshooting/ipv6.md | 0 .../troubleshooting/overview.md | 0 .../troubleshooting/stuck_on_revision.md | 0 .../troubleshooting/update_does_not_work.md | 0 sites/docs/src/pages/docs/index.astro | 9 ++++++- 20 files changed, 60 insertions(+), 51 deletions(-) rename sites/docs/src/content/docs/{usage => guides}/configuration/introduction.md (93%) rename sites/docs/src/content/docs/{usage => guides}/configuration/modify_tools.md (86%) rename sites/docs/src/content/docs/{usage => guides}/configuration/running_offline.md (100%) rename sites/docs/src/content/docs/{usage => guides}/configuration/system_requirements.md (84%) rename sites/docs/src/content/docs/{usage => guides}/quick_start/installation.md (95%) rename sites/docs/src/content/docs/{usage => guides}/quick_start/introduction.md (98%) rename sites/docs/src/content/docs/{usage => guides}/quick_start/your_first_pipeline.md (81%) rename sites/docs/src/content/docs/{usage => guides}/reference_genomes.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/aquire_lock_error.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/basics.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/configuration.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/crash_halfway.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/docker_permissions.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/early_crash.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/input_output.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/ipv6.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/overview.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/stuck_on_revision.md (100%) rename sites/docs/src/content/docs/{usage => guides}/troubleshooting/update_does_not_work.md (100%) diff --git a/sites/docs/src/content/docs/usage/configuration/introduction.md b/sites/docs/src/content/docs/guides/configuration/introduction.md similarity index 93% rename from sites/docs/src/content/docs/usage/configuration/introduction.md rename to sites/docs/src/content/docs/guides/configuration/introduction.md index 3b79a7474b..9cca72df8e 100644 --- a/sites/docs/src/content/docs/usage/configuration/introduction.md +++ b/sites/docs/src/content/docs/guides/configuration/introduction.md @@ -8,7 +8,7 @@ parentWeight: 20 ## Configure nf-core pipelines -Each nf-core pipeline comes with a set of “sensible defaults” for a typical analysis of full size data. +Each nf-core pipeline comes with a set of “sensible defaults” for a "typical" analysis of a full size dataset. While the defaults are a great place to start, you will certainly want to modify these to fit your own data and system requirements. For example, modifying a tool flag of compute resources allocated for a process. When a pipeline is launched, Nextflow will look for config files in several locations. @@ -28,10 +28,8 @@ Configuration sources are reported below and listed in order of priority: While some of these files are already included in the nf-core pipeline repository (e.g., the `nextflow.config` file in the nf-core pipeline repository), some are automatically identified on your local system (e.g., the `nextflow.config` in the launch directory), and others are only included if they are specified using run options (e.g., `-params-file`, and `-c`). +:::warning You should not clone and manually edit an nf-core pipeline. Manually edited nf-core pipelines cannot be updated to more recent versions of the pipeline without overwriting your changes. You also risk moving away from the canonical pipeline and losing reproducibility. - -:::tip -Configuration locations can be used to your benefit. For example, use `-profile` for your cluster configuration, `~/.nextflow/config` for your personal configuration, and `-c specific_run.config` for run-specific configuration. ::: ### Parameters @@ -138,7 +136,7 @@ nextflow run nf-core/rnaseq -profile docker --input --outdir ` caps keep requests from getting out of hand and crashing the entire pipeline run. If a particular job exceeds the process-specific default resources and is retried, only resource requests (cpu, memory, or time) that have not yet reached the value set with `--max_` will be increased during the retry. @@ -61,10 +59,6 @@ You may wish to increase resources if you get a pipeline reporting a step failin You should not modify the `base.config` of the pipeline! But always modify resource requests in your own custom or institutioanl config file! ::: -:::warning -You should not modify the `base.config` of the pipeline! But always modify resource requests in your own custom or institutioanl config file! -::: - Where possible, pipeline steps are tuned to make use of the resources available. For example, if a tool allows specification of the number of CPUs or threads to use with the parameter (e.g., `-p`), nf-core pipeline modules will use this parameter when executing the tool, with the number of CPUs being derived from either the `base.config` or your own custom config file. @@ -97,7 +91,9 @@ The `* task.attempt` means that these values are doubled and automatically retri If you want to use the `check_max()` function in a custom configuration file, you must copy the [check_max function](https://github.com/nf-core/tools/blob/99961bedab1518f592668727a4d692c4ddf3c336/nf_core/pipeline-template/nextflow.config#L206-L237) to the bottom of your custom config ::: -To modify the memory to all processes with the `process_high` label you can use the `withLabel` process selector in your config file. For example: +Selectors can be used to fine file specific parts of an nf-core pipeline. + +To modify the memory to **all processes with a specific label** you can use the `withLabel` process selector. For example: ```groovy process { @@ -107,8 +103,9 @@ process { } ``` -You can also modify the memory of a specific process by using the process' name. -For example, for the step of the pipeline with the name `STAR_ALIGN`, you would use the `withName` process selector. For example: +This example would modify the memory for all processes with the `process_high` label. + +To modify the memory to **specific processes** you can use the `withName` process selector. For example: ```groovy process { @@ -118,7 +115,7 @@ process { } ``` -Using `withName` allows you to optimise the resource usage of the pipeline to a very fine grained level. +This example would modify the memory of the `STAR_ALIGN` process. If a pipeline uses a tool multiple times you may need to specify the whole 'execution path' of the module. For example: diff --git a/sites/docs/src/content/docs/usage/quick_start/installation.md b/sites/docs/src/content/docs/guides/quick_start/installation.md similarity index 95% rename from sites/docs/src/content/docs/usage/quick_start/installation.md rename to sites/docs/src/content/docs/guides/quick_start/installation.md index c094e576e6..7c8c7083b7 100644 --- a/sites/docs/src/content/docs/usage/quick_start/installation.md +++ b/sites/docs/src/content/docs/guides/quick_start/installation.md @@ -202,14 +202,13 @@ The respective tooling for a profile must be installed prior to pipeline executi - [Apptainer](https://apptainer.org/) - Open source version of Singularity (split from Singularity in 2021) - :::warning - Currently, nf-core pipelines run with `-profile apptainer` will build using - docker containers instead of using pre-built singularity containers. - - To use the singularity containers, use `-profile singularity` instead. - This works because `apptainer` simply defines `singularity` as an alias - to the `apptainer` command. - ::: + +:::warning +Currently, nf-core pipelines run with `-profile apptainer` will build using +docker containers instead of the pre-built singularity containers. To use the singularity containers, use `-profile singularity` instead. +This works because `apptainer` simply defines `singularity` as an alias +to the `apptainer` command. +::: - [Podman](https://podman.io/), [Charliecloud](https://hpc.github.io/charliecloud/), and [Shifter](https://www.nersc.gov/research-and-development/user-defined-images/) - All alternatives to Docker, often used on HPC systems diff --git a/sites/docs/src/content/docs/usage/quick_start/introduction.md b/sites/docs/src/content/docs/guides/quick_start/introduction.md similarity index 98% rename from sites/docs/src/content/docs/usage/quick_start/introduction.md rename to sites/docs/src/content/docs/guides/quick_start/introduction.md index 3bcee0d77f..9e89a4d761 100644 --- a/sites/docs/src/content/docs/usage/quick_start/introduction.md +++ b/sites/docs/src/content/docs/guides/quick_start/introduction.md @@ -46,18 +46,20 @@ Each released pipeline has a dedicated page that includes 6 documentation sectio Each section should be explored to understand what the pipeline does and how it can be configured. -## Any questions? Wanna help out? Join the nf-core community! +## Join the nf-core community! + +Any questions? Wanna help out? Join the nf-core community! The nf-core Slack is the place to get help, share expertise, and connect with other community members. +We hope to see you there! +

Join the nf-core Slack

-We hope to see you there! - **Important to know** - **Community conduct**: It’s important that this community is an open, collaborative, and welcoming environment. Read our nf-core [Code of Conduct](https://nf-co.re/code_of_conduct). diff --git a/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md b/sites/docs/src/content/docs/guides/quick_start/your_first_pipeline.md similarity index 81% rename from sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md rename to sites/docs/src/content/docs/guides/quick_start/your_first_pipeline.md index 414d708ef1..33378630b1 100644 --- a/sites/docs/src/content/docs/usage/quick_start/your_first_pipeline.md +++ b/sites/docs/src/content/docs/guides/quick_start/your_first_pipeline.md @@ -17,15 +17,22 @@ Run Nextflow’s example [Hello world!](https://github.com/nextflow-io/hello) pi nextflow run hello ``` -If everything has been installed properly you should see the pipeline launch and run to completion. - -Congratulations! You have just run your first Nextflow pipeline! +If everything has been installed properly you should see the pipeline launch and run to completion: + +```console + N E X T F L O W ~ version 24.04.4 +Launching `https://github.com/nextflow-io/hello` [cheesy_sax] DSL2 - revision: afff16a9b4 [master] +executor > local (4) +[2c/3092e1] sayHello (4) [100%] 4 of 4 ✔ +Hello world! +Bonjour world! +Hola world! +Ciao world! +``` ### nf-core pipelines -Each nf-core pipeline will have different run requirements. - -It is best to test it is working using a small dataset that is known to work before running a pipeline with your own data. +Each nf-core pipeline will have different run requirements. It is best to test it is working using a small dataset that is known to work before running a pipeline with your own data. nf-core pipelines come packed with a `test` profile that uses small files from the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository. The `test` profile is extensively tested and should run without extra configuration. nf-core pipelines come packed with directives for containers and environments that can be flexibly enabled using software dependency profiles (e.g., `docker`, `singularity`, and `conda`). By using a profile for software dependencies an nf-core pipeline should run with all required tools. @@ -41,6 +48,4 @@ nextflow run nf-core/rnaseq -profile test,docker --outdir results If everything has been installed properly you should see the pipeline launch and run to completion. -Congratulations! You have just run your first nf-core pipeline! - What's next? Check out the [Configuration](/docs/usage/configuration/introduction) guide for more information about adding your own data and configuring a pipeline to better suit your needs. diff --git a/sites/docs/src/content/docs/usage/reference_genomes.md b/sites/docs/src/content/docs/guides/reference_genomes.md similarity index 100% rename from sites/docs/src/content/docs/usage/reference_genomes.md rename to sites/docs/src/content/docs/guides/reference_genomes.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/aquire_lock_error.md b/sites/docs/src/content/docs/guides/troubleshooting/aquire_lock_error.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/aquire_lock_error.md rename to sites/docs/src/content/docs/guides/troubleshooting/aquire_lock_error.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/basics.md b/sites/docs/src/content/docs/guides/troubleshooting/basics.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/basics.md rename to sites/docs/src/content/docs/guides/troubleshooting/basics.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/configuration.md b/sites/docs/src/content/docs/guides/troubleshooting/configuration.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/configuration.md rename to sites/docs/src/content/docs/guides/troubleshooting/configuration.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/crash_halfway.md b/sites/docs/src/content/docs/guides/troubleshooting/crash_halfway.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/crash_halfway.md rename to sites/docs/src/content/docs/guides/troubleshooting/crash_halfway.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/docker_permissions.md b/sites/docs/src/content/docs/guides/troubleshooting/docker_permissions.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/docker_permissions.md rename to sites/docs/src/content/docs/guides/troubleshooting/docker_permissions.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/early_crash.md b/sites/docs/src/content/docs/guides/troubleshooting/early_crash.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/early_crash.md rename to sites/docs/src/content/docs/guides/troubleshooting/early_crash.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/input_output.md b/sites/docs/src/content/docs/guides/troubleshooting/input_output.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/input_output.md rename to sites/docs/src/content/docs/guides/troubleshooting/input_output.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/ipv6.md b/sites/docs/src/content/docs/guides/troubleshooting/ipv6.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/ipv6.md rename to sites/docs/src/content/docs/guides/troubleshooting/ipv6.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/overview.md b/sites/docs/src/content/docs/guides/troubleshooting/overview.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/overview.md rename to sites/docs/src/content/docs/guides/troubleshooting/overview.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/stuck_on_revision.md b/sites/docs/src/content/docs/guides/troubleshooting/stuck_on_revision.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/stuck_on_revision.md rename to sites/docs/src/content/docs/guides/troubleshooting/stuck_on_revision.md diff --git a/sites/docs/src/content/docs/usage/troubleshooting/update_does_not_work.md b/sites/docs/src/content/docs/guides/troubleshooting/update_does_not_work.md similarity index 100% rename from sites/docs/src/content/docs/usage/troubleshooting/update_does_not_work.md rename to sites/docs/src/content/docs/guides/troubleshooting/update_does_not_work.md diff --git a/sites/docs/src/pages/docs/index.astro b/sites/docs/src/pages/docs/index.astro index fe9d371173..40133f94a3 100644 --- a/sites/docs/src/pages/docs/index.astro +++ b/sites/docs/src/pages/docs/index.astro @@ -10,7 +10,14 @@ let docs = await getCollection('docs'); import SidebarNav from '@components/sidebar/SidebarNav.astro'; let sections: SidebarEntry[] = [ - { type: 'group', id: 'usage', label: 'Usage', entries: [], collapsed: true, icon: 'octicon:play-16', weight: 1 }, + { + type: 'group', + id: 'guides', + label: 'Guides', + entries: [], + collapsed: true, + icon: 'octicon:play-16', + weight: 1 }, { type: 'group', id: 'contributing', From 210d30ca2fcb4b13b79a6f025cd780ddf0f00aa8 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 9 Sep 2024 10:57:36 +0200 Subject: [PATCH 24/26] Fix index --- sites/docs/src/pages/docs/index.astro | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sites/docs/src/pages/docs/index.astro b/sites/docs/src/pages/docs/index.astro index 40133f94a3..21d2a4745b 100644 --- a/sites/docs/src/pages/docs/index.astro +++ b/sites/docs/src/pages/docs/index.astro @@ -17,7 +17,7 @@ let sections: SidebarEntry[] = [ entries: [], collapsed: true, icon: 'octicon:play-16', - weight: 1 }, + weight: 1}, { type: 'group', id: 'contributing', From d1ce24cc3989de94a6008ceeecb80bae4616d009 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Mon, 9 Sep 2024 11:07:58 +0200 Subject: [PATCH 25/26] Fix index again --- sites/docs/src/pages/docs/index.astro | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sites/docs/src/pages/docs/index.astro b/sites/docs/src/pages/docs/index.astro index 21d2a4745b..a23dd8ad2b 100644 --- a/sites/docs/src/pages/docs/index.astro +++ b/sites/docs/src/pages/docs/index.astro @@ -17,7 +17,8 @@ let sections: SidebarEntry[] = [ entries: [], collapsed: true, icon: 'octicon:play-16', - weight: 1}, + weight: 1, + }, { type: 'group', id: 'contributing', From bcfe614635e43d313ff765b03936bf711f8b3723 Mon Sep 17 00:00:00 2001 From: christopher-hakkaart Date: Wed, 11 Sep 2024 09:14:22 +0200 Subject: [PATCH 26/26] Blend in #2537 --- .../docs/guides/quick_start/installation.md | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/sites/docs/src/content/docs/guides/quick_start/installation.md b/sites/docs/src/content/docs/guides/quick_start/installation.md index 7c8c7083b7..4cc98cd577 100644 --- a/sites/docs/src/content/docs/guides/quick_start/installation.md +++ b/sites/docs/src/content/docs/guides/quick_start/installation.md @@ -12,7 +12,7 @@ Nextflow must be installed on the system where you launch an nf-core pipeline. Nextflow can be used on any POSIX-compatible system (Linux, macOS, etc), and on Windows through [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux). It requires Bash 3.2 (or later) and [Java 11 (or later, up to 22)](https://www.oracle.com/java/technologies/downloads/?er=221886). :::note -A personal installation of Nextflow is recommended to simplify updates and version control. +Use a personal installation of Nextflow where possible, instead of a system-wide installation. ::: ### Install Java @@ -84,7 +84,7 @@ This will create the `nextflow` executable in the current directory. nextflow info ``` -Alternatively, Nextflow can be installed using `conda` via [Bioconda](https://bioconda.github.io/): +Alternatively, to install Nextflow with [Bioconda](https://bioconda.github.io/), first set up Bioconda according to the [Bioconda documentation](https://bioconda.github.io/#usage), notably setting up channels: 1. Add conda channels: @@ -106,7 +106,7 @@ Alternatively, Nextflow can be installed using `conda` via [Bioconda](https://bi ``` :::note - To deactivate the `env_nf` conda environment you can use the `conda deactivate` command. + Use `conda deactivate env_nf` deactivate the `env_nf` environment. ::: ### Update Nextflow @@ -180,7 +180,7 @@ See [nf-core tools](/docs/nf-core-tools) for more information. Analysis pipelines often chain together the execution of multiple tools. Historically, all tools would need to be installed manually and was often a source of great frustration and irreproducibility. -nf-core pipelines utilise the built-in support for software packaging that Nextflow offers. +nf-core pipelines utilise Nextflow's built-in support for software packaging that Nextflow offers. Using profiles, software dependencies can be managed through various packaging (e.g., Docker, Singularity, and Conda). These ensure that you have the correct tool and version of the tool that each pipeline needs to correctly execute. @@ -190,15 +190,14 @@ The respective tooling for a profile must be installed prior to pipeline executi - Needs system administrator for install - [Docker](https://docs.docker.com/install/) - Typically used locally, on single-user servers, and the cloud - - Analysis runs in a _container_, which behaves like an isolated operating system + - Analysis runs in a container, which behaves like an isolated operating system - Typically requires system root access, though a _"rootless mode"_ is available - - Needs system administrator for install + - Requires system administrator for install - [Singularity](https://www.sylabs.io/) - Often used as an alternative to Docker on HPC systems - Also runs _containers_, and can optionally create these from Docker images - - Does not need root access or any daemon processes - - Needs system administrator for install - - Needs system administrator for install + - Does not require root access or any daemon processes + - Requires system administrator for install - [Apptainer](https://apptainer.org/) - Open source version of Singularity (split from Singularity in 2021) @@ -217,8 +216,8 @@ to the `apptainer` command. - Can be installed by any user - Can be installed by any user - Less reproducible than Docker / Singularity - - There can be changes in low-level package dependencies over time - - The software still runs in your native operating system environment and so core system functions can differ + - Low-level changes in package dependencies may occur over time + - Software still runs in your native operating system environment — core system functions may differ - [Mamba](https://mamba.readthedocs.io/) - A faster implementation of Conda