v2.5.1 - Gold Otter Patch

This release fixes a sync error with Black linting in pipelines.

• Patch release to fix black linting in pipelines (#1789)
• Add isort options to pyproject.toml (#1792)
• Lint pyproject.toml file exists and content (#1795)
• Update GitHub PyPI package release action to v1 (#1785)

v2.5 - Gold Otter

Full changelog: 2.4.1...2.5

Template

• Bumped Python version to 3.7 in the GitHub linting in the workflow template (#1680)
• Fix bug in pipeline readme logo URL (#1590)
• Switch CI to use setup-nextflow action to install Nextflow (#1650)
• Add CITATION.cff #361
• Add Gitpod and Mamba profiles to the pipeline template (#1673)
• Remove call to getGenomeAttribute in main.nf when running nf-core create without iGenomes (#1670)
• Make nf-core create fail if Git default branch name is dev or TEMPLATE (#1705)
• Convert console snippets to bash snippets in the template where applicable (#1729)
• Add branch field to module entries in modules.json to record what branch a module was installed from (#1728)
• Add customisation option to remove all GitHub support with nf-core create (#1766)

Linting

• Check that the .prettierignore file exists and that starts with the same content.
• Update readme.py nf version badge validation regexp to accept any signs before version number (#1613)
• Add isort configuration and GitHub workflow (#1538)
• Use black also to format python files in workflows (#1563)
• Add check for mimetype in the input parameter. (#1647)
• Check that the singularity and docker tags are parsable. Add --fail-warned flag to nf-core modules lint (#1654)
• Handle exception in nf-core modules lint when process name doesn't start with process (#1733)

General

• Remove support for Python 3.6 (#1680)
• Add support for Python 3.9 and 3.10 (#1680)
• Invoking Python with optimizations no longer affects the program control flow (#1685)
• Update readme to drop --key option from nf-core modules list and add the new pattern syntax
• Add --fail-warned flag to nf-core lint to make warnings fail (#1593)
• Add --fail-warned flag to pipeline linting workflow (#1593)
• Updated the nf-core package requirements (#1620, #1757, #1756)
• Remove dependency of the mock package and use unittest.mock instead (#1696)
• Fix and improve broken test for Singularity container download (#1622)
• Use $XDG_CACHE_HOME or ~/.cache instead of $XDG_CONFIG_HOME or ~/config/ as base directory for API cache
• Switch CI to use setup-nextflow action to install Nextflow (#1650)
• Add tests for nf-core modules update and ModulesJson.
• Add CI for GitLab remote #1646
• Add CITATION.cff #361
• Allow customization of the nf-core pipeline template when using nf-core create (#1548)
• Add Refgenie integration: updating of nextflow config files with a refgenie database (#1090)
• Fix --key option in nf-core lint when supplying a module lint test name (#1681)
• Add no_git=True when creating a new pipeline and initialising a git repository is not needed in nf-core lint and nf-core bump-version (#1709)
• Move strip_ansi_code function in lint to utils.py
• Simplify control flow and don't use equality comparison for None and booleans
• Replace use of the deprecated distutils Version object with that from packaging (#1735)
• Add code to cancel CI run if a new run starts (#1760)
• CI for the API docs generation now uses the ubuntu-latest base image (#1762)
• Add option to hide progress bars in nf-core lint and nf-core modules lint with --hide-progress.

Modules

• Add --fix-version flag to nf-core modules lint command to update modules to the latest version (#1588)
• Fix a bug in the regex extracting the version from biocontainers URLs (#1598)
• Update how we interface with git remotes. (#1626)
• Add prompt for module name to nf-core modules info (#1644)
• Update docs with example of custom git remote (#1645)
• Command nf-core modules test obtains module name suggestions from installed modules (#1624)
• Add --base-path flag to nf-core modules to specify the base path for the modules in a remote. Also refactored modules.json code. (#1643) Removed after (#1754)
• Rename methods in ModulesJson to remove explicit reference to modules.json
• Fix inconsistencies in the --save-diff flag nf-core modules update. Refactor nf-core modules update (#1536)
• Fix bug in ModulesJson.check_up_to_date causing it to ask for the remote of local modules
• Handle errors when updating module version with nf-core modules update --fix-version (#1671)
• Make nf-core modules update --save-diff work when files were created or removed (#1694)
• Get the latest common build for Docker and Singularity containers of a module (#1702)
• Add short option for --no-pull option in nf-core modules
• Add nf-core modules patch command (#1312)
• Add support for patch in nf-core modules update command (#1312)
• Add support for patch in nf-core modules lint command (#1312)
• Add support for custom remotes in nf-core modules lint (#1715)
• Make nf-core modules commands work with arbitrary git remotes (#1721)
• Add links in README.md for info and patch commands (#1722])
• Fix misc. issues with --branch and --base-path (#1726)
• Add branch field to module entries in modules.json to record what branch a module was installed from (#1728)
• Fix broken link in nf-core modules info(#1745)
• Fix unbound variable issues and minor refactoring #1742
• Recreate modules.json file instead of complaining about incorrectly formatted file. (#1741
• Add support for patch when creating modules.json file (#1752)

v2.4.1 - Cobolt Koala Patch

Somehow it wouldn't feel like a real nf-core/tools release if there wasn't some error with the automatic template sync that required an immediate patch release.. 🤔

Hopefully this release will fill everyone's inboxes with automated update pull requests.. 🤞🏻

• Patch release to try to fix the template sync (#1585)
• Avoid persistent temp files from pytests (#1566)
• Add option to trigger sync manually on just nf-core/testpipeline

v2.4 - Cobolt Koala

This release started off as a patch release to fix some errors introduced in 2.3 (eg. pipeline email template errors). However, it slowly collected enough new features to warrant a proper minor release of its own.

Many thanks to everyone who contributed!

Full Changelog: 2.3.2...2.4

Template

• ✨ Add actions workflow to respond to @nf-core-bot fix linting comments on pipeline PRs
• Fix Prettier formatting bug in completion email HTML template (#1509)
• Fix bug in pipeline readme logo URL
• Set the default DAG graphic output to HTML to have a default that does not depend on Graphviz being installed on the host system (#1512).
• Removed retry strategy for AWS tests CI, as Nextflow now handles spot instance retries itself
• Add .prettierignore file to stop Prettier linting tests from running over test files
• Made module template test command match the default used in nf-core modules create-test-yml (#1562)
• Removed black background from Readme badges now that GitHub has a dark mode, added Tower launch badge.
• Don't save md5sum for versions.yml when running nf-core modules create-test-yml (#1511)
• Read entire lines when sniffing the samplesheet format (fix #1561)

General

• ✨ Add actions workflow to respond to @nf-core-bot fix linting comments on nf-core/tools PRs
• Use $XDG_CONFIG_HOME or ~/.config/nf-core instead of ~/.nfcore for API cache (the latter can be safely deleted)
• Consolidate GitHub API calls into a shared function that uses authentication from the gh GitHub cli tool or GITHUB_AUTH_TOKEN to avoid rate limiting (#1499)
• Add an empty line to modules.json, params.json and nextflow-schema.json when dumping them to avoid prettier errors.
• Remove empty JSON schema definition groups to avoid usage errors (#1419)
• Bumped the minimum version of rich from v10 to v10.7.0

Modules

• ✨ Add a new command nf-core modules mulled which can generate the name for a multi-tool container image.
• ✨ Add a new command nf-core modules test which runs pytests locally.
• ✨ Print include statement to terminal when modules install (#1520)
• Allow follow links when generating test.yml file with nf-core modules create-test-yml (1570)
• Escaped test run output before logging it, to avoid a rich MarkupError

Linting

• Don't allow a .nf-core.yaml file, should be .yml (#1515).
• shell blocks now recognised to avoid error when: condition has too many lines (#1557)
• Fixed error when using comments after input tuple lines (#1542)
• Don't lint the shell block when script is used (1558)
• Check that template is used in script blocks
• Tweaks to CLI output display of lint results

v2.3.2 - Mercury Vulture Fixed Formatting

Very minor patch release to fix the full size AWS tests and re-run the template sync, which partially failed due to GitHub pull-requests being down at the time of release.

Template

• Updated the AWS GitHub actions to let nf-core/tower-action use it's defaults for pipeline and git sha (#1488)
• Add prettier editor extension to gitpod.yml in template (#1485)
• Remove traces of markdownlint in the template (#1486
• Remove accidentally added line in CHANGELOG.md in the template (#1487)
• Update linting to check that .editorconfig is there and .yamllint.yml isn't.

v2.3.1 - Mercury Vulture Formatting

⚠️ TLDR: Run Prettier on your pipeline's codebase before attempting the template sync merge.

The sync PR may be a little big because of many major changes (whitespace, quotation mark styles etc). To help with the merge, we highly recommend running Prettier on your pipeline's codebase before attempting the template merge.

Please copy .editorconfig and .prettierrc.yml from the template to your pipeline root first, as they configure the behaviour of Prettier.

---

This patch release is primarily to address problems that we had in the v2.3 release with code linting. Instead of resolving those specific issues, we chose to replace the linting tools (markdownlint, yamllint) with a new tool: Prettier

This is a fairly major change and affects a lot of files. However, it will hopefully simplify future usage. Prettier can auto-format many different file formats (for pipelines the most relevant are markdown and YAML) and is extensible with plugins (Nextflow, anyone?). It tends to be a bit less strict than markdownlint and yamllint and importantly can fix files for you rather than just complaining.

To run Prettier, go to the base of the repository where .editorconfig and .prettierrc.yml are located. Make sure your git status is clean so that the changes don't affect anything you're working on and run:

prettier --write .

This runs Prettier and tells it to fix any issues it finds in place.

Please note that there are many excellent integrations for Prettier available, for example VSCode can be set up to automatically format files on save.

---

Template

• Replace markdownlint and yamllint with Prettier for linting formatting / whitespace (#1470)
• Add CI test using editorconfig-checker for other file types to look for standardised indentation and formatting (#1476)
• Add md5sum check of versions.yml to test.yml on the modules template.
• Update bundled module wrappers to latest versions (#1462)
• Renamed assets/multiqc_config.yaml to assets/multiqc_config.yml (yml not yaml) (#1471)

General

• Convert nf-core/tools API / lint test documentation to MyST (#1245)
• Build documentation for the nf-core modules lint tests (#1250)
• Fix some colours in the nf-core/tools API docs (#1467)
• Install tools inside GitPod Docker using the repo itself and not from Conda.
• Rewrite GitHub Actions workflow for publishing the GitPod Docker image.
• Improve config for PyTest so that you can run pytest instead of pytest tests/ (#1461)
• New pipeline lint test multiqc_config that checks YAML structure instead of basic file contents (#1461)
• Updates to the GitPod docker image to install the latest version of nf-core/tools

v2.3 - Mercury Vulture

Another big release that's been coming for quite some time (apologies to all who have been running with dev for weeks / months).

One or two major bugfixes, such as making nf-core download work again with the latest DSL2 formats. See below for the full list of changes in this release.

Full Changelog: 2.2...2.3

Template

• Removed mention of --singularity_pull_docker_container in pipeline README.md
• Replaced equals with ~ in nf-core headers, to stop false positive unresolved conflict errors when committing with VSCode.
• Add retry strategy for AWS megatests after releasing nf-core/tower-action v2.2
• Added .nf-core.yml file with repository_type: pipeline for modules commands
• Update igenomes path to the BWAIndex to fetch the whole version0.6.0 folder instead of only the genome.fa file
• Remove pinned Node version in the GitHub Actions workflows, to fix errors with markdownlint
• Bumped nf-core/tower-action to v3 and removed pipeline and revision from the AWS workflows, which were not needed
• Add yamllint GitHub Action.
• Add .yamllint.yml to avoid line length and document start errors (#1407)
• Add --publish_dir_mode back into the pipeline template (nf-core/rnaseq#752)
• Add optional loading of of pipeline-specific institutional configs to nextflow.config
• Make --outdir a mandatory parameter (nf-core/tools#1415)

General

• Updated nf-core download to work with latest DSL2 syntax for containers (#1379)
• Made nf-core modules create detect repository type with explicit .nf-core.yml instead of random readme stuff (#1391)
• Added a Gitpod environment and Dockerfile (#1384)
  • Adds conda, Nextflow, nf-core, pytest-workflow, mamba, and pip to base Gitpod Docker image.
  • Adds GH action to build and push Gitpod Docker image.
  • Adds Gitpod environment to template.
  • Adds Gitpod environment to tools with auto build of nf-core tool.
• Shiny new command-line help formatting (#1403)
• Call the command line help with -h as well as --help (was formerly just the latter) (#1404)
• Add .yamllint.yml config file to avoid line length and document start errors in the tools repo itself.
• Switch to yamllint-github-actionto be able to configure yaml lint exceptions (#1404)
• Prevent module linting KeyError edge case (#1321)
• Bump-versions: Don't trim the trailing newline on files, causes editorconfig linting to fail (#1265)
• Handle exception in nf-core list when a broken git repo is found (#1273)
• Updated URL for pipeline lint test docs (#1348)
• Updated nf-core create to tolerate failures and retry when fetching pipeline logos from the website (#1369)

Modules

• New command nf-core modules info that prints nice documentation about a module to the terminal ✨ (#1427)
• Linting a pipeline now fails instead of warning if a local copy of a module does not match the remote (#1313)
• Fixed linting bugs where warning was incorrectly generated for:
  • Module does not emit software version
  • Container versions do not match
  • input: / output: not being specified in module
  • Allow for containers from other biocontainers resource as defined here
• Fixed traceback when using stageAs syntax as defined here
• Added nf-core schema docs command to output pipline parameter documentation in Markdown format for inclusion in GitHub and other documentation systems (#741)
• Allow conditional process execution from the configuration file (#1393)
• Add linting for when condition(#1397)
• Added modules ignored table to nf-core modules bump-versions. (#1234)
• Added --conda-package-version flag for specifying version of conda package in nf-core modules create. (#1238)
• Add option of writing diffs to file in nf-core modules update using either interactive prompts or the new --diff-file flag.
• Fixed edge case where module names that were substrings of other modules caused both to be installed (#1380)
• Tweak handling of empty files when generating the test YAML (#1376)
  • Fail linting if a md5sum for an empty file is found (instead of a warning)
  • Don't skip the md5 when generating a test file if an empty file is found (so that linting fails and can be manually checked)
• Linting checks test files for TODO statements as well as the main module code (#1271)
• Handle error if manifest isn't set in nextflow.config (#1418)

v2.2 - Lead Liger

Template

• Update repo logos to utilize GitHub's #gh-light/dark-mode-only, to switch between logos optimized for light or dark themes. The old repo logos have to be removed (in docs/images and assets/).
• Deal with authentication with private repositories
• Bump minimun Nextflow version to 21.10.3
• Convert pipeline template to updated Nextflow DSL2 syntax
• Solve circular import when importing nf_core.modules.lint
• Disable cache in nf_core.utils.fetch_wf_config while performing test_wf_use_local_configs.
• Modify software version channel handling to support multiple software version emissions (e.g. from mulled containers), and multiple software versions.
• Update dumpsoftwareversion module to correctly report versions with trailing zeros.
• Remove params.hostnames from the pipeline template (#1304)
• Update .gitattributes to mark installed modules and subworkflows as linguist-generated (#1311)
• Adding support for Julia package environments to nextflow.config(#1317)
• New YAML issue templates for pipeline bug reports and feature requests, with a much richer interface (#1165)
• Update AWS test GitHub Actions to use v2 of nf-core/tower-action
• Post linting comment even when linting.yml fails
• Update CONTRIBUTION.md bullets to remove points related to scrape_software_versions.py
• Update AWS test to set Nextflow version to 21.10.3

General

• Made lint check for parameters defaults stricter [#992]
  • Default values in nextflow.config must match the defaults given in the schema (anything with { in, or in main.nf is ignored)
  • Defaults in nextflow.config must now match the variable type specified in the schema
  • If you want the parameter to not have a default value, use null
  • Strings set to false or an empty string in nextflow.config will now fail linting
• Bump minimun Nextflow version to 21.10.3
• Changed questionary ask() to unsafe_ask() to not catch KeyboardInterupts (#1237)
• Fixed bug in nf-core launch due to revisions specified with -r not being added to nextflow command. (#1246)
• Update regex in readme test of nf-core lint to agree with the pipeline template (#1260)
• Update 'fix' message in nf-core lint to conform to the current command line options. (#1259)
• Fixed bug in nf-core list when NXF_HOME is set
• Run CI test used to create and lint/run the pipeline template with minimum and latest edge release of NF (#1304)
• New YAML issue templates for tools bug reports and feature requests, with a much richer interface (#1165)
• Handle synax errors in Nextflow config nicely when running nf-core schema build (#1267)
• Erase temporary files and folders while performing Python tests (pytest)
• Remove base Dockerfile used for DSL1 pipeline container builds
• Run tests with Python 3.10
• #1363 Fix tools CI workflow nextflow versions.

Modules

• Fixed typo in module_utils.py.
• Fixed failing lint test when process section was missing from module. Also added the local failing tests to the warned section of the output table. (#1235)
• Added --diff flag to nf-core modules update which shows the diff between the installed files and the versions
• Update nf-core modules create help texts which were not changed with the introduction of the --dir flag
• Check if README is from modules repo
• Update module template to DSL2 v2.0 (remove functions.nf from modules template and updating main.nf (#1289)
• Substitute get process/module name custom functions in module main.nf using template replacement (#1284)
• Linting now fails instead of warning if a local copy of a module does not match the remote (#1313)
• Check test YML file for md5sums corresponding to empty files (#1302)
• Exit with an error if empty files are found when generating the test YAML file (#1302)

v2.1 - Zinc Zebra

Template

• Correct regex pattern for file names in nextflow_schema.json
• Remove . from nf-core/tools command examples
• Update Nextflow installation link in pipeline template (#1201)
• Command hostname is not portable [#1212]
• Changed how singularity and docker links are written in template to avoid duplicate links

General

• Changed names of some flags with -r as short options to make the flags more consistent between commands.

Modules

• Added consistency checks between installed modules and modules.json (#1199)
• Added support excluding or specifying version of modules in .nf-core.yml when updating with nf-core modules install --all (#1204)
• Created nf-core modules update and removed updating options from nf-core modules install
• Added missing function call to nf-core lint (#1198)
• Fix nf-core lint not filtering modules test when run with --key (#1203)
• Fixed nf-core modules install not working when installing from branch with -b (#1218)
• Added prompt to choose between updating all modules or named module in nf-core modules update
• Check if modules is installed before trying to update in nf-core modules update
• Verify that a commit SHA provided with --sha exists for install/update commands
• Add new-line to main.nf after bump-versions command to make ECLint happy

v2.0.1 - Palladium Platypus Junior

Template

• Critical tweak to add --dir declaration to nf-core lint GitHub Actions linting.yml workflow

General

• Add --dir declaration to nf-core sync GitHub Actions sync.yml workflow