Merge pull request #545 from nf-core/dev

Version 1.9 release

.github/.coveragerc

@@ -0,0 +1,2 @@
+[run]
+omit = nf_core/pipeline-template/*

.github/CONTRIBUTING.md

@@ -38,7 +38,7 @@ The HTML will then be generated in `docs/api/_build/html`.
 
 
 ## Tests
-When you create a pull request with changes, [Travis CI](https://travis-ci.com/) will run automatic tests.
+When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
 Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
 
 There are two types of tests that run:
@@ -54,11 +54,11 @@ python -m pytest .
 ```
 
 ### Lint Tests
-The nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to.
+nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to.
 To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
 
 The nf-core/tools repo itself contains the master template for creating new nf-core pipelines.
-Travis is set up to create a new pipeline from this template and then run the lint tests on it.
+Once you have created a new pipeline from this template GitHub Actions is automatically set up to run lint tests on it.
 This ensures that any changes we make to either the linting or the template stay in sync.
 You can replicate this process locally with the following commands:
 

.github/RELEASE_CHECKLIST.md

@@ -0,0 +1,18 @@
+## Before release
+
+1. Check issue milestones to see outstanding issues to resolve if possible or transfer to the milestones for the next release e.g. [`v1.9`](https://github.com/nf-core/tools/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.9)
+2. Create a PR to `dev` to bump the version in  `CHANGELOG.md` and `setup.py`.
+3. Make sure all CI tests are passing!
+4. Create a PR from `dev` to `master`
+5. Make sure all CI tests are passing again (additional tests are run on PRs to `master`)
+6. Request review (2 approvals required)
+7. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*.
+8. Merge the PR into `master`
+9. Once CI tests on commit have passed, create a new release copying the `CHANGELOG` for that release into the description section.
+
+## After release
+
+1. Check the automated template synchronisation has been triggered properly. This should automatically open PRs directly to individual pipeline repos with the appropriate changes to update the pipeline template.
+2. Check that the automatic `PyPi` deployment has worked: [pypi.org/project/nf-core](https://pypi.org/project/nf-core/)
+3. Check `BioConda` has an automated PR to bump the version, and merge. eg. [bioconda/bioconda-recipes #20065](https://github.com/bioconda/bioconda-recipes/pull/20065)
+4. Create a tools PR to `dev` to bump back to the next development version in  `CHANGELOG.md` and `setup.py`

.github/workflows/code-tests.yml

@@ -0,0 +1,49 @@
+name: Python tests
+# This workflow is triggered on pushes and PRs to the repository.
+on: [push, pull_request]
+
+jobs:
+  PythonLint:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+      name: Check out source-code repository
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v1
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install python dependencies
+      run: |
+        python -m pip install --upgrade pip pytest pytest-datafiles pytest-cov mock jsonschema
+        pip install .
+
+    - name: Install Nextflow
+      run: |
+        mkdir /tmp/nextflow
+        cd /tmp/nextflow
+        wget -qO- get.nextflow.io | bash
+        sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
+
+    - name: Lint with flake8
+      run: |
+        pip install flake8
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+
+    - name: Test with pytest
+      run: python3 -m pytest --color=yes --cov-report=xml --cov-config=.github/.coveragerc --cov=nf_core
+
+    - uses: codecov/codecov-action@v1
+      name: Upload code coverage report
+      with:
+        if: success()
+        token: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/create-lint-wf.yml

@@ -0,0 +1,31 @@
+name: Create a pipeline and lint it
+on: [push, pull_request]
+
+jobs:
+  MakeTestWorkflow:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        name: Check out source-code repository
+
+      - name: Set up Python 3.7
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.7
+
+      - name: Install python dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install .
+
+      - name: Install Nextflow
+        run: |
+          mkdir /tmp/nextflow
+          cd /tmp/nextflow
+          wget -qO- get.nextflow.io | bash
+          sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
+
+      - name: Run nf-core tools
+        run: |
+          nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
+          nf-core lint nf-core-testpipeline

.github/workflows/deploy-pypi.yml

@@ -0,0 +1,31 @@
+name: Publish nf-core to PyPI
+on:
+  release:
+    types: [published]
+
+jobs:
+  build-n-publish:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+        name: Check out source-code repository
+
+      - name: Set up Python 3.7
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.7
+
+      - name: Install python dependencies
+        run: |
+          python -m pip install --upgrade pip setuptools wheel
+          pip install .
+
+      - name: Build the distribution
+        run: python setup.py sdist bdist_wheel
+
+      - name: Publish dist to PyPI
+        if: github.repository == 'nf-core/tools'
+        uses: pypa/gh-action-pypi-publish@master
+        with:
+          password: ${{ secrets.pypi_password }}

.github/workflows/markdown-lint.yml

@@ -0,0 +1,18 @@
+name: Lint Markdown
+on: [push, pull_request]
+
+jobs:
+  Markdown:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: actions/setup-node@v1
+        with:
+          node-version: '10'
+
+      - name: Install markdownlint
+        run: npm install -g markdownlint-cli
+
+      - name: Run Markdownlint
+        run: markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml

.github/workflows/sync.yml

@@ -0,0 +1,29 @@
+name: Sync template
+on:
+  release:
+    types: [published]
+
+jobs:
+  sync-all:
+    name: Sync all pipelines
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+        name: Check out source-code repository
+
+      - name: Set up Python 3.7
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.7
+
+      - name: Install python dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install .
+
+      - name: Run synchronisation
+        if: github.repository == 'nf-core/tools'
+        env:
+          AUTH_TOKEN: ${{ secrets.nf_core_bot_auth_token }}
+        run: nf-core sync --all --username nf-core-bot --auth-token $AUTH_TOKEN

.github/workflows/tools-api-docs.yml

@@ -0,0 +1,41 @@
+name: nf-core/tools API docs
+on:
+  push:
+    branches: [master, dev]
+
+jobs:
+  build-n-publish:
+    name: Build and publish nf-core to PyPI
+    runs-on: ubuntu-18.04
+
+    steps:
+      - name: Check out source-code repository
+        uses: actions/checkout@v2
+
+      - name: Set up Python 3.7
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.7
+
+      - name: Install python dependencies
+        run: |
+          python -m pip install --upgrade pip Sphinx sphinxcontrib-napoleon
+          pip install .
+
+      - name: Build HTML docs
+        run: make --directory ./docs/api html
+
+      - name: Push docs to api-doc branch
+        if: github.repository == 'nf-core/tools'
+        run: |
+          git checkout --orphan api-doc
+          git rm -r --cache .
+          git config user.email "core@nf-co.re"
+          git config user.name "nf-core-bot"
+          git add docs
+          git commit --message "nf-core/tools docs build
+
+          $GITHUB_REF - $GITHUB_SHA
+          "
+          git remote add nf-core https://github.com/nf-core/tools.git
+          git push --force --set-upstream nf-core api-doc

.gitignore

@@ -2,7 +2,6 @@
 
 .coverage
 .pytest_cache
-docs/api/_build
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

CHANGELOG.md

@@ -1,5 +1,47 @@
 # nf-core/tools: Changelog
 
+## v1.9
+
+### Continuous integration
+
+* Travis CI tests are now deprecated in favor of GitHub Actions within the pipeline template.
+  * `nf-core bump-version` support has been removed for `.travis.yml`
+  * `nf-core lint` now fails if a `.travis.yml` file is found
+* Ported nf-core/tools Travis CI automation to GitHub Actions
+* Fixed the build for the nf-core/tools API documentation on the website
+
+### Template
+
+* Rewrote the documentation markdown > HTML conversion in Python instead of R
+* Removed the requirement for R in the conda environment
+* Make `params.multiqc_config` give an _additional_ MultiQC config file instead of replacing the one that ships with the pipeline
+* Ignore only `tests/` and `testing/` directories in `.gitignore` to avoid ignoring `test.config` configuration file
+* Rephrase docs to promote usage of containers over Conda to ensure reproducibility
+* Stage the workflow summary YAML file within MultiQC work directory
+
+### Linting
+
+* Removed linting for CircleCI
+* Allow any one of `params.reads` or `params.input` or `params.design` before warning
+* Added whitespace padding to lint error URLs
+* Improved documentation for lint errors
+* Allow either `>=` or `!>=` in nextflow version checks (the latter exits with an error instead of just warning) [#506](https://github.com/nf-core/tools/issues/506)
+* Check that `manifest.version` ends in `dev` and throw a warning if not
+  * If running with `--release` check the opposite and fail if not
+* Tidied up error messages and syntax for linting GitHub actions branch tests
+* Add YAML validator
+* Don't print test results if we have a critical error
+
+### Other
+
+* Fix automatic synchronisation of the template after releases of nf-core/tools
+* Improve documentation for installing `nf-core/tools`
+* Replace preprint by the new nf-core publication in Nature Biotechnology :champagne:
+* Use `stderr` instead of `stdout` for header artwork
+* Tolerate unexpected output from `nextflow config` command
+* Add social preview image
+* Added a [release checklist](.github/RELEASE_CHECKLIST.md) for the tools repo
+
 ## v1.8
 
 ### Continuous integration
@@ -18,6 +60,7 @@
 * Updated documentation for `nf-core download`
 * Fixed typo in `nf-core launch` final command
 * Handle missing pipeline descriptions in `nf-core list`
+* Migrate tools package CI to GitHub Actions
 
 ### Linting
 
@@ -113,7 +156,7 @@
   to avoid weird behavior such as making an `s3:/` directory locally when using
   an AWS S3 bucket as the `--outdir`.
 * Fix workflow.onComplete() message when finishing pipeline
-* Update URL for joining the nf-core slack to https://nf-co.re/join/slack
+* Update URL for joining the nf-core slack to [https://nf-co.re/join/slack](https://nf-co.re/join/slack)
 * Add GitHub Action for CI and Linting
 * [Increased default time limit](https://github.com/nf-core/tools/issues/370) to 4h
 * Add direct link to the pipeline slack channel in the contribution guidelines
@@ -124,7 +167,7 @@
 * Template configured to use logo fetched from website
 * New option `--email_on_fail` which only sends emails if the workflow is not successful
 * Add file existence check when checking software versions
-* Fixed issue [https://github.com/nf-core/tools/issues/165] - Use `checkIfExists`
+* Fixed issue [#165](https://github.com/nf-core/tools/issues/165) - Use `checkIfExists`
 * Consistent spacing for `if` statements
 * Add sensible resource labels to `base.config`