diff --git a/VERSION b/VERSION
index f25af7f1..b6db450d 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-${yyyy}${mm}${dd}
+0.9.0-${snapshot}
diff --git a/docs/adr/ADR-004_Agree_CICD_pipeline_structure.md b/docs/adr/ADR-004_Agree_CICD_pipeline_structure.md
new file mode 100644
index 00000000..6e6caadf
--- /dev/null
+++ b/docs/adr/ADR-004_Agree_CICD_pipeline_structure.md
@@ -0,0 +1,89 @@
+# ADR-004: Agree CI/CD pipeline structure
+
+>| | |
+>| ------------ | --- |
+>| Date | `15/09/2022` |
+>| Status | `RFC` |
+>| Deciders | `Engineering` |
+>| Significance | `Construction techniques` |
+>| Owners | `Dan Stefaniuk, Nick Sparks` |
+
+---
+
+- [ADR-004: Agree CI/CD pipeline structure](#adr-004-agree-cicd-pipeline-structure)
+ - [Context](#context)
+ - [Decision](#decision)
+ - [Assumptions](#assumptions)
+ - [Drivers](#drivers)
+ - [Options](#options)
+ - [Outcome](#outcome)
+ - [Rationale](#rationale)
+ - [Consequences](#consequences)
+ - [Compliance](#compliance)
+ - [Tags](#tags)
+
+## Context
+
+Continuous integration and continuous delivery pipeline is to organise all steps required to go from idea to a releasable software using automation of the development process. The key ideas upon it is founded are as follows:
+
+- The reliable, repeatable production of high quality software.
+- The application of scientific principles, experimentation, feedback and learning.
+- The pipeline (or set of workflows) as a mechanism to organise and automate the development process.
+
+For this to work it is essential to apply principles and practices noted in the [NHSE Software Engineering Quality Framework](https://github.com/NHSDigital/software-engineering-quality-framework)
+
+Requirements:
+
+- Implement the exemplar CI/CD pipeline using GitHub workflows and actions
+- Incorporate the four main CI/CD stages, which are as follows:
+ 1. Commit, max. execution time 2 mins
+ 2. Test, max. execution time 5 mins
+ 3. Build, max. execution time 3 mins
+ 4. Acceptance, max. execution time 10 mins
+- Provide `publish`, `deploy` and `rollback` workflows as the complementary processes
+- Maintain simplicity in the pipeline but ensure it is scalable and extensible for larger projects
+- Enable parallel execution of jobs to speed up the overall process
+- Prevent the workflow from being triggered twice, i.e. when pushing to a branch with an existing pull request
+- Implement good CI/CD practices, such as:
+ - Setting the build time variables at the start of the process
+ - Storing the tooling versions like Terraform, Python and Node.js in the `./.tools-version` file (dependency management)
+ - Storing the software/project version in the `VERSION` file at the project root-level or in an artifact directory
+ - Keeping the main workflow modular
+ - Ensuring a timeout is set for each job
+ - Listing environment variables
+ - Making actions portable, e.g. allowing them to be run on a workstation or Azure DevOps using scripts
+ - Providing testable CI/CD building blogs
+
+## Decision
+
+### Assumptions
+
+TODO: state the assumptions
+
+### Drivers
+
+TODO: list the drivers
+
+### Options
+
+TODO: table, SEE: the [CI/CD pipeline](../developer-guides/CICD_pipeline.md) high-level design.
+
+### Outcome
+
+TODO: decision outcome
+
+### Rationale
+
+TODO: rationale
+
+## Consequences
+
+TODO: consequences
+
+## Compliance
+
+TODO: how the success is going to be measured
+
+## Tags
+
+`#maintainability, #testability, #deployability, #modularity, #simplicity, #reliability`
diff --git a/docs/developer-guides/CICD_pipeline.md b/docs/developer-guides/CICD_pipeline.md
new file mode 100644
index 00000000..727fc8ea
--- /dev/null
+++ b/docs/developer-guides/CICD_pipeline.md
@@ -0,0 +1,269 @@
+# Developer Guide: CI/CD pipeline
+
+- [Developer Guide: CI/CD pipeline](#developer-guide-cicd-pipeline)
+ - [The pipeline high-level workflow model](#the-pipeline-high-level-workflow-model)
+ - [Workflow stages](#workflow-stages)
+ - [End-to-end workflow stages](#end-to-end-workflow-stages)
+ - [Stage triggers](#stage-triggers)
+ - [Branch review workflow](#branch-review-workflow)
+ - [PR review workflow](#pr-review-workflow)
+ - [Publish workflow](#publish-workflow)
+ - [Deploy workflow](#deploy-workflow)
+ - [Rollback workflow](#rollback-workflow)
+ - [Environments and artefact promotion](#environments-and-artefact-promotion)
+ - [Resources](#resources)
+
+## The pipeline high-level workflow model
+
+```mermaid
+flowchart LR
+ Review --> Publish
+ Publish --> Deploy
+ Deploy --> Rollback
+```
+
+## Workflow stages
+
+### End-to-end workflow stages
+
+```mermaid
+flowchart LR
+ commit_local["Commit
(local githooks)"] --> commit_remote
+ commit_remote["Commit
(remote)"] --> Test
+ Test --> Build
+ Build --> Acceptance
+ Acceptance --> Publish
+ Publish --> Deploy
+ Deploy --> Rollback
+```
+
+### Stage triggers
+
+| Workflow | Stage | `main` branch trigger | Task branch trigger |
+|---------:|:------------------------|:---------------------------:|:----------------------:|
+| Review | Commit (local githooks) | - | on commit |
+| Review | Commit (remote) | on merge | on push |
+| Review | Test | on merge | on push |
+| Review | Build | on merge | on push, if PR is open |
+| Review | Acceptance | on merge | on push, if PR is open |
+| Publish | Publish | on tag | - |
+| Deploy | Deploy | on tag | - |
+| Rollback | Rollback | on demand or on healthcheck | - |
+
+- Publish:
+ - When merged, create snapshot release
+ - When tagged, crate Release Candidate
+- Deploy
+ - Only deploy RCs
+ - Deploy to specified environment
+
+### Branch review workflow
+
+```mermaid
+flowchart LR
+ subgraph commit_local["Commit (local githooks)"]
+ direction TB
+ clA["Scan secrets"]
+ clB["Check file format"]
+ clC["Check markdown format"]
+ clD["Check Terraform format"]
+ clE["Scan dependencies"]
+ clA --> clB
+ clB --> clC
+ clC --> clD
+ clD --> clE
+ end
+ subgraph commit_remote["Commit (remote)"]
+ direction TB
+ crA["Scan secrets"]
+ crB["Check file format"]
+ crC["Check markdown format"]
+ crD["Lint Terraform"]
+ crE["Count lines of code"]
+ crF["Scan dependencies"]
+ crA -.- crB
+ crB -.- crC
+ crC -.- crD
+ crD -.- crE
+ crE -.- crF
+ end
+ subgraph test[Test]
+ direction TB
+ tA["Linting"]
+ tB["Unit tests"]
+ tC["Test coverage"]
+ tD["Perform static analysis"]
+ tA -.- tB
+ tB --> tC
+ tB --> tD
+ end
+ subgraph branch_review["Branch review"]
+ direction LR
+ commit_local --> commit_remote
+ commit_remote --> test
+ end
+ branch_review --> build
+ build["Build"] --> acceptance
+ acceptance["Acceptance"] --> publish
+ publish["Publish"] --> deploy
+ deploy["Deploy"] --> rollback["Rollback"]
+```
+
+### PR review workflow
+
+```mermaid
+flowchart LR
+ subgraph commit_remote["Commit (remote)"]
+ direction TB
+ crA["Scan secrets"]
+ crB["Check file format"]
+ crC["Check markdown format"]
+ crD["Lint Terraform"]
+ crE["Count lines of code"]
+ crF["Scan dependencies"]
+ crA -.- crB
+ crB -.- crC
+ crC -.- crD
+ crD -.- crE
+ crE -.- crF
+ end
+ subgraph test["Test"]
+ direction TB
+ tA["Linting"]
+ tB["Unit tests"]
+ tC["Test coverage"]
+ tD["Perform static analysis"]
+ tA -.- tB
+ tB --> tC
+ tB --> tD
+ end
+ subgraph build["Build"]
+ direction TB
+ bA["Artefact (back-end)"]
+ bB["Artefact (front-end)"]
+ bA -.- bB
+ end
+ subgraph acceptance["Acceptance"]
+ direction TB
+ aA["Environment set up"]
+ aB["Contract test"]
+ aC["Security test"]
+ aD["UI test"]
+ aE["UI performance test"]
+ aF["Integration test"]
+ aG["Accessibility test"]
+ aH["Load test"]
+ aI["Environment tear down"]
+ aA --> aB
+ aA --> aC
+ aA --> aD
+ aA --> aE
+ aA --> aF
+ aA --> aG
+ aA --> aH
+ aB --> aI
+ aC --> aI
+ aD --> aI
+ aE --> aI
+ aF --> aI
+ aG --> aI
+ aH --> aI
+ end
+ subgraph pr_review["PR review"]
+ direction LR
+ commit_remote --> test
+ test --> build
+ build --> acceptance
+ end
+ branch_review["Branch review"] --> pr_review
+ pr_review --> publish
+ publish["Publish"] --> deploy
+ deploy["Deploy"] --> rollback["Rollback"]
+```
+
+### Publish workflow
+
+```mermaid
+flowchart LR
+ subgraph publish["Publish"]
+ direction TB
+ pA["Set CI/CD metadata"]
+ pB["Publish artefacts"]
+ pC["Send notification"]
+ pA --> pB
+ pB --> pC
+ end
+ branch_review["Branch review"] --> pr_review
+ pr_review["PR review"] --> publish
+ publish --> deploy
+ deploy["Deploy"] --> rollback["Rollback"]
+```
+
+### Deploy workflow
+
+```mermaid
+flowchart LR
+ subgraph deploy["Deploy"]
+ direction TB
+ dA["Set CI/CD metadata"]
+ dB["Deploy to an environment"]
+ dC["Send notification"]
+ dA --> dB
+ dB --> dC
+ end
+ branch_review["Branch review"] --> pr_review
+ pr_review["PR review"] --> publish
+ publish["Publish"] --> deploy
+ deploy --> rollback["Rollback"]
+```
+
+### Rollback workflow
+
+```mermaid
+flowchart LR
+ subgraph rollback["Rollback"]
+ direction TB
+ dA["Set CI/CD metadata"]
+ dB["Rollback an environment"]
+ dC["Send notification"]
+ dA --> dB
+ dB --> dC
+ end
+ branch_review["Branch review"] --> pr_review
+ pr_review["PR review"] --> publish
+ publish["Publish"] --> deploy
+ deploy["Deploy"] --> rollback
+```
+
+## Environments and artefact promotion
+
+```mermaid
+flowchart LR
+ subgraph branch_review["Branch review"]
+ direction LR
+ bA("local")
+ end
+ subgraph pr_review["PR Review"]
+ direction LR
+ prA["ephemeral
dev environments"]
+ prB["automated acceptance
test environments"]
+ prA --> prB
+ end
+ subgraph deploy1["Deploy (high-instance)"]
+ direction LR
+ d1A["non-prod
environments"]
+ end
+ subgraph deploy2["Deploy (Live)"]
+ direction LR
+ d2A["prod environment"]
+ end
+ branch_review --> pr_review
+ pr_review --> deploy1
+ deploy1 --> deploy2
+```
+
+## Resources
+
+- Blog post [Going faster with continuous delivery](https://aws.amazon.com/builders-library/going-faster-with-continuous-delivery/)
+- Blog post [Automating safe, hands-off deployments](https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/)
+- Book [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://www.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/)
diff --git a/scripts/config/repository-template.yaml b/scripts/config/repository-template.yaml
index eb37ceec..984fa0e2 100644
--- a/scripts/config/repository-template.yaml
+++ b/scripts/config/repository-template.yaml
@@ -1 +1,19 @@
update-from-template:
+ modules: [ "terraform", "docker", "tests", "githooks", "reports", "config" ]
+ ignore:
+
+cicd-config:
+ stage:
+ build:
+ only-when-pr-present: true
+ acceptance:
+ only-when-pr-present: true
+
+version:
+ build-datetime: 2023-02-21T10:46:17+0000
+ template:
+ url: https://github.com/nhs-england-tools/repository-template
+ branch: main
+ commit-hash: 94834f9ecd87d96b6f43a6f0307f2e7ee905b1b4
+ tags: []
+ release-notes-url:
diff --git a/scripts/docker/docker.lib.sh b/scripts/docker/docker.lib.sh
index 18787105..f8cc3473 100644
--- a/scripts/docker/docker.lib.sh
+++ b/scripts/docker/docker.lib.sh
@@ -124,21 +124,8 @@ function docker-clean() {
function version-create-effective-file() {
local dir=${dir:-$PWD}
- local version_file="$dir/VERSION"
- local build_datetime=${BUILD_DATETIME:-$(date -u +'%Y-%m-%dT%H:%M:%S%z')}
- if [ -f "$version_file" ]; then
- # shellcheck disable=SC2002
- cat "$version_file" | \
- sed "s/\(\${yyyy}\|\$yyyy\)/$(date --date="${build_datetime}" -u +"%Y")/g" | \
- sed "s/\(\${mm}\|\$mm\)/$(date --date="${build_datetime}" -u +"%m")/g" | \
- sed "s/\(\${dd}\|\$dd\)/$(date --date="${build_datetime}" -u +"%d")/g" | \
- sed "s/\(\${HH}\|\$HH\)/$(date --date="${build_datetime}" -u +"%H")/g" | \
- sed "s/\(\${MM}\|\$MM\)/$(date --date="${build_datetime}" -u +"%M")/g" | \
- sed "s/\(\${SS}\|\$SS\)/$(date --date="${build_datetime}" -u +"%S")/g" | \
- sed "s/\(\${hash}\|\$hash\)/$(git rev-parse --short HEAD)/g" \
- > "$dir/.version"
- fi
+ file="$dir/VERSION" output_file="$dir/.version" _replace-version-placeholders
}
# ==============================================================================
@@ -220,7 +207,6 @@ function _replace-image-latest-by-specific-version() {
local dir=${dir:-$PWD}
local versions_file="${TOOL_VERSIONS:=$(git rev-parse --show-toplevel)/.tool-versions}"
local dockerfile="${dir}/Dockerfile.effective"
- local build_datetime=${BUILD_DATETIME:-$(date -u +'%Y-%m-%dT%H:%M:%S%z')}
if [ -f "$versions_file" ]; then
# First, list the entries specific for Docker to take precedence, then the rest but exclude comments
@@ -234,9 +220,26 @@ function _replace-image-latest-by-specific-version() {
done
fi
- if [ -f "$dockerfile" ]; then
+ file="$dockerfile" _replace-version-placeholders
+ # Do not ignore the issue if 'latest' is used in the effective image and ensure the ignore rule is removed, if present]
+ sed -Ei "/#.*hadolint.*ignore=DL3007$/d" "$dockerfile"
+}
+
+# Replace version placeholders.
+# Arguments (provided as environment variables):
+# file=[path to the file to use as the input]
+# output_file=[path to the file to use as the output, default is the same as the input file]
+# BUILD_DATETIME=[build date and time in the '%Y-%m-%dT%H:%M:%S%z' format generated by the CI/CD pipeline, default is current date and time]
+function _replace-version-placeholders() {
+
+ local file="$file"
+ local output_file="${output_file:-$file}"
+ local build_datetime=${BUILD_DATETIME:-$(date -u +'%Y-%m-%dT%H:%M:%S%z')}
+
+ if [ -f "$file" ]; then
# shellcheck disable=SC2002
- cat "$dockerfile" | \
+ cat "$file" | \
+ sed "s/\(\${snapshot}\|\$snapshot\)/$(date --date="${build_datetime}" -u +"%Y%m%d.%H%M%S")/g" | \
sed "s/\(\${yyyy}\|\$yyyy\)/$(date --date="${build_datetime}" -u +"%Y")/g" | \
sed "s/\(\${mm}\|\$mm\)/$(date --date="${build_datetime}" -u +"%m")/g" | \
sed "s/\(\${dd}\|\$dd\)/$(date --date="${build_datetime}" -u +"%d")/g" | \
@@ -244,12 +247,9 @@ function _replace-image-latest-by-specific-version() {
sed "s/\(\${MM}\|\$MM\)/$(date --date="${build_datetime}" -u +"%M")/g" | \
sed "s/\(\${SS}\|\$SS\)/$(date --date="${build_datetime}" -u +"%S")/g" | \
sed "s/\(\${hash}\|\$hash\)/$(git rev-parse --short HEAD)/g" \
- > "$dockerfile.tmp"
- mv "$dockerfile.tmp" "$dockerfile"
+ > "$file.tmp"
+ mv "$file.tmp" "$output_file"
fi
-
- # Do not ignore the issue if 'latest' is used in the effective image
- sed -Ei "/# hadolint ignore=DL3007$/d" "${dir}/Dockerfile.effective"
}
# Append metadata to the end of Dockerfile.