Skip to content

Commit

Permalink
ci: add treefmt as code formatting multiplexer, refactor CI to avoid …
Browse files Browse the repository at this point in the history
…duplication, reorg CI into DevOps workflow (#4219)

* ci: add treefmt as code formatting multiplexer

* style: linter findings in entrypoint.sh script

* style: linter findings for yaml and yml

* style: linter findings for json files

* style: linter findings for nix files

* style: linter findings for js files

* style: linter findings for dashboards.sh

* style: linter findings for md and mdx files

* chore: remove unused clang formatter in treefmt config

* style: linter findings for mdx files

* ci: exclude Grafana dashboard JSON files from prettier formatting

* Revert "style: linter findings for json files"

This reverts commit f40c2e1.

* ci: exclude Grafana dashboard JSON files from all formatting as we use the grafana export style

* style: linter findings for json files

* doc: update changelog

* ci(refactor): use composite action to avoid duplication in elixir workflow

* doc: update changelog

* ci: prevent workflow runs for certain conditions and allow scheduled runs

* ci(refactor): use reusable workflow to check paths

* ci(fix): correct output syntax for check_paths workflow and setting base branch

* ci(refactor): use reusable workflows for streamlined DevOps pipeline

* ci(fix): add write permission for packages in DevOps workflow

* ci(test): test DevOps workflow

* ci(test): test DevOps workflow

* ci(fix): Update condition for spell_check, ensure_linting, elixir, and ghcr_build workflows to reflect empty result instead of false

* ci: revert test DevOps

* ci(refactor): allow ghcr_build parallel to elixir test

* ci(refactor): Remove redundant check_paths job from elixir.yml, elixir_test.yml, and spell_check.yml workflows, check is done in devops.yml

* feat: add treefmt-nix to nix flake (#4219 - @JakobLichterfeld)

* ci: ensure proper linting via treefmt

* ci(test): test ensure_linting workflow

* ci(fix): checkout code for spell_checker to access file to check

* ci(fix): allow impure in ensure_linting workflow

* Revert "ci(test): test ensure_linting workflow"

This reverts commit a67b17e.

* ci(fix): correct use of flake-utils for formatter and checks

Co-authored-by: scottbot95 <scottbot95@gmail.com>

* ci(fix): correct use of flake-utils for treefmt

Co-authored-by: scottbot95 <scottbot95@gmail.com>

* refactor: Remove unnecessary imports in flake

* ci(fix): correct syntax in flake

* ci(refactor): Remove unused code in flake.nix

* style: standardised style for input url in flake

* ci(fix): treefmt-nix config with existing options

* ci(feat): Add Nix binary cache and update treefmt command in CI workflow

* ci(refactor): Remove unused code in flake.nix

* fix: include devShell packages only on supported platforms

* fix: update hash for mix-deps package in flake.nix

* ci(fix): Update treefmt command in CI workflow

* ci(test): test ensure_linting workflow

* feat: ensure mix deps are present in devShell

* ci(feat): use flake-parts to enable treefmt-nix

* feat: use flake-parts

* fix: correct use of flake-parts for package build

* doc: update CI badge URL for devops workflow

* ci(fix): handle empty path filter output

* ci: remove --impure flag from treefmt command in CI workflow

* fix: correct treefmt.config settings in formatter.nix

* fix: correct flake-parts inputs, avoid with in imports

* fix: correct program name for mix-format in formatter.nix

* feat: devenv via flake-parts

* fix: correct use of legacy nix code with flake-parts

* ci(fix): correct nix develop command in ensure_linting.yml

* refactor: list imports explicitly in flake, rename folder to flake-modules to be precise

* style: use tabs for indent size to format sh

* style: use nixfmt-rfc-style

* Revert "style: use nixfmt-rfc-style"

This reverts commit 0820561.

* style: use nixfmt style

* fix: remove glibcLocales from optional dependencies to avoid  "A definition for option `packages."[definition 4-entry 16]"' is not of type `package'."

* fix: remove inotify-tools from optional dependencies to avoid  "A definition for option `packages."[definition 4-entry 16]"' is not of type `package'."

* fix: Remove inotify-tools and glibcLocales from optional dependencie

* fix: correct file paths in flake.nix to version

* fix: add ELIXIR_ERL_OPTIONS to shell environment to force utf8 locale

* fix: add LOCALE_ARCHIVE to shell environment in flake.nix

* Revert "fix: add LOCALE_ARCHIVE to shell environment in flake.nix"

This reverts commit d45f6e3.

* ci(refactor): rename workflow to elixir_dep_verification_and_static_analysis.yml to better reflect the intention,, remove duplicate checks

* ci(debug): debug locale settings

* Revert "ci(debug): debug locale settings"

This reverts commit 9b402f3.

* Revert "fix: add ELIXIR_ERL_OPTIONS to shell environment to force utf8 locale"

This reverts commit d02419c.

* fix: add LOCALE_ARCHIVE to shell environment in flake.nix

* Revert "fix: add LOCALE_ARCHIVE to shell environment in flake.nix"

This reverts commit 761b437.

* fix: add LANG=C.UTF-8 to shell environment in flake.nix

* fix: add mix local.rebar and mix local.hex commands to flake.nix

* fix: pin devenv to version without unix socket bug

* chore: update nixpkgs to nixos-24.05 and update dependencies

* doc: add treefmt config comments

* ci: do not expose treefmt formatter programs in devshell

* fix: correct use of module option to enable PostgreSQL server in flake.nix

* Revert "chore: update nixpkgs to nixos-24.05 and update dependencies"

This reverts commit a6ea3f2.

* feat: consistent use of erlang 26 and elixir 1_16 in flake

* ci: switch to macOS runner for linting workflow

* Revert "ci: do not expose treefmt formatter programs in devshell"

This reverts commit 1ecfa45.

* Revert "ci: switch to macOS runner for linting workflow"

This reverts commit 7b43066.

* ci: Remove nixpkgs channel specification in ensure_linting workflow

* ci(debug): Add debug output for PATH and NIX_PATH in flake.nix

* Revert "ci(debug): Add debug output for PATH and NIX_PATH in flake.nix"

This reverts commit 07faec5.

* fix: avoid the need for impure for devenv

see #4245

* fix: remove invalid custom build.check for formatter and use default

* style: linter findings

* fix: Add emptyTest to avoid nix flake check test execution on non-Linux systems

* chore: Remove LANG=C.UTF-8 from enterShell in flake.nix

* ci(fix): Remove --impure flag from treefmt command in CI mode

* ci(fix): avoid impure mode in ensure_linting workflow

* style: linter findings

* ci(debug): debug elixir version and locale

* chore: Update flake.lock dependencies

* feat: use newer devenv as unix socket bug is fixed in upstream

cachix/devenv#1497

* fix: set rebar3 path in devenv

* Revert "ci(debug): debug elixir version and locale"

This reverts commit 7ecdc77.

* ci: re-enable path check in DevOps workflow

* doc: update Development and Contributing guide with nix and treefmt

* ci: use PostgreSQL 17

* style: linter findings

* ci(fix): ensure cache name in build action does not contain invalid characters to avoid invalid reference format

* doc: update changelog

* Revert "ci(fix): ensure cache name in build action does not contain invalid characters to avoid invalid reference format"

This reverts commit 02abb03.

* ci: remove branch restriction for check_paths workflow to increase sec

* ci(fix): run ghcr build workflow only for specific conditions

- Ensure workflow only run if there are no changes to the .github folder
- Allow workflow to run on workflow call or PRs from forks
- Prevent duplicate runs for PRs from non-forks
- Avoid invalid reference format for cache name in PRs from our repository

* doc: update changelog

* fix: update hash for mix-deps package in flake.nix

* fix: disable flakeCheck for formatter, as mix format need the dep to be fetched beforehand

* ci(fix): run ghcr build workflow only for specific conditions

* fix: move nixosModules.default to top-level attribute set

* refactor: remove unnecessary config nesting in formatter.nix

* ci(fix): ensure version for buildx is set to correct name

---------

Co-authored-by: scottbot95 <scottbot95@gmail.com>
  • Loading branch information
JakobLichterfeld and scottbot95 committed Oct 19, 2024
1 parent dfd8ed8 commit 06f5f6d
Show file tree
Hide file tree
Showing 41 changed files with 1,508 additions and 1,028 deletions.
2 changes: 1 addition & 1 deletion .github/actions/build/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ inputs:
is_dockerhub_pushed:
description: "Need docker hub login?"
required: true
default: ''
default: ""
docker_password:
description: "Docker password"
required: true
Expand Down
75 changes: 75 additions & 0 deletions .github/actions/setup-elixir-and-cache-deps/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: "Setup Elixir and Cache Dependencies"
description: "Setup Elixir, OTP and cache dependencies"
inputs:
elixir-version:
description: "Elixir version"
required: false
default: "1.16.2"
otp-version:
description: "OTP version"
required: false
default: "26"
cache-name-deps:
description: "Cache name for dependencies"
required: true
cache-name-compiled:
description: "Cache name for compiled build"
required: true
mix-env:
description: "Mix environment"
required: false
default: "dev"
ELIXIR_ASSERT_TIMEOUT:
description: "Elixir assert timeout"
required: false
default: "1000"
outputs:
elixir-version:
description: "The Elixir version used in the setup"
value: ${{ steps.beam.outputs.elixir-version }}
otp-version:
description: "The OTP version used in the setup"
value: ${{ steps.beam.outputs.otp-version }}
runs:
using: "composite"
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7

- name: Setup Elixir and OTP
id: beam
uses: erlef/setup-beam@b9c58b0450cd832ccdb3c17cc156a47065d2114f # v1.18.1
with:
elixir-version: ${{ inputs.elixir-version }}
otp-version: ${{ inputs.otp-version }}

- name: Cache deps
id: cache-deps
uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
with:
path: deps
key: ${{ runner.os }}-mix-${{ inputs.cache-name-deps }}-${{ hashFiles('**/mix.lock') }}
restore-keys: |
${{ runner.os }}-mix-${{ inputs.cache-name-deps }}-
- name: Cache compiled build
id: cache-build
uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2
with:
path: |
_build
priv/cldr/locales
key: ${{ runner.os }}-mix-${{ inputs.cache-name-compiled }}-${{ hashFiles('**/mix.lock') }}
restore-keys: |
${{ runner.os }}-mix-${{ inputs.cache-name-compiled }}-
${{ runner.os }}-mix-
- name: Clean to rule out incremental build as a source of flakiness
if: github.run_attempt > 3
run: |
mix deps.clean --all
mix clean
shell: sh

- name: Install dependencies
run: mix deps.get
shell: sh
23 changes: 7 additions & 16 deletions .github/workflows/buildx.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: Publish Docker images

on:
workflow_dispatch:
workflow_call:
schedule:
- cron: "0 3 * * *"
push:
Expand All @@ -10,7 +11,8 @@ on:
paths:
- "**/*"
- "!.github/**" # Important: Exclude PRs related to .github from auto-run
- "!.github/workflows/**" # Important: Exclude PRs related to .github from auto-run
- "!.github/workflows/**" # Important: Exclude PRs related to .github/workflows from auto-run
- "!.github/actions/**" # Important: Exclude PRs related to .github/actions from auto-run
env:
REGISTRY_IMAGE: teslamate/teslamate

Expand All @@ -20,22 +22,11 @@ permissions:

jobs:
check_paths:
runs-on: ubuntu-latest
outputs:
githubfolder: ${{ steps.filter.outputs.githubfolder }}
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7

- uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
id: filter
with:
filters: |
githubfolder:
- '.github/**'
uses: ./.github/workflows/check_paths.yml

teslamate_build:
needs: check_paths
if: needs.check_paths.outputs.githubfolder == 'false' || github.event_name == 'schedule'
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -71,7 +62,7 @@ jobs:
needs:
- check_paths
- teslamate_build
if: needs.check_paths.outputs.githubfolder == 'false' || github.event_name == 'schedule'
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
steps:
- name: Checkout
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
Expand All @@ -92,7 +83,7 @@ jobs:
type=edge
grafana:
needs: check_paths
if: needs.check_paths.outputs.githubfolder == 'false' || github.event_name == 'schedule'
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
Expand Down
41 changes: 41 additions & 0 deletions .github/workflows/check_paths.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Check paths

on:
workflow_call:
# Map the workflow outputs to job outputs
outputs:
githubfolder:
description: "changes to .github folder"
value: ${{ jobs.check_paths.githubfolder }}
push:
paths:
- "**/*"
- "!.github/**" # Important: Exclude PRs related to .github from auto-run
- "!.github/workflows/**" # Important: Exclude PRs related to .github/workflows from auto-run
- "!.github/actions/**" # Important: Exclude PRs related to .github/actions from auto-run
pull_request_target:
branches: ["master"]
paths:
- "**/*"
- "!.github/**" # Important: Exclude PRs related to .github from auto-run
- "!.github/workflows/**" # Important: Exclude PRs related to .github/workflows from auto-run
- "!.github/actions/**" # Important: Exclude PRs related to .github/actions from auto-run

permissions:
contents: read

jobs:
check_paths:
runs-on: ubuntu-latest
outputs:
githubfolder: ${{ steps.filter.outputs.githubfolder }}
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7

- uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
id: filter
with:
base: "master" # needed to set as a called workflow does not have direct access to repository.default_branch
filters: |
githubfolder:
- '.github/**'
58 changes: 58 additions & 0 deletions .github/workflows/devops.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: DevOps

on:
workflow_dispatch:
push:
paths:
- "**/*"
- "!.github/**" # Important: Exclude PRs related to .github from auto-run
- "!.github/workflows/**" # Important: Exclude PRs related to .github/workflows from auto-run
- "!.github/actions/**" # Important: Exclude PRs related to .github/actions from auto-run
pull_request:
branches: ["master"]
paths:
- "**/*"
- "!.github/**" # Important: Exclude PRs related to .github from auto-run
- "!.github/workflows/**" # Important: Exclude PRs related to .github/workflows from auto-run
- "!.github/actions/**" # Important: Exclude PRs related to .github/actions from auto-run

permissions:
contents: read
packages: write

jobs:
check_paths:
uses: ./.github/workflows/check_paths.yml

spell_check:
needs: check_paths
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
uses: ./.github/workflows/spell_check.yml

ensure_linting:
needs:
- check_paths
- spell_check
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
uses: ./.github/workflows/ensure_linting.yml

elixir_dep_verification_and_static_analysis:
needs:
- check_paths
- ensure_linting
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
uses: ./.github/workflows/elixir_dep_verification_and_static_analysis.yml

elixir_test:
needs:
- check_paths
- ensure_linting
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
uses: ./.github/workflows/elixir_test.yml

ghcr_build:
needs:
- check_paths
- elixir_dep_verification_and_static_analysis
if: needs.check_paths.outputs.githubfolder != 'true' || github.event_name == 'schedule'
uses: ./.github/workflows/ghcr_build.yml
Loading

0 comments on commit 06f5f6d

Please sign in to comment.