Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch transient CI failures due to busy file system #214

Open
wants to merge 12 commits into
base: develop
Choose a base branch
from
Open

Patch transient CI failures due to busy file system #214

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
aa4aeff
Add patch for IO in the CI
tim-griesbach Jan 15, 2025
678d541
CI: Apply patch for IO
tim-griesbach Jan 15, 2025
eb190eb
CI: Increase IO sleeping time
tim-griesbach Jan 15, 2025
04971f1
Fix typo
tim-griesbach Jan 15, 2025
4b2d334
CI: IO_SLEEP_TIME -> SC_IO_SLEEP_TIME
tim-griesbach Jan 16, 2025
4587e43
CI: Indent
tim-griesbach Jan 16, 2025
b4afeb4
Implement a portable sleep function
tim-griesbach Jan 16, 2025
bcf0a91
CI: Update patch to use sc_sleep
tim-griesbach Jan 16, 2025
b35d283
CI: Increase IO sleeping time for Autotools
tim-griesbach Jan 16, 2025
3e1fa93
CI: Do not return inside of the macro
tim-griesbach Jan 16, 2025
66685aa
CI: Increase IO sleeping time
tim-griesbach Jan 16, 2025
dec97f1
CI: Increase to half second IO sleeping time
tim-griesbach Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 30 additions & 8 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ on:
- ".github/workflows/composite-cmake/**"
pull_request:

env:
# Due to busy file system problems on the CI runners, we introduced a patch to
# sleep if the MPI count of an I/O operation is unexpectedly 0 and then retry
# the I/O operation once. The time is in milliseconds.
IO_SLEEP_TIME: 5
cburstedde marked this conversation as resolved.
Show resolved Hide resolved

jobs:

linux-multi:
Expand All @@ -29,6 +35,9 @@ jobs:
- name: Checkout source code
uses: actions/checkout@main

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Run bootstrap script
run: ./bootstrap

Expand All @@ -37,23 +46,26 @@ jobs:
run: |
DIR="checkMPIdebug_shared" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-mpi --enable-debug --disable-shared \
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter"
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

- name: Make check with MPI, without debug
shell: bash
run: |
DIR="checkMPI" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-mpi CFLAGS="-O2"
../configure --enable-mpi CFLAGS="-O2" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
cburstedde marked this conversation as resolved.
Show resolved Hide resolved
make -j V=0
make -j check V=0

- name: Make check without MPI, with debug
shell: bash
run: |
DIR="checkdebug" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-debug CFLAGS="-O0 -g -Wall -Wno-uninitialized"
../configure --enable-debug CFLAGS="-O0 -g -Wall -Wno-uninitialized" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

Expand All @@ -66,7 +78,8 @@ jobs:
shell: bash
run: |
DIR="checkMPIdebugCXX" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-mpi --enable-debug CFLAGS="-O0" CC=mpicxx
../configure --enable-mpi --enable-debug CFLAGS="-O0" CC=mpicxx \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

Expand All @@ -75,15 +88,16 @@ jobs:
run: |
DIR="checkOpenMPMPIdebug" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-openmp="-fopenmp" --enable-mpi \
--enable-debug CFLAGS="-O0"
--enable-debug CFLAGS="-O0" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

- name: Make distcheck
shell: bash
run: |
DIR="distcheck" && mkdir -p "$DIR" && cd "$DIR"
../configure
../configure CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j distcheck V=0

- name: Upload log files
Expand All @@ -110,6 +124,9 @@ jobs:
- name: Checkout source code
uses: actions/checkout@main

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Run bootstrap script
run: ./bootstrap

Expand All @@ -119,7 +136,8 @@ jobs:
DIR="checkMPIdebug_valgrind" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-mpi --enable-debug \
--disable-shared --enable-valgrind \
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter"
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

Expand Down Expand Up @@ -149,6 +167,9 @@ jobs:
with:
fetch-depth: 0

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Identify version
shell: bash
run: |
Expand All @@ -165,7 +186,8 @@ jobs:
../configure --enable-mpi --enable-debug \
CFLAGS="-O0 -g -pedantic -Wall -Wextra -Werror \
-Wno-unused-parameter -Wno-builtin-declaration-mismatch \
-Wno-implicit-fallthrough"
-Wno-implicit-fallthrough" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0
make -j distcheck V=0
Expand Down
24 changes: 24 additions & 0 deletions .github/workflows/ci_cmake.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ env:
CTEST_PARALLEL_LEVEL: 0
CMAKE_INSTALL_PREFIX: ~/local
CMAKE_PREFIX_PATH: ~/local
# Due to busy file system problems on the CI runners, we introduced a patch to
# sleep if the MPI count of an I/O operation is unexpectedly 0 and then retry
# the I/O operation once. The time is in milliseconds.
IO_SLEEP_TIME: 5

jobs:

Expand Down Expand Up @@ -47,6 +51,9 @@ jobs:
- uses: actions/checkout@v4
name: Checkout source code

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Install system dependencies
if: ${{ matrix.mpi }}
run: |
Expand Down Expand Up @@ -76,6 +83,9 @@ jobs:
- uses: actions/checkout@v4
name: Checkout source code

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Install system dependencies
if: ${{ matrix.mpi }}
run: |
Expand Down Expand Up @@ -112,6 +122,14 @@ jobs:
- name: Checkout source code
uses: actions/checkout@v4

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Set macro for I/O waiting
shell: bash
# use an enviroment variable to prepend the flag to the bulit-in compiler flags
run: echo "CFLAGS=-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME" >> "$GITHUB_ENV"
cburstedde marked this conversation as resolved.
Show resolved Hide resolved

- name: CMake configure
run: cmake --preset default -DSC_ENABLE_MPI:BOOL=${{ matrix.mpi }} -DSC_TEST_WITH_VALGRIND:BOOL=${{ matrix.valgrind }}

Expand Down Expand Up @@ -154,6 +172,9 @@ jobs:
- uses: actions/checkout@v4
name: Checkout source code

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Install system dependencies
run: brew install open-mpi

Expand All @@ -177,6 +198,9 @@ jobs:
- uses: actions/checkout@v4
name: Checkout source code

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- run: echo "CMAKE_INSTALL_PREFIX=$HOME/local" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append

- run: echo "CMAKE_PREFIX_PATH=$HOME/local" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
Expand Down
18 changes: 15 additions & 3 deletions .github/workflows/ci_darwin.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ on:
- ".github/workflows/composite-cmake/**"
pull_request:

env:
# Due to busy file system problems on the CI runners, we introduced a patch to
# sleep if the MPI count of an I/O operation is unexpectedly 0 and then retry
# the I/O operation once. The time is in milliseconds.
IO_SLEEP_TIME: 5

jobs:
darwin:
runs-on: macos-latest
Expand All @@ -26,6 +32,9 @@ jobs:
- uses: actions/checkout@main
name: Checkout source code

- name: Patch I/O for CI by sleeping if I/O intensity is too high
run: git apply doc/patch/patch-CI-IO.patch

- name: Install system dependencies
run: brew install open-mpi libtool automake

Expand All @@ -36,23 +45,26 @@ jobs:
run: |
DIR="checkdebug" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-debug \
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter"
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

- name: Make check with MPI and debug
run: |
DIR="checkMPIdebug" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-mpi --enable-debug \
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter"
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

- name: Make check with MPI, debug and C++ compiler
run: |
DIR="checkMPIdebugCXX" && mkdir -p "$DIR" && cd "$DIR"
../configure --enable-mpi --enable-debug CC=mpicxx \
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter"
CFLAGS="-O0 -g -Wall -Wextra -Wno-unused-parameter" \
CPPFLAGS="-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME"
make -j V=0
make -j check V=0

Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/composite-cmake/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ runs:
using: "composite"

steps:
- name: Set macro for I/O waiting
shell: bash
# use an enviroment variable to prepend the flag to the bulit-in compiler flags
run: echo "CFLAGS=-DSC_HACK_MPI_SLEEP=$IO_SLEEP_TIME" >> "$GITHUB_ENV"
- name: CMake configure
shell: bash
run: >-
Expand Down
Loading
Loading