NIFI-12178 add integration-tests and docker-tests github actions

[ChrisSamo632]

Summary

NIFI-12178 add integration-tests and docker-tests github actions

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 21

Licensing

• [ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy`
• [ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• [ ] Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for working on these additional workflows @ChrisSamo632. Having automated execution of working integration tests and Docker builds will be very useful.

On initial review, I raised several implementation questions. At a high level, it would be very useful to run integration tests locally, so it seems like a more Maven-centric approach to integration test selection would be better. That may present some challenges, but I was wondering whether you considered those options. I also have a few questions on some of the particular build steps and scripting commands.

[ChrisSamo632]

@exceptionfactory I've rebased from latest main as there have been quite a few changes over the past few days.

I also noticed I'd missed the nifi-toolkit tests from the new docker-tests workflow, which are failing on main due to the file-manager command no longer being present in the CLI - something I spotted and corrected also in NIFI-12175

[exceptionfactory]

Thanks for making the updates @ChrisSamo632. I noted a few more things on this review, in particular, selecting just Azul Zulu as the JDK should simplify the Docker and Integration Test workflows.

[exceptionfactory]

See note above on removing the TODO given the existence of the Jira issue.

[ChrisSamo632]

@exceptionfactory RE #7858 (comment) (skipping the rm -rf Clean step for the docker-tests)

The last run of the Workflow in this PR that didn't include the rm -rf ran out of disk space, it would appear these removals are still required.

Building without the ZIP wouldn't work - the dockermaven builds use the ZIP files (I don't want to change that either as that's what the dockerhub builds do when downloading the build convenience artefacts from the Apache Archive servers), so the clean up step:

• cleans the Maven module target folders (except for the assembly modules as that's what we need for the Docker Image build step)
• deletes the un-zipped files from the assembly module target folders

The Docker Image builds each create a new Image (TAR file) containing the assembled ZIP, then unzips them before deleting the ZIP file - this uses a lot of disk space

[exceptionfactory]

@exceptionfactory RE #7858 (comment) (skipping the rm -rf Clean step for the docker-tests)

The last run of the Workflow in this PR that didn't include the rm -rf ran out of disk space, it would appear these removals are still required.

Building without the ZIP wouldn't work - the dockermaven builds use the ZIP files (I don't want to change that either as that's what the dockerhub builds do when downloading the build convenience artefacts from the Apache Archive servers), so the clean up step:

* cleans the Maven module `target` folders (except for the `assembly` modules as that's what we need for the Docker Image build step)

* deletes the un-zipped files from the assembly module `target` folders

The Docker Image builds each create a new Image (TAR file) containing the assembled ZIP, then unzips them before deleting the ZIP file - this uses a lot of disk space

Thanks for the additional background @ChrisSamo632, that makes sense now, it sounds like the current approach is the best option.

[ChrisSamo632]

@exceptionfactory have rebased from main latest after you merged NIFI-12175 - all new & existing test Workflows now passing without errors or warnings (from the Github Runners) 👍

[exceptionfactory]

Thanks for addressing the feedback and making adjustments @ChrisSamo632, this will be a helpful way to improve maintenance visibility for Docker builds and integration tests. +1 merging