Release v3.3.0

What's Changed

General enhancement :

• Remove diagnostic suite while keeping the relevant information in the claim file
• Retire TNF_MINIKUBE_ONLY
• Retire TNF_OC_DEBUG_IMAGE_ID
• Add tag/digest support in container certification query
• Add allow-list for tainted modules
• Ensure catalog suites are always ordered
• Display failed reason for failed and skipped tests
• Limit the number of local shell sessions to 1
• Add more unit test coverage
• Enhance and fix bugs in the REEL framework
• Ability to skip only multus tests in networking suite

Improved test cases

• Update lifecycle terminationGracePeriod TC. The tests passes only when the param is set by the customer (no matter which value).
• Update TCs to avoid fast stopping, continue testing even if one target fails, this ensures all targets are tested
• Add TC to check for image pull policy best practice
• Fix multus skip logic
• Update network connectivity test to use nsenter/ping on debug pods
• Check if CSI is certified
• Fix bug in cluster role binding TC
• Update pod-recreation and lifecycle tests to support StatefulSet

List of commits

• remove 3.0.x branch(due to failure to build), add 3.1.x branch to matrix by @edcdavid in #477
• GolangCI v1.43.0; various fixes by @sebrandon1 in #476
• Escape the new line char in the utils.ExecuteCommand function. by @greyerof in #467
• CNFCERT-170:HPA support (autodiscovery and scaling) by @aabughosh in #459
• delete hpa from pods and run scaling test by @aabughosh in #478
• Fix URL for node selector identifier by @javierpena in #480
• cnfcert170: move the delete to be before running with container] by @aabughosh in #485
• Fix Multus log with multus IP by @edcdavid in #479
• Optimize test target discovery with multiple namespaces by @hamadise in #482
• Update CONTRIBUTING.md to reflect current status by @javierpena in #484
• Add tests for isMaster/isWorker by @sebrandon1 in #483
• Retiring TNF_MINIKUBE_ONLY by @jun-rh in #487
• Add more test coverage to autodiscover by @sebrandon1 in #481
• Update links to files from the partner repo by @jun-rh in #489
• Archive test run artifacts in GitHub Actions by @javierpena in #491
• Namespaces TC extended with CRs namespace check. by @greyerof in #468
• CNFCER-185: add handler to check image pull policy by @aabughosh in #488
• Utility function for creating generic test handler by @jun-rh in #492
• More typo cleanup by @sebrandon1 in #495
• Fix the capability test in case of multi lines from the command output by @shimritproj in #493
• Add static HTML page to visualize claim.json by @javierpena in #494
• Added claim file logger function in tnf (test.go) package by @greyerof in #498
• Add unit tests for containerid package by @sebrandon1 in #497
• Fixing Multus skip logic by @edcdavid in #499
• Fix for ReelMatch() always getting called by @jun-rh in #496
• Combo TC improvements. by @greyerof in #501
• Insert claim.json file in results HTML page by @javierpena in #500
• add automount service test by @hamadise in #490
• Remove dead func by @sebrandon1 in #503
• Skipping certification tests when test input not present by @edcdavid in #505
• Add more unit tests for autodiscover by @sebrandon1 in #504
• Update Go to 1.17.4 by @sebrandon1 in #509
• Parsing and wait for termination fix in daemonset by @edcdavid in #510
• Utility function to collect status on lifecycle test failure by @jun-rh in #512
• Ensure drained nodes are uncordoned if test times out by @javierpena in #502
• Access-control: continue TCs with other Pods when one fails. by @greyerof in #508
• Observability: Continue TCs with other CUTs/CRDs when one container fails. by @greyerof in #515
• Refactoring oc session access for recreating on demand by @jun-rh in #516
• Network connectivity test based on running nsenter/ping on debug pods by @edcdavid in #511
• removing dependency on ip utility (via nsenter) by @edcdavid in #518
• Remove partner pod dead code and doc update by @edcdavid in #514
• Added clarification about the functionality of the affiliated-certification test suite to documentation by @theemj in #522
• Stop passing REDHAT_RHEL_REGISTRY in run-container.sh by @javierpena in #525
• Updated descriptions of affiliated-cert tests and regenerated CATALOG.md by @theemj in #527
• Update fsdiff test with oc session recovery on timeout by @jun-rh in #526
• run networking tests once per pod by @hamadise in #521
• Add utils package unit tests by @sebrandon1 in #523
• Enable diagnostic suite tests in non-OCP environments by @javierpena in #519
• Run cluster role binding test once per pod by @javierpena in #530
• Add unit tests for diagnostic suite by @sebrandon1 in #528
• Remove TNF_OC_DEBUG_IMAGE_ID env var by @javierpena in #529
• Remove TNF_PARTNER_NAMESPACE by @jun-rh in #534
• Add unit tests for common/env by @sebrandon1 in #535
• Affiliated-certification TS: ensure all containers/operators are tested. by @greyerof in #524
• CNFCERT-164:Scaling test on stateful sets by @aabughosh in #513
• Operator TS: ensure all operators are tested. by @greyerof in #533
• Platform alteration hp TC: added coverage for single hugepages param. by @greyerof in #531
• Update Go to 1.17.5 by @sebrandon1 in #538
• lifecycle: improve logging, test all pods if one fails by @javierpena in #532
• Remove several unused handlers by @javierpena in #544
• Readme and version update by @jun-rh in #541
• Display failure reason for failed and skipped tests by @javierpena in #545
• Remove unused readremotefile handler by @javierpena in #539
• Statefulset support for pod-recreation (node drain) test by @aabughosh in #543
• CATALOG: Restore Cluster CSI info TC. by @greyerof in #548
• Networking TS: ensure all networks/namespaces are tested. by @greyerof in #536
• Missed version upgrade; 1.43.x by @sebrandon1 in #551
• Catalog correction by @jun-rh in #553
• Add unit test coverage gate by @sebrandon1 in #552
• Fix more typos by @sebrandon1 in #555
• Removing naked returns by @sebrandon1 in #546
• update statefulset scaling identifier by @aabughosh in #550
• Replace usage of ifconfig with ip by @javierpena in #554
• Update platform-alteration/sysctl-config in CATALOG.md by @javierpena in #557
• Remove dead code and out-of-date comment by @jun-rh in #558
• Improve CNI plugin collection by @javierpena in #556
• Expose the interactive context from Oc by @jun-rh in #563
• Make the networking tests run once per pod instead of once per container by @edcdavid in #560
• Update copyright notice with year 2022 by @jun-rh in #564
• Add tests for csv_info.go by @sebrandon1 in #559
• Limit the number of local shell sessions to 1 by @jun-rh in #565
• Bump github.com/spf13/cobra from 1.2.1 to 1.3.0 by @dependabot in #568
• Bump google.golang.org/grpc from 1.42.0 to 1.43.0 by @dependabot in #569
• Add more autodiscover tests by @sebrandon1 in #567
• Add best practice ref for automount token by @sebrandon1 in #572
• Fix missed Go 1.17.5 upgrade by @sebrandon1 in #573
• Lifecycle's TC for terminationGracePeriod param refactored. by @greyerof in #562
• Use labels in test cases by @javierpena in #571
• Fix false positives caused by the grep expression in clusterrolebinding.go by @jun-rh in #575
• Improve git management in Makefile and release workflow by @javierpena in #574
• #1 enhance catalog browsing by @jgato in #561
• Add unit tests for config.go by @sebrandon1 in #578
• Collect and parse image source for containers under test by @jun-rh in #579
• Add test label to catalog by @javierpena in #582
• Update multus networking TC id by @javierpena in #577
• Include discovered container images in the container certification test by @jun-rh in #581
• Update Go to 1.17.6 by @sebrandon1 in #585
• Check if port that listening is declared by @shimritproj in #566
• Bump coverage threshold to 70% by @sebrandon1 in #590
• Fix simple typo by @sebrandon1 in #588
• Catalog query retry on errors only by @jun-rh in #589
• Ensure catalog suites are always ordered by @javierpena in #586
• Bump github.com/onsi/gomega from 1.17.0 to 1.18.0 by @dependabot in #594
• Re-enable the unused linter by @sebrandon1 in #593
• Add unit test for FindDebugPods by @sebrandon1 in #592
• Add allow-list for tainted modules by @sebrandon1 in #580
• Add tag/digest support in container certification query by @jun-rh in #595
• Bump google.golang.org/grpc from 1.43.0 to 1.44.0 by @dependabot in #598
• Bump GolangCI-lint to 1.44.0 by @sebrandon1 in #600
• Add type field to the CNI plugin info in claim file. by @greyerof in #601
• Bump github.com/onsi/gomega from 1.18.0 to 1.18.1 by @dependabot in #606
• Upgrade Ginkgo to v2.1.1 by @sebrandon1 in #608
• Lifecycle TS: avoid some TCs to stop after each failure. by @greyerof in #609
• Add freshness grade check to container certification test by @jun-rh in #599
• Add identifer tests by @sebrandon1 in #597
• Fix install-tools path by @sebrandon1 in #610
• cnfcert-203: check csi certified by @aabughosh in #570
• Add accesscontrol tests by @sebrandon1 in #602
• Diagnostic TS removal by @greyerof in #591
• Backport for skip Multus label support by @edcdavid in #614
• Add case for docker-pullable by @sebrandon1 in #613
• update partner to v3.3.0 by @hamadise in #615

New Contributors

• @javierpena made their first contribution in #480
• @theemj made their first contribution in #522
• @jgato made their first contribution in #561

Full Changelog: v3.1.0...v3.3.0