spark-3.5-scala-2.13/GHSA-8qv5-68g4-248j/GHSA-2jc4-r94c-rp7h-fix

[jamie-albert]

Context and implications of the changes:

The inability to update org.scala-lang:scala-library to version 2.13.9 or higher in spark-3.5.x arises from a critical build issue documented in SPARK-44376. This problem stems from the transition to Scala 2.13.11 (SPARK-40497) and the deprecation of the -target argument in favor of -release in the scala-maven-plugin. This change introduces stricter compatibility checks, breaking builds when using Java 11 or later.

The key errors include inaccessible sun.* classes like DirectBuffer and Unsafe, which are not exported in Java versions above 8 (source). Despite manual attempts to adjust Maven configurations, such as switching -target:jvm-1.8 to -release:8, compilation failures persist due to inherent restrictions imposed by the -release argument. This issue is compounded by the Scala Maven plugin's automatic addition of the -release argument for Scala 2.13.9 and above, leading to incompatibilities when targeting Java 8 compatibility while running on newer Java versions.

While upstream remediated by upgrading to 2.13.11, even 2.13.9 is not possible with scala-maven-plugin version 4.8.0. Currently, scala.version is hardcoded to 2.13.8, and fixing this requires implementation of the following PRs:

1. https://github.com/apache/spark/pull/41626/files
2. https://github.com/apache/spark/pull/42899/files

The only reason remediation is achievable is due to the following conditions:

1. It is already merged upstream and awaiting a 4.0.0 release.
2. The commits are not intertwined with other more complicated initiatives or functional changes.
3. Support for Java 8 runtime dependency is not dropped with this implementation.
4. This is a critical CVE.
5. Thorough package and image-level testing is in place.
6. Downgrading scala-maven-plugin to 4.7.1 does not introduce any new CVEs beyond those already existing in 4.8.0.

This was a more involved remediation due to:

1. The dynamic setting of this version property via change-scala-version.sh.
2. The malformed scala.version property, which does not follow usual pom.xml conventions, with the same property value defined twice in the pom.xml, causing maven/pombump to not function.
3. The incredibly careful and diligent investigation required to gain confidence in the fix, alongside the experience required from working on previous spark-3.5 issues.

Bonus: Was able to tack on an additional remediation for ivy/GHSA-2jc4-r94c-rp7h

[jamie-albert]

Remaining CVEs are expected and filed here: wolfi-dev/advisories#11298