-
Notifications
You must be signed in to change notification settings - Fork 96
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add release notes for spark images gen25 (#5074)
- Loading branch information
Showing
3 changed files
with
161 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
159 changes: 159 additions & 0 deletions
159
src/docs/ocean-spark/docker-images-release-notes/gen25.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| # gen25 release notes (2024-11-26) | ||
|
|
||
| ## Changelog | ||
|
|
||
| - upgrade spark 3.4 images to spark 3.4.4 | ||
| - upgrade Nvidia images from spark 3.3.4 to spark 3.4.3 | ||
| - images now have a named 'spark' user matching user id 185 | ||
| - aws sdk upgraded to v1.12.777 | ||
| - pyarrow upgraded to v18.0.0 | ||
| - ipython upgraded to v8.29.0 | ||
| - guava upgraded to v32.1.3 | ||
| - deltalake upgraded to v3.2.1 | ||
|
|
||
| ## Available tags | ||
|
|
||
| ### platform-3.4.4-hadoop-3.3.6-java-11-scala-2.12-python-3.10-gen25 | ||
|
|
||
| #### Additional tags | ||
|
|
||
| - `platform-3.4-gen25` | ||
| - `platform-3.4.3-gen25` | ||
|
|
||
| #### Dependency versions | ||
|
|
||
| | name | version | | ||
| | :-----: | :-----: | | ||
| | Spark | 3.4.4 | | ||
| | Hadoop | 3.3.6 | | ||
| | Java | 11 | | ||
| | Scala | 2.12.17 | | ||
| | Python | 3.10.15 | | ||
| | Hive | 2.3.9 | | ||
| | Pyarrow | 18.0.0 | | ||
|
|
||
| #### Supported connectors | ||
|
|
||
| | name | version | | ||
| | :-----------------------------: | :-----------------: | | ||
| | S3 (`s3a://` or `s3://`) | AWS 1.12.367 | | ||
| | GCS (`gs://`) | 2.2.11-shaded | | ||
| | ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | | ||
| | ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | | ||
| | Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | | ||
| | AWS Glue | Hive 2.3.9 | | ||
|
|
||
| #### Additional formats | ||
|
|
||
| | name | version | | ||
| | :-------: | :-----------: | | ||
| | Delta | 2.4.0 | | ||
| | Snowflake | Not supported | | ||
|
|
||
| ### platform-3.5.3-hadoop-3.3.6-java-11-scala-2.12-python-3.10-gen25 | ||
|
|
||
| #### Additional tags | ||
|
|
||
| - `platform-3.5-gen25` | ||
| - `platform-3.5.3-gen25` | ||
|
|
||
| #### Dependency versions | ||
|
|
||
| | name | version | | ||
| | :-----: | :-----: | | ||
| | Spark | 3.5.3 | | ||
| | Hadoop | 3.3.6 | | ||
| | Java | 11 | | ||
| | Scala | 2.12.18 | | ||
| | Python | 3.10.15 | | ||
| | Hive | 2.3.9 | | ||
| | Pyarrow | 18.0.0 | | ||
|
|
||
| #### Supported connectors | ||
|
|
||
| | name | version | | ||
| | :-----------------------------: | :-----------------: | | ||
| | S3 (`s3a://` or `s3://`) | AWS 1.12.367 | | ||
| | GCS (`gs://`) | 2.2.14-shaded | | ||
| | ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | | ||
| | ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | | ||
| | Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | | ||
| | AWS Glue | Hive 2.3.9 | | ||
|
|
||
| #### Additional formats | ||
|
|
||
| | name | version | | ||
| | :-------: | :-----------: | | ||
| | Delta | Not supported | | ||
| | Snowflake | Not supported | | ||
|
|
||
| ### platform-3.5.3-hadoop-3.3.6-java-8-scala-2.12-python-3.10-gen25 | ||
|
|
||
| #### Dependency versions | ||
|
|
||
| | name | version | | ||
| | :-----: | :-----: | | ||
| | Spark | 3.5.3 | | ||
| | Hadoop | 3.3.6 | | ||
| | Java | 8 | | ||
| | Scala | 2.12.18 | | ||
| | Python | 3.10.15 | | ||
| | Hive | 2.3.9 | | ||
| | Pyarrow | 18.0.0 | | ||
|
|
||
| #### Supported connectors | ||
|
|
||
| | name | version | | ||
| | :-----------------------------: | :-----------------: | | ||
| | S3 (`s3a://` or `s3://`) | AWS 1.12.367 | | ||
| | GCS (`gs://`) | 2.2.14-shaded | | ||
| | ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | | ||
| | ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | | ||
| | Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | | ||
| | AWS Glue | Hive 2.3.9 | | ||
|
|
||
| #### Additional formats | ||
|
|
||
| | name | version | | ||
| | :-------: | :-----------: | | ||
| | Delta | Not supported | | ||
| | Snowflake | Not supported | | ||
|
|
||
| ### platform-3.4.3-nvidia-hadoop-3.3.6-java-8-scala-2.12-python-3.10-gen25 | ||
|
|
||
| #### Dependency versions | ||
|
|
||
| | name | version | | ||
| | :-----: | :-----: | | ||
| | Spark | 3.4.3 | | ||
| | Hadoop | 3.3.6 | | ||
| | Java | 8 | | ||
| | Scala | 2.12.17 | | ||
| | Python | 3.10.15 | | ||
| | Hive | 2.3.9 | | ||
| | Pyarrow | 18.0.0 | | ||
|
|
||
| #### Supported connectors | ||
|
|
||
| | name | version | | ||
| | :-----------------------------: | :-----------------: | | ||
| | S3 (`s3a://` or `s3://`) | AWS 1.12.367 | | ||
| | GCS (`gs://`) | 2.2.11-shaded | | ||
| | ADLS gen1 (`adl://`) | ADLS SDK 2.3.9 | | ||
| | ADLS gen2 (`abfss://`) | Azure Storage 7.0.1 | | ||
| | Azure Blob Storage (`wasbs://`) | Azure Storage 7.0.1 | | ||
| | AWS Glue | Hive 2.3.9 | | ||
|
|
||
| #### Additional formats | ||
|
|
||
| | name | version | | ||
| | :-------: | :-----------: | | ||
| | Delta | 2.4.0 | | ||
| | Snowflake | Not supported | | ||
|
|
||
| #### GPU Nvidia | ||
|
|
||
| | name | version | | ||
| | :----------------: | :-----: | | ||
| | Cuda Driver | 11.8.89 | | ||
| | Rapids-4-Spark Lib | 24.10.0 | |