Getting started to build/test the code locally #916

minnieshi · 2025-01-03T22:47:20Z

minnieshi
Jan 3, 2025

on mac os, installed mill; and run the compile by
mill __.compile
then I tried to run any of the below without success

mill spark-excel[2.12.20,2.4.8].test
./mill __.test
mill spark-excel[2.12.20,3.2.4].test.testOnly dev.mauch.spark.excel.PlainNumberReadSuite
mill __.test

i used sdkman, java version currently is below

openjdk version "17.0.6" 2023-01-17 LTS

example failure output

[122/122] ============================== spark-excel[2.12.20,2.4.8].test ============================== 32s
1 tasks failed
spark-excel[2.12.20,2.4.8].test.test 88 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files must infer correct schema with inferSchema=true
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files with input_file_name
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read whole folder with partition
and 83 more ...

another example:

$ mill spark-excel[2.12.20,3.2.4].test.test dev.mauch.spark.excel.PlainNumberReadSuite
[122] java.lang.IllegalArgumentException: Argument unrecognized by ScalaTest's Runner: dev.mauch.spark.excel.PlainNumberReadSuite
[122/122] spark-excel[2.12.20,3.2.4].test.test
[122] at org.scalatest.tools.ArgsParser$.parseArgs(ArgsParser.scala:451)
[122] at org.scalatest.tools.Framework.runner(Framework.scala:983)
[122] at mill.testrunner.TestRunnerUtils$.getTestTasks(TestRunnerUtils.scala:116)
[122] at mill.testrunner.TestRunnerUtils$.runTestFramework0(TestRunnerUtils.scala:204)
[122] at mill.testrunner.TestRunnerMain0$.main0(TestRunnerMain0.scala:38)
[122] at mill.testrunner.TestRunnerMain0.main0(TestRunnerMain0.scala)
[122] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[122] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[122] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[122] at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[122] at mill.testrunner.entrypoint.TestRunnerMain.main(TestRunnerMain.java:42)
[122] java.lang.IllegalArgumentException: Argument unrecognized by ScalaTest's Runner: dev.mauch.spark.excel.PlainNumberReadSuite
[122/122] ============================== spark-excel[2.12.20,3.2.4].test.test dev.mauch.spark.excel.PlainNumberReadSuite ============================== 1s
1 tasks failed
spark-excel[2.12.20,3.2.4].test.test Test reporting Failed: /Users/blahblah/code/spark-excel/out/spark-excel/2.12.20/3.2.4/test/test.dest/out.json does not exist

the failure when i had plainly run

mill __.test

[1256] Run completed in 25 seconds, 948 milliseconds.
[1256] Total number of tests run: 111
[1256] Suites: completed 21, aborted 0
[1256] Tests: succeeded 23, failed 88, canceled 7, ignored 0, pending 0
[1256] *** 88 TESTS FAILED ***
[1262/1262] ============================== __.test ============================== 35s
10 tasks failed
spark-excel[2.12.20,2.4.8].test.test 88 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files must infer correct schema with inferSchema=true
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files with input_file_name
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read whole folder with partition
and 83 more ...
spark-excel[2.12.20,3.0.3].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.0.3/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.2.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.2.4/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.3.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.3.4/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.4.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.4.4/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.5.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.5.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.2.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.2.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.3.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.3.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.4.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.4.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.5.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.5.4/test/test.dest/out.json does not exist
MacBookPro:spark-excel min.shi$

nightscape · 2025-01-03T23:22:47Z

nightscape
Jan 3, 2025
Maintainer

Hi @minnieshi, can you give JDK 11 a try?
Only the newer Spark versions (not sure which ones exactly) are compatible with JDK 17.
There can still be a low number of test errors which I haven't exactly figured out, but it should be just two or three.

0 replies

minnieshi · 2025-01-03T23:42:41Z

minnieshi
Jan 3, 2025
Author

i did this

sdk use java 11.0.25-amzn
mill spark-excel[2.12.20,3.2.4].test

[122] *** 4 TESTS FAILED ***
[122/122] ============================== spark-excel[2.12.20,3.2.4].test ============================== 92s
1 tasks failed
spark-excel[2.12.20,3.2.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false

will report result for
mill __.test soon

2 replies

minnieshi Jan 3, 2025
Author

the
mill __.test output is below
@nightscape

I believe the test output is no longer related to my local environment setup.
if you agree, then I will open/fix the issue with a separate topic/title.

[1262/1262] ============================== __.test ============================== 427s
10 tasks failed
spark-excel[2.12.20,3.0.3].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.12.20,3.1.3].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.12.20,3.2.4].test.test 5 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.IntegrationSuite with maxRowsInMemory = None; maxByteArraySize = Some(100000000) reads files without headers correctly
spark-excel[2.12.20,3.3.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.12.20,3.4.4].test.test 5 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.IntegrationSuite with maxRowsInMemory = None; maxByteArraySize = Some(100000000) handles multi-line column headers correctly
spark-excel[2.12.20,3.5.4].test.test 5 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.IntegrationSuite with maxRowsInMemory = Some(1); maxByteArraySize = None reads blank cells as null and empty string cells as ""
spark-excel[2.13.15,3.2.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.13.15,3.3.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.13.15,3.4.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.13.15,3.5.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false

nightscape Jan 4, 2025
Maintainer

Yes, these look like real test failures. They are different than the ones that fail for me sometimes. For me it's just dev.mauch.spark.excel.IntegrationSuite, whereas for you there's also the dev.mauch.spark.excel.v2.PlainNumberReadSuite.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting started to build/test the code locally #916

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Getting started to build/test the code locally #916

minnieshi Jan 3, 2025

example failure output

another example:

the failure when i had plainly run

Replies: 2 comments · 2 replies

nightscape Jan 3, 2025 Maintainer

minnieshi Jan 3, 2025 Author

minnieshi Jan 3, 2025 Author

nightscape Jan 4, 2025 Maintainer

minnieshi
Jan 3, 2025

Replies: 2 comments 2 replies

nightscape
Jan 3, 2025
Maintainer

minnieshi
Jan 3, 2025
Author

minnieshi Jan 3, 2025
Author

nightscape Jan 4, 2025
Maintainer