Maven fails to build after ozone ufs upgrade to 1.0.0 #12338

Xenorith · 2020-10-17T05:19:15Z

Alluxio Version:
2.4.0

Describe the bug
After merging the PR to bump ozone ufs version (#12226), the following maven command hangs:
mvn -T 4C clean install -Dmaven.javadoc.skip=true -DskipTests -Dlicense.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -P'\''!ufs-hadoop-1.2,!ufs-hadoop-2.2,!ufs-hadoop-2.6'\''

When I run jstack on the process, the one running thread approximately has the following stacktrace:

"BuilderThread 17" #60 prio=5 os_prio=0 tid=0x00007fcea0106800 nid=0xbaa runnable [0x00007fcdb9c2a000]
   java.lang.Thread.State: RUNNABLE
        at org.jdom2.Element.isAncestor(Element.java:1052)
        at org.jdom2.ContentList.checkPreConditions(ContentList.java:222)
        at org.jdom2.ContentList.add(ContentList.java:244)
        at org.jdom2.Element.addContent(Element.java:950)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.insertAtPreferredLocation(MavenJDOMWriter.java:292)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.iterateExclusion(MavenJDOMWriter.java:488)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.updateDependency(MavenJDOMWriter.java:1335)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.iterateDependency(MavenJDOMWriter.java:386)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.updateModel(MavenJDOMWriter.java:1623)
        at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.write(MavenJDOMWriter.java:2156)
        at org.apache.maven.plugins.shade.pom.PomWriter.write(PomWriter.java:75)
        at org.apache.maven.plugins.shade.mojo.ShadeMojo.rewriteDependencyReducedPomIfWeHaveReduction(ShadeMojo.java:1049)
        at org.apache.maven.plugins.shade.mojo.ShadeMojo.createDependencyReducedPom(ShadeMojo.java:978)
        at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:538)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
        at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:200)
        at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:196)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

This seems to be an issue with the maven shading plugin. Looking up at org.apache.maven.plugins.shade.pom.MavenJDOMWriter.insertAtPreferredLocation(MavenJDOMWriter.java:292), I found that other people have encountered a similar issue:

https://stackoverflow.com/questions/52221326/maven-shade-plugin-failed-to-execute-goal
- but it seems like our plugin version is sufficiently up to date
[SPARK-26095][build] Disable parallelization in make-distibution.sh. apache/spark#23061
- this seems to avoid the issue by removing parallelism, but this would drastically slow down our builds

To Reproduce
See above mvn command

Expected behavior
Maven command completes

Urgency
Offending PR has been reverted to avoid build issues (#12337)

The text was updated successfully, but these errors were encountered:

This reverts commit 118769b to fix Alluxio#12338 otherwise, maven build may stuck in shading Cherry-pick of existing commit. orig-pr-link: Alluxio#12337 orig-commit: Alluxio/alluxio@c1e07c5 orig-commit-author: Bin Fan <fanbin103@gmail.com> pr-link: n/a change-id: cid-64f10e66696d1241496852cbd51ceddf5f1d1a2d

maobaolong · 2020-10-19T15:18:12Z

@Xenorith Thank you for report this issue, it seems is the same issue like this https://issues.apache.org/jira/browse/MSHADE-148.

I think the root cause of this issue is Shade plugin using the wrong project while building the dependency reduced pom in parallel builds. Reference https://www.mail-archive.com/dev@maven.apache.org/msg122620.html

The above screenshot is snapped while I debug the maven shaded plugin under the parallel build, It makes me feel strange that the current project is different with the result.getProject(), since maybe we cannot depend the update of the maven-shade-plugin, There are 4 options to work around:

Add <createDependencyReducedPom>false</createDependencyReducedPom> to the alluxio/shaded/client/pom.xml to avoid generated DependencyReducedPom.xml
Removing parallelism build
Add provide scope dependency between shade/client, shade/hadoop and shade/ozone to make them sequential.
Patch [MSHADE-148] Don't attach exclusions that are already present apache/maven-shade-plugin#16 and use the patched version maven-shade-plugin.

After an overview underfs/ozone and underfs/wasb, I don't find a way can avoid ClassCastException if do not shade ozone.

The reason is that there are a class cast in org.apache.hadoop.ipc.ProtobufRpcEngine#invoke

      final Message theRequest = (Message) args[1];

The Message is loaded by as shaded version, so it is alluxio.shaded.hdfs.com.google.protobuf.Message, but the args[1] is from Ozone server, so it is com.google.protobuf.Message if without shaded.

Xenorith · 2020-10-19T18:41:40Z

Thanks for looking into this so quickly @maobaolong

I am not familiar enough with maven to help decide which option is best to work around the issue. I would strongly prefer we avoid Removing parallelism because it would drastically slow down build times. Otherwise, I cannot say much about the other options you proposed.

@apc999 Can you provide some guidance on what is the best way to proceed with this issue?

maobaolong · 2020-10-19T23:59:55Z

@Xenorith Sure, we need to support parallelism build, there are one more option

The SPARK project is suffering from this issue too, https://issues.apache.org/jira/browse/SPARK-8819 this is a option to support parallelism build with let createDependencyReducedPom=false default, but support a release flag to let createDependencyReducedPom=true and non-parallelism build for release build, this is we don't need to create dependency deduced pom file every time, when we need, just with release flag.

apc999 · 2020-10-20T18:07:30Z

@maobaolong @Xenorith Thanks for getting to the bottom of this problem!
To me removing reduced pom (like https://stackoverflow.com/a/11322920/3161750) is a good way to go as it seems not really used at all (https://stackoverflow.com/a/35155294/3161750)

@calvinjia any input?

calvinjia · 2020-10-21T04:54:32Z

@apc999 The dependency reduced pom is relevant when published so I think @maobaolong ’s proposed solution is the best of both worlds.

apc999 · 2020-10-21T05:44:44Z

I actually think the generated pom in src dir is not relevant to depending modules. There will be a pom in .m2/repository/org/alluxio/alluxio-shaded-ozone/2.4.0 which is identical to the reduced dependency pom. maybe we can test it to have createDependencyReducedPom=fase and see if .m2/repository/org/alluxio/alluxio-shaded-ozone/2.4.0/alluxio-shaded-ozone-2.4.0.pom is identical to initial pom or the generated pom @calvinjia

calvinjia · 2020-10-21T17:17:54Z

Sure, my understanding is with false it will have the original pom

maobaolong · 2020-10-22T13:46:31Z

@apc999 I have checked that if we set createDependencyReducedPom=false, the .m2/repository/org/alluxio/alluxio-shaded-ozone/2.4.0/alluxio-shaded-ozone-2.4.0.pom will be the same as alluxio/shaded/ozone/pom.xml

#12338 This is a workaround for MSHADE-148, which leads to an infinite loop when building Alluxio. There are two scenarios. - Releasing Alluxio Maven build with `-Prelease`. - If using newer than `3.3.x` version of maven, you also have to use sequential build, Like `mvn clean install -Prelease xxxx`. - If using lower version maven before `3.3.x`, you can use parallel build, e.g. `mvn clean install -T 4 -Prelease xxxx` - Not Releasing Alluxio If you are not releasing Alluxio, you can build Alluxio like before, please don't add the `-Prelease` into you build command line. e.g. `mvn clean install -T 4 xxxx` This patch adds a -Prelease. If present, it will set createDependencyReducedPom true. The consequences are: If you are releasing Alluxio with this profile, you are fine as long as before. If you are releasing Alluxio without this profile, the `alluxio-shaded-xxx-yyy.pom` will be the same as the pom.xml under the module directory. If you are not releasing Alluxio but you are using this profile, you may run into #12338 If you are not releasing Alluxio and you did not include this profile, you are fine. This is all documented in pom.xml and tested locally with maven 3.6.3. pr-link: #12367 change-id: cid-2f26f91c9f3198d9dac11633888de0a9f8af2ce6

apc999 · 2020-10-23T17:44:02Z

This issue has been fixed by #12367

#12338 This is a workaround for MSHADE-148, which leads to an infinite loop when building Alluxio. There are two scenarios. - Releasing Alluxio Maven build with `-Prelease`. - If using newer than `3.3.x` version of maven, you also have to use sequential build, Like `mvn clean install -Prelease xxxx`. - If using lower version maven before `3.3.x`, you can use parallel build, e.g. `mvn clean install -T 4 -Prelease xxxx` - Not Releasing Alluxio If you are not releasing Alluxio, you can build Alluxio like before, please don't add the `-Prelease` into you build command line. e.g. `mvn clean install -T 4 xxxx` This patch adds a -Prelease. If present, it will set createDependencyReducedPom true. The consequences are: If you are releasing Alluxio with this profile, you are fine as long as before. If you are releasing Alluxio without this profile, the `alluxio-shaded-xxx-yyy.pom` will be the same as the pom.xml under the module directory. If you are not releasing Alluxio but you are using this profile, you may run into #12338 If you are not releasing Alluxio and you did not include this profile, you are fine. This is all documented in pom.xml and tested locally with maven 3.6.3. pr-link: #12367 change-id: cid-2f26f91c9f3198d9dac11633888de0a9f8af2ce6

Xenorith added the type-bug This issue is about a bug label Oct 17, 2020

apc999 mentioned this issue Oct 17, 2020

Revert "Upgrade Ozone ufs to 1.0.0 version" #12337

Merged

alluxio-bot closed this as completed in #12337 Oct 17, 2020

apc999 reopened this Oct 17, 2020

Xenorith mentioned this issue Oct 17, 2020

Revert "Upgrade Ozone ufs to 1.0.0 version" #12342

Merged

HelloHorizon closed this as completed in c1e07c5 Oct 19, 2020

Xenorith reopened this Oct 19, 2020

maobaolong mentioned this issue Oct 22, 2020

Add a releaseProfile to generate dependency reduced pom #12367

Merged

maobaolong mentioned this issue Oct 23, 2020

Upgrade Ozone ufs to 1.0.0 version #12371

Merged

apc999 closed this as completed Oct 23, 2020

This was referenced Oct 23, 2020

Port changes of [#12367] to branch-2.4 #12377

Merged

Port changes of [#12371] to branch-2.4 #12380

Merged

apc999 mentioned this issue Oct 27, 2020

Use -T 1 -Prelease when publishing tarballs #12392

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Maven fails to build after ozone ufs upgrade to 1.0.0 #12338

Maven fails to build after ozone ufs upgrade to 1.0.0 #12338

Xenorith commented Oct 17, 2020 •

edited

Loading

maobaolong commented Oct 19, 2020 •

edited

Loading

Xenorith commented Oct 19, 2020

maobaolong commented Oct 19, 2020

apc999 commented Oct 20, 2020

calvinjia commented Oct 21, 2020

apc999 commented Oct 21, 2020

calvinjia commented Oct 21, 2020

maobaolong commented Oct 22, 2020

apc999 commented Oct 23, 2020

Maven fails to build after ozone ufs upgrade to 1.0.0 #12338

Maven fails to build after ozone ufs upgrade to 1.0.0 #12338

Comments

Xenorith commented Oct 17, 2020 • edited Loading

maobaolong commented Oct 19, 2020 • edited Loading

Xenorith commented Oct 19, 2020

maobaolong commented Oct 19, 2020

apc999 commented Oct 20, 2020

calvinjia commented Oct 21, 2020

apc999 commented Oct 21, 2020

calvinjia commented Oct 21, 2020

maobaolong commented Oct 22, 2020

apc999 commented Oct 23, 2020

Xenorith commented Oct 17, 2020 •

edited

Loading

maobaolong commented Oct 19, 2020 •

edited

Loading