Separate ClassLoader per test

[WolfgangHG]

I try to migrate some JUnit4 tests to JUnit5 and struggle with a problem: the project uses arquillian to deploy war files to a webserver and runs tests in the server side. Thus, the war file is created with Shrinkwrap and arquillian deploys it to the container.

Some tests in this project simulate the arquillian approach without actually using arquillian: they create a war file in one classloader, then they do some tests in a separate classloader to test for example that all necessary classes are contained in the package. Currently, they do this with a BlockJUnit4ClassRunner, which is no longer available in JUnit5: https://github.com/arquillian/arquillian-extension-warp/blob/main/impl/src/test/java/org/jboss/arquillian/warp/impl/testutils/SeparatedClassloaderRunner.java
A test that uses it is https://github.com/arquillian/arquillian-extension-warp/blob/main/impl/src/test/java/org/jboss/arquillian/warp/impl/client/deployment/TestDeploymentEnricherClassPath.java

How could I handle this with JUnit5? I found the "LauncherInterceptor" (https://junit.org/junit5/docs/snapshot/user-guide/#launcher-api-launcher-interceptors-custom, added by #3091), but this one seems to be created only once per test suite run. So I would have to create a separate project for each of thoses tests with one LauncherInterceptor that invokes the method that creates the archive? Or is there a better approach?

[marcphilipp]

I created a sample based on a ServiceLoader-registered LauncherSessionListener here: https://github.com/marcphilipp/gradle-sandbox/tree/master/classloader-per-test-class

It installs a custom classloader for the entire session that then checks if a custom class loader is needed for each test class:
https://github.com/marcphilipp/gradle-sandbox/blob/baaa1972e939f5817f54a3d287611cef0601a58d/classloader-per-test-class/src/test/java/org/example/ClassLoaderReplacingLauncherSessionListener.java#L23-L44

For the detection I used a custom @SeparateClassLoader annotation that specifies a factory for creating the class loader. In the sample, a URLClassLoader is created that (to mimick your use case) calls a static method annotated with a custom @Classpath annotation on the test class:
https://github.com/marcphilipp/gradle-sandbox/blob/baaa1972e939f5817f54a3d287611cef0601a58d/classloader-per-test-class/src/test/java/org/example/CustomClassLoaderFactory.java

When you run ./gradlew test, you should see output similar to the following:

org.example.RegularTests class loader: jdk.internal.loader.ClassLoaders$AppClassLoader@639fee48 (app)
org.example.SeparateClassLoaderTests class loader: java.net.URLClassLoader@2f4948e4 (org.example.SeparateClassLoaderTests)
Closing java.net.URLClassLoader@2f4948e4

This should work for all class-based test engines, not just Jupiter. But there's a Jupiter-specific issue for providing an extension API (#3028). Feel free to chime in!

[holly-cummins]

I've done something similar to what @marcphilipp suggests. I create a FacadeClassLoader and set it as the thread context classloader, and it delegates to the 'real' classloader for each test. It mostly works well.

[WolfgangHG]

Cool, many thanks! Will give it a try tomorrow.

[WolfgangHG]

Whow, this is rather tricky... Still trying to understand the code and adapt it to the existing tests...
I think I stumbled across #3028 while researching my question ;-).

@marcphilipp Attached is a reworked version of your sample. I converted it to maven and switched the URLClassLoader to a ShrinkWrapClassLoader.
classloader-per-test-class_maven.zip

Several questions:

1. You declared @ExtendWith on the annotation SeparateClassLoader. I assume I can declare it on the tests instead? As it works in my sample, I think this is no problem.

   @SeparateClassLoader(factory = CustomClassLoaderFactory.class)
   @ExtendWith(SeparateClassLoader.Extension.class)
   public class SeparateClassLoaderTests {
   

   Reason: the @SeparatedClassPath annotation in my old code is similar to @ClassPath in your sample and is defined on a method that creates the shrinkwrap archive. So, I don't need a @SeparateClassLoader annotation at class level.

2. In CustomClassLoaderFactory, you create the "filteringClassLoader" with a base classloader ClassLoader.getSystemClassLoader().
   The old SeparatedClassloaderRunner in my code used ClassLoader.getSystemClassLoader().getParent() instead. I first copied this code, and it resulted in maven not discovering any tests. After switching to your approach, the tests are found. Hope this causes no side effects?

3. For my own understanding: What is the purpose of your "filteringClassLoader" and "loadClass" returning null for the test class?

      var filteringClassLoader = new ClassLoader(ClassLoader.getSystemClassLoader()) {
           @Override
           protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
               return name.equals(testClass.getName()) ? null : super.loadClass(name, resolve);
           }
       };
   

   I found that if I remove it and just use a system classloader as base classloader for my, the tests print the wrong class loader (not the URLClassloader in your sample or the ShrinkWrapClassLoader in my sample). This is the output without "filteringClassLoader":
   org.example.SeparateClassLoaderTests class loader: jdk.internal.loader.ClassLoaders$AppClassLoader@73d16e93 (app)
   So, the "filteringClassLoader" seems to be important, but I don't understand why ;-)

But after adding the "filteringClassLoader" to my real word sample, the content of my test method breaks with a error "...is in unnamed module of loader 'app'" => will have to do further research.

[WolfgangHG]

About the "...unnamed module..." error (last paragraph of my previous question): it happens when the @Test method uses a class that is part of the current project impl in "src/main/java". Those classes are are added to a separate jar in the ShrinkWrapClassLoader. But they are loaded from the wrong classloader ("jdk.internal.loader.ClassLoaders$AppClassLoader"). And this seems to cause the module error.

It works if I return "null" for those classes in the filteringClassLoader. But I don't understand why this happens, which brings me back to my previous question 3 ;-).

I could not reproduce the module error with my modified sample, but I can reproduce the problem that the framework class is loaded from the wrong classloader without the workaround to exclude it in the filteringClassLoader.

[marcphilipp]

1. You declared @ExtendWith on the annotation SeparateClassLoader. I assume I can declare it on the tests instead? As it works in my sample, I think this is no problem.

Yes, that's fine. I only did that to ensure both are always used together using JUnit's composed annotation feature.

2. In CustomClassLoaderFactory, you create the "filteringClassLoader" with a base classloader ClassLoader.getSystemClassLoader().
The old SeparatedClassloaderRunner in my code used ClassLoader.getSystemClassLoader().getParent() instead. I first copied this code, and it resulted in maven not discovering any tests. After switching to your approach, the tests are found. Hope this causes no side effects?

I guess it depends on which class loader is used for what. Maven needs to use the same class loader to load JUnit's classes. Otherwise, it won't be able to work with it's API.

3. For my own understanding: What is the purpose of your "filteringClassLoader" and "loadClass" returning null for the test class?

The purpose is to pretend the test class wasn't yet loaded so the child class loader will load it again (after asking its parent).

It works if I return "null" for those classes in the filteringClassLoader. But I don't understand why this happens, which brings me back to my previous question 3 ;-).

I would expect that to be necessary. In my simplistic example, the test class doesn't depend on any other classes but in your case you'd have to filter out an entire layer (by returning null) so that it's loaded with the class loader you want.

[WolfgangHG]

I thought I am done with my current implementation - the tests using the @SeparatedClassLoader now work.

But another test fails with a ClassNotFoundException, which is probably caused by the SeparatedClassloaderLauncherSessionListener installing a classloader. This test
https://github.com/WolfgangHG/arquillian-extension-warp/blob/junit5/impl/src/test/java/org/jboss/arquillian/warp/impl/client/transformation/TestMigratedInspection.java creates internally a new classloader and seems to serialize a class to bytecode the the original classloader, the tries to deserialize it in the new classloader.

@marcphilipp Do you have any idea about this and do you have a suggestion how to avoid it?

The exception is:

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.234 s <<< FAILURE! -- in org.jboss.arquillian.warp.impl.client.transformation.TestMigratedInspection
[ERROR] org.jboss.arquillian.warp.impl.client.transformation.TestMigratedInspection.testMigrate -- Time elapsed: 0.217 s <<< ERROR!
java.lang.IllegalStateException: Cannot migrate transformed inspection back to original name
        at org.jboss.arquillian.warp.impl.client.transformation.MigratedInspection.migrate(MigratedInspection.java:78)
        at org.jboss.arquillian.warp.impl.client.transformation.MigratedInspection.<init>(MigratedInspection.java:54)
        at org.jboss.arquillian.warp.impl.client.transformation.TestMigratedInspection.testMigrate(TestMigratedInspection.java:42)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
Caused by: java.lang.IllegalStateException: Unable to invoke static method deserializeFromBytes from class org.jboss.arquillian.warp.impl.utils.SerializationUtils
        at org.jboss.arquillian.warp.impl.client.separation.SeparateInvocator.invokeStatic(SeparateInvocator.java:85)
        at org.jboss.arquillian.warp.impl.client.separation.SeparateInvocator$2.invoke(SeparateInvocator.java:207)
        at org.jboss.arquillian.warp.impl.client.separation.SeparateInvocator$InvokeSeparately.run(SeparateInvocator.java:224)
        at org.jboss.arquillian.warp.impl.client.separation.SeparateInvocator.adapt(SeparateInvocator.java:209)
        at org.jboss.arquillian.warp.impl.client.separation.SeparateInvocator.adaptArgs(SeparateInvocator.java:171)
        at org.jboss.arquillian.warp.impl.client.separation.SeparateInvocator$SeparationHandler.invoke(SeparateInvocator.java:96)
        at com.sun.proxy.$Proxy10.process(Unknown Source)
        at org.jboss.arquillian.warp.impl.client.transformation.MigratedInspection.migrate(MigratedInspection.java:75)
        ... 5 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.arquillian.warp.impl.client.separation.SeparateInvocator.invokeStatic(SeparateInvocator.java:82)
        ... 12 more
Caused by: java.lang.IllegalStateException: java.lang.ClassNotFoundException: org.jboss.arquillian.warp.generated.A0c5cdda5-6111-4716-82a3-cec799b1c937
        at org.jboss.arquillian.warp.impl.utils.SerializationUtils.deserializeFromBytes(SerializationUtils.java:53)
        ... 17 more
Caused by: java.lang.ClassNotFoundException: org.jboss.arquillian.warp.generated.A0c5cdda5-6111-4716-82a3-cec799b1c937
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:398)
        at java.base/java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:763)
        at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2025)
        at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1892)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2223)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1709)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:500)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:458)
        at org.jboss.arquillian.warp.impl.utils.SerializationUtils.deserializeFromBytes(SerializationUtils.java:49)
        ... 17 more

A simple suggestion could be to extract all tests using the @SeparatedClassLoader to a new project, which installs the listener.

[marcphilipp]

I'm not sure but it sounds like it's using the wrong class loader now. Is that a problem with ClassLoader.getSystemClassLoader() vs. ClassLoader.getSystemClassLoader().getParent() that you're using as a parent of the installed class loader?

[WolfgangHG]

The ClassLoader.getSystemClassLoader() problem in my previous question happened only for tests annotated with @SeparatedClassPath, as the relevant code is invoked only in this case. This specific test executes only the SeparatedClassloaderLauncherSessionListener part that installs an additional classloader to check for annotated tests.

The code invoked by TestMigratedInspection invokes ClassLoader.getSystemClassLoader().getParent() at several places, but replacing them with ClassLoader.getSystemClassLoader() does not help. I don't have much knowledge of this project (I started doing some maintenance work some time ago after it was abandoned for a long time), so I don't know why most code was written and what purpose it has ;-). You do neither I fear ;-).

If you don't have an idea, I think the easiest resolution is to extract all @SeparatedClassPath tests to an extra module, thus avoiding the classloader conflict.

[marcphilipp]

You do neither I fear ;-)

That's correct 😅

If you don't have an idea, I think the easiest resolution is to extract all @SeparatedClassPath tests to an extra module, thus avoiding the classloader conflict.

I think so, too. 👍

[WolfgangHG]

@marcphilipp Thanks for your help and the suggested sample. So I mark this discussion as "closed". I will continue working on migrating my test suite. Unfortunately, arquillian is also not fully ready for JUnit5, so other prerequisites are missing.

Hopefully, one day #3028 will be implemented and an easier apporach for a SeparateClassLoader can be used.