Unable to load Lua54 natives when running/debugging in IntelliJ with minecraftforge+gradle

[GHXX]

Describe the bug
This might simply be user-error on my part, however, it appears that, upon creating an instance of Lua54, the luajava library attempts to use the com.badlogic.gdx.utils.SharedLibraryLoader to load lua5464.dll, however locating this dll fails, as I suspect it would need to be located in a different package/jar.

To Reproduce
Steps to reproduce the behavior (assuming you are on Windows, otherwise step 2 differs slightly):

1. Clone this repository: https://github.com/GHXX/luajavatest
2. Run genIntellijRuns.bat (should say BUILD SUCCESSFUL at the end)
3. Open the folder as a project using IntelliJ IDEA (in my case 2021.2.4; you probably have to trust the project)
4. Reload All Gradle projects (Ctrl+Shift+A --> "Reload all Gradle[...]" , only seems to show up if you type at most "Reload all")
5. Check the runClient runconfiguration to make sure Java17 or JDK17 is selected (see first attached image)
6. Simply Run the project using the green play-button in IntelliJ

Current behavior
While the class Lua54 is loaded properly, the natives are not found by the SharedLibraryLoader, possibly because they are located in a jar that is separate from the jar containing the SharedLibraryLoader (see image 2; the highlighted line causes input to be null). This is represented by the stacktrace below.

at com.badlogic.gdx.utils.SharedLibraryLoader.readFile(SharedLibraryLoader.java:137) ~[gdx-jnigen-loader-2.3.1.jar%23107!/:?] {}
at com.badlogic.gdx.utils.SharedLibraryLoader.loadFile(SharedLibraryLoader.java:293) ~[gdx-jnigen-loader-2.3.1.jar%23107!/:?] {}
at com.badlogic.gdx.utils.SharedLibraryLoader.load(SharedLibraryLoader.java:124) ~[gdx-jnigen-loader-2.3.1.jar%23107!/:?] {}
at party.iroiro.luajava.util.GlobalLibraryLoader.load(GlobalLibraryLoader.java:62) ~[luajava-3.5.0.jar%23106!/:?] {}
at party.iroiro.luajava.lua54.Lua54Natives.<init>(Lua54Natives.java:139) ~[lua54-3.5.0.jar%23105!/:?] {}
at party.iroiro.luajava.lua54.Lua54.getNatives(Lua54.java:55) ~[lua54-3.5.0.jar%23105!/:?] {}
at party.iroiro.luajava.lua54.Lua54.<init>(Lua54.java:44) ~[lua54-3.5.0.jar%23105!/:?] {}
at com.example.examplemod.ExampleMod.<init>(ExampleMod.java:88) ~[%23202!/:?] {re:classloading}
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] {}
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?] {}
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] {}
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?] {}
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?] {}
at net.minecraftforge.fml.javafmlmod.FMLModContainer.constructMod(FMLModContainer.java:68) ~[javafmllanguage-1.20.1-47.2.0.jar%23198!/:?] {}
at net.minecraftforge.fml.ModContainer.lambda$buildTransitionHandler$10(ModContainer.java:123) ~[fmlcore-1.20.1-47.2.0.jar%23201!/:?] {}
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] {}
at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1796) ~[?:?] {}
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?] {}
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?] {}
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?] {}
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?] {}
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?] {}

Due to this I am unable to instantiate the Lua54 class.

Expected behavior
Luajava should successfully load the natives, resulting in no exception being thrown upon executing the following line of code:

var lua = new Lua54();

Platform:

• Arch: x86_64

• OS: Windows 10 64 bit

• Lua Version: Lua54

• IntelliJ IDEA 2021.2.4

• Gradle 8.1.1

• JDK 17

• LuaJava 3.5.0

Additional context
I am new-ish to gradle, so perhaps the actual issue lies in the build.gradle script (https://github.com/GHXX/luajavatest/blob/master/build.gradle), although the non-native parts of Luajava are being loaded properly it seems. Despite getting that to work, I cannot think of any other things to try at this point, even after a few days of digging through the internet.

Additionally, when packing all luajava dependencies (including the natives) via shading into the mod-jar and loading this outside of the development environment, so in a normal minecraft-client 1.20.1 installation, with forge, it loads successfully. This is not included in the minimal example. But the top-level directory structure of the working mod jar is shown in image 3.

Lua54 was the only LUA version I have tried. I also have not tried LuaJit.
The build-outputs are located in the /build folder. Interesting subfolders are classes/, classpath/ and libs/(which contains the output jar), and possibly others.

1. Run configuration screenshot:
   

2. Root of the exception chain:
   

3. Working jar when using it outside of the dev environment:
   

[gudzpoz]

Thanks for the detailed reproducing steps! It seems you are using ForgeGradle. As is mentioned in their documentation that "Non-Minecraft dependencies are not loaded by Forge by default in the development environment", have you tried replacing implementation with minecraftLibrary in the following line?

https://github.com/GHXX/luajavatest/blob/ceb7c887bcd32c0ff6a5a79f19eb91c18b730986/build.gradle#L160

[GHXX]

Yes, i have tried this, and i have just now double-checked, but the result appears to be exactly the same (SharedLibraryLoader.class.getResourceAsStream("/" + path) returns null as before and thus loading throws the same exception like before).

[GHXX]

Here is an even more simple example by @00asdf only using gradle and java17: https://github.com/00asdf/LuaJavaTest
The same error happens as in the minecraft forge case (essentially the same error, but of course the stacktrace doesnt mention minecraftforge). Running as java8, exactly the same error happens.

Setup steps (although this should be fairly standard):

1. Clone https://github.com/00asdf/LuaJavaTest
2. Open in IntelliJ IDEA (same version as before aka 2021.2.4, but also reproducible in 2022.3.3)
3. Rightclick Main, Run.. (Image 1) to create runconfigs
4. Change SDK version in the run configuration to JDK17 or Java17
   4.1. Set Project SDK in Project settings to 17 as well as the language level
   4.2. Set Gradle JVM Version to 17 also (Image 2)
5. Reload gradle project
6. Run and observe that the program crashes. (see exception below)

Exception:

Exception in thread "main" java.lang.LinkageError: Unable to find natives or init
	at party.iroiro.luajava.lua54.Lua54.getNatives(Lua54.java:57)
	at party.iroiro.luajava.lua54.Lua54.<init>(Lua54.java:44)
	at dev.asdf00.Main.main(Main.java:7)
Caused by: java.lang.IllegalStateException: com.badlogic.gdx.utils.SharedLibraryLoadRuntimeException: Couldn't load shared library 'lua5464.dll' for target: Windows 10, 64-bit
	at party.iroiro.luajava.lua54.Lua54Natives.<init>(Lua54Natives.java:145)
	at party.iroiro.luajava.lua54.Lua54.getNatives(Lua54.java:55)
	... 2 more
Caused by: com.badlogic.gdx.utils.SharedLibraryLoadRuntimeException: Couldn't load shared library 'lua5464.dll' for target: Windows 10, 64-bit
	at com.badlogic.gdx.utils.SharedLibraryLoader.load(SharedLibraryLoader.java:128)
	at party.iroiro.luajava.util.GlobalLibraryLoader.load(GlobalLibraryLoader.java:62)
	at party.iroiro.luajava.lua54.Lua54Natives.<init>(Lua54Natives.java:139)
	... 3 more
Caused by: com.badlogic.gdx.utils.SharedLibraryLoadRuntimeException: Unable to read file for extraction: lua5464.dll
	at com.badlogic.gdx.utils.SharedLibraryLoader.readFile(SharedLibraryLoader.java:137)
	at com.badlogic.gdx.utils.SharedLibraryLoader.loadFile(SharedLibraryLoader.java:293)
	at com.badlogic.gdx.utils.SharedLibraryLoader.load(SharedLibraryLoader.java:124)
	... 5 more

Image 1:

Image 2:

[gudzpoz]

It turns out jnigen does not support JPMS and uses SharedLibraryLoader.class.getResourceAsStream to load libraries. SharedLibraryLoader.class.getResourceAsStream actually works when the jnigen JAR is put on the class path. However, in the configuration of Forge MDK, all JARs are put on the module path, which eventually makes SharedLibraryLoader.class.getResourceAsStream search only in its own JAR instead.

https://github.com/libgdx/gdx-jnigen/blob/bc9e53afad3809e46dd7b0c5298ca76b26047718/gdx-jnigen-loader/src/main/java/com/badlogic/gdx/utils/SharedLibraryLoader.java#L182

I will try to come up with a solution (probably by submitting a PR in the jnigen repo). But before a new release of jnigen, I don't see an easy fix. (Maybe you can try to modify the runClient config to put the JAR on classpath? But I am not exactly sure how to do that anyways.)

---

By the way, the second reproducing repo has the dependency line wrong:

https://github.com/00asdf/LuaJavaTest/blob/8bedcbdd546665b6137a73a944f2d011e45a5f58/build.gradle#L20

(It misses the classifier suffix :natives-desktop.)

[GHXX]

By the way, the second reproducing repo has the dependency line wrong:

https://github.com/00asdf/LuaJavaTest/blob/8bedcbdd546665b6137a73a944f2d011e45a5f58/build.gradle#L20

(It misses the classifier suffix :natives-desktop.)

Ah yes, you are totally right, adding the :natives-desktop does make it work properly (in the non-minecraft forge case).

---

It turns out jnigen does not support JPMS and uses SharedLibraryLoader.class.getResourceAsStream to load libraries. SharedLibraryLoader.class.getResourceAsStream actually works when the jnigen JAR is put on the class path. However, in the configuration of Forge MDK, all JARs are put on the module path, which eventually makes SharedLibraryLoader.class.getResourceAsStream search only in its own JAR instead.

https://github.com/libgdx/gdx-jnigen/blob/bc9e53afad3809e46dd7b0c5298ca76b26047718/gdx-jnigen-loader/src/main/java/com/badlogic/gdx/utils/SharedLibraryLoader.java#L182

I will try to come up with a solution (probably by submitting a PR in the jnigen repo). But before a new release of jnigen, I don't see an easy fix. (Maybe you can try to modify the runClient config to put the JAR on classpath? But I am not exactly sure how to do that anyways.)

I'll go ahead and try adding the jnigen jar onto the classpath as you suggested. I am not sure if I'll be able to get this to work, but if I do then I will let you know.
As I said though, I am not certain whether this is possible (especially in a clean way), and thus I would appreciate if you could look into a more proper solution (as you already mentioned).

[GHXX]

Update on this: I did manage to get it to work (in another repository), but it involves unpacking all 3 luajava related jars (lua54 x2 and luajava) (and also the gdx-jnigen-loader jar) into the build/classes folder which essentially contains the compilation result of the mod. So quite a lot of hackery going on there (as I feared) but it no longer appears to fail which is a relief, and thus I expect the library to work fine, but I have not yet tested this.

In the future it would be nice to have a more proper solution for this, thus I'd suggest we keep this issue ticket open in the meantime as it essentially still is an ongoing problem.

[dayo05]

I highly suggest to pack native library to other method for Minecraft Mod development. They uses completely different way to load sub-jars and its resource so you can encounter unexpected issues like this.

Instead, I recommend System.load to load native library from mod initialization code. This method is easy to manage/control and easy to check where was wrong at dependency.

[dayo05]

you can consider to use Mixin when library is not allowed to change the directory of native library but it seems super overkill...

I have no idea about better solution, but to support loading native library from specified path can be solution

[GHXX]

@gudzpoz Looks like the gdx-jnigen pr was just merged, so to take away some of the stress i want to point out that we have recently been prototyping a custom lua vm as we needed some extra features, such as pausing, saving and restoring entire VM states
So, in hindsight, apologies for requesting this (as it might not be used afterall :( ), and no rush implementing this change, but it might still be useful for other people, who knows.

[gudzpoz]

Thanks! Not supporting JPMS is definitely a bug in this library (as well as jnigen) and there is nothing to be sorry about reporting it. So thank you again for submitting this issue! (But unfortunately we might need to wait until gdx-jnigen releases a newer version containing this change.)