Experimental async_controller (luac)

[TheEt1234]

Repo: https://github.com/TheEt1234/async_controller/tree/main

Problems:

Ratelimiting

basically solved, just need to figure out what to do if someone spams like 50 of them in a mapblock... i think mesecons_debug can figure that one out it can't

And also needs more testing

Discussion (has some important context)

starts from https://discord.com/channels/513329453741637637/719950700485935145/1214324327185649684

For those without access to Discord, the discussion mentioned above, has been added to this thread.

[Klaranth]

et
i did a thing https://github.com/TheEt1234/async_controller/tree/main [needs to get seriously tested]

Huhhila
no tests implemented yet?

et
Do i need to do that
probably... hmm
well, i guess testing every single thing intended/working in the enviroment can't hurt, todo: tomorrow
Also the code is dogwater, because i have no idea how to propertly share enviroment variables

et
ok so i have made """tests"""
https://github.com/TheEt1234/async_controller/blob/main/Tests.md and also added a metatool and the readme is actually readable now
so is the code, instead of 50 arguments, i just have a table like a sane person

should i make an issue at https://github.com/pandorabox-io/in-game/issues for this though it will still require a lot more testing than what i have

now i guess i am gonna try making a survival friendly copy paste tool

SX
also added a metatool; If so then isn't this already survival friendly copy/paste tool?

et
i meant uh, copy pasting buildings; that wasn't related to the async luac

SX
Oh well then go ahead, yeah I thought it was related to async luac stuff.
I guess main concerns will be how many threads one can spawn with this thing and how it controls resource usage.
But yeah, guess would be worth it to open in-game repo issue for discussion.

et
and how it doesn't controls resource usage.
it time outs like normal except i made the maxevents be 10x more (changable)
and also burns like normal
many threads one can spawn with this thing
given enough async controllers, infinitely many, dont know how i am gonna limit that, maybe certain amount of async controllers per mapblock? idk

SX
It'll probably still be good Proof Of Concept, resource limits for actual multiplayer use could almost sure be added later fairly easily if basic thing works well enough. If it works fine it would likely be better way even if limited similar to current pandorabox luac.

et
also it is still gonna get ratelimited by mesecons_debug if you spam digiline_send i think so thats at least nice
it would likely be better way even if limited similar to current pandorabox luac.
It is already limited that way, except maxevents by default are 10 times bigger

SX
is it at most single concurrent thread per controller?

et
single minetest.handle_async call per event/luac trigger
so yes(?)

SX
So it can be more than one concurrent thread per controller if say you send 2 events within 1 second and execution takes more than a second, then single controller will run 2 parallel programs right?

et
yes
maybe that could be limited, there are 2 ways to do that:

1. delay the event (i don't know how)
2. drop the event

SX
or might work well enough to just increase heat value by total utilized time and let it burn :luacontroller_burnt:

et
by total utilized time
in what, like increasing it by the amount of seconds it took? or miliseconds?

SX
yeah, counting how long each thread used up, summing up to heat value when thread returns. that would allow utilizing overheat function that already exists and through that might also already offer good enough configuration. of course it needs some balancing but as a concept it might work well enough without having to limit threads (if thread spawn rate is anyway somewhat limited).

et
(if thread spawn rate is anyway somewhat limited).
Technically it's limited by 20 threads/second
yeah, counting how long each thread used up, summing up to heat value when thread returns
so just heat=heat+execution_time

SX
basically that yeah, just have to protect measurement well enough so it is not too easy to spoof it. probably have to measure it inside the worker anyway to get accurate timings.

et
so just heat=heat+execution_time
oh... its gonna be more complicated than that....
because the only way to modify heat is thru mesecon.do_overheat [heat = heat+1]
mesecon.cooldown [sets heat to 0]
how would that be spoofed....
if its like
local start=get_time()
run_code()
local end_=get_time()
local time_took=start-end_

SX
Basically that yes, and for spoofing it you have to make sure that whatever is inside run_code() can't gain access to local variable start

et
that would be seriously bad if that could happen

SX
I've not played with minetest workers, no idea how well those are protected. If there's a way to gain access to debugger then that at least must be disbled. And of course any unsafe functions that might allow executing direct instructions/bytecode which in turn would be basically same as debugger access.

et
luacontroller basically temporarily restricts enviroment so that no outside things can be accessed/modified
and yes minetest async does have working debug that's why i was able to do this whole thing

so when adding minetest.debug(time_took)
repeat until timeout -- took around 183 microseconds

repeat
string.rep("a",64000)
until timeout
-- around 19 000 microseconds

-- these are really random, it takes hoewer much my computer feels like

@sx how much more should the bottom one be punished than the top one, i can only increase the heat of the luacontroller by +1 (so heat can't be like 19.35+math.pi)

SX
Seems like you do have default lua controller instruction limiter enabled which is probably main reason for these results, I mean first one hits that limiter before it can do much work. I guess might be good to bump up or even remove instruction limiter to test properly.
Could also see what things like s = "x" while 1 do s = s .. "x" end does.

et
i am testing "how much resources can an async controller use up before it timeouts"
and based on that, how should i punish the async controller for consuming resources

et
i should probably disallow this...
yeaahh... ig i can limit the maximum amount of digiline messages sent

et
ok i have done that

SX
Though async has nothing to do with end results really and regular luac does the same as long as you do not limit outgoing messages.

iceyou
yeah most likely it doesn't affect it but I have no idea how frog implements async functionality into it

SX
Async processing in minetest is through workers that will be synced with main thread through basic locks, basically end results / responses from async function is handled just like fully synchronous single thread program.

SX
There's no real threading available, just asynchronous workers that take a task, can be executed and then main thread will check it each server step and grab results if worker has finished its thing. Worker itself does not have access to world or game main thread-

et
see readme, if the async luac times out or errors it will still send out the messages
also i just realised i spelled environment wrong... let's see if somebody notices
but i've limited it to 150/event, rest get dropped

et
minetest.handle_async(async_function, callback,arg1,arg2,arg3...)

iceyou
yeah functionally this tells me right about nothing

et
does this tell you anything https://github.com/TheEt1234/async_controller/blob/main/init.lua#L610 and yes before there were like 100 arguments given to the minetest.handle_async function

SX
That timing out or errors are simply just decision for how to proceed, basically digiline_send is exactly same as it is with regular lua controller: it will collect messages and send those after program is completed. Only thing that limits regular luac is instruction limits and error handling which just do not exist in async luac like they do in regular luac.

iceyou
(Idk how minetest.handle_async even works)

et
neither do i it asyncs
then when the async function returns the callback gets called

instruction limits do exist in async luac, they are just 10x bigger than in normal luac
i think it would be a good idea if we made a github issue for this.. or does it need more testing?
and also right it needs the ratelimiting

SX
Everything written here will be forgotten very soon unless it gets copied to github issue, which is why I did recommend creating one

et
well, it can be forgotten, because this is mostly discussing the implementation/limits of it

SX
Thing is we'll be discussing all that again when it comes to actually planning to use it on server

birdlover32767
imma try doing some suggestions

et
also i changed the burnt texture to be this
async_controller_burnt_top.png

@sx soo do i just create the issue and copy over some context

frogTheSecond
also one thing i forgot to mention (i think), i added a print log and remade the print() and added clearterm()

SX
A lot of discussion so would be nice. Additionally if there's a lot of specific implementation discussion then some of that detail could maybe also be their own issues (possibly linked to main issue) on your own repo too if you'd like to keep main idea issue a bit cleaner.

et
yes, just like the normal luac (at least feels like)

[TheEt1234]

i've rewritten it a bit (seperated the init.lua file into smaller files)

[TheEt1234]

the ratelimiting problem has been solved (hard 10ms [previously 20ms] limit per event) and some of the bad luac bugs are harder to do now

and also fixed that repeat mem.i = { mem.i, mem.i } until timeout bug that made the serializer go waaa by disallowing tables like that

[birdlover32767]

i feel like the recipe is a little too... cheap because of the computing power
maybe something like

mese block digiline integrated circuit
digiline mooncontroller digiline
silicon lump digiline bronze ingot
-> 1 async luacontroller
i mean you have plenty of spare ores, right

[S-S-X]

Could also make sense to use multiple lua controllers and maybe few ic's for recipe as it will be multi core in the end anyway.

Like 3 - 6 luac on bottom row(s), maybe ic or two on top of that and some copper for heat transfer (block if you feel it would be too cheap otherwise). For recipes I think it is good to think what it actually does and try to reflect that on recipe if possible.

[TheEt1234]

Sorry for not responding, i was kinda distracted with developing a new mod

I will change the recipe then, maybe to something like

(where luac = luacontroller, copp = copper block and heat = heatsink)

copp, heat, copp
luac, luac, luac
luac, luac, luac

[TheEt1234]

[SwissalpS]

I like this recipe a lot more.

Another kinda expensive item is digimese.

IIRC heatsink is from [digistuff]. Maybe you want an alternative recipe for servers that don't have that mod installed.

[birdlover32767]

digimese can be replaced with a mese block/whatever
heatsinks can be replaced with a mesecon

also better textures could be used (most namely the burnt async controller)

edit: here are my textures

[TheEt1234]

(most namely the burnt async controller)

:( i thought that one was good

but seriously burnt luacontrollers to me always looked like angry luacontrollers

[birdlover32767]

fine, the new burnt texture is

ok but seriously the real burnt texture (with a face) is

[SwissalpS]

I was about to say, at 16x16 pixels none of the text will be readable.

I don't mind a frowny face, but it might be a bit over the top. Just a shade darker red might be sufficient.

[TheEt1234]

i would love if it was more different from the digistuff component textures (why are they so overused) but i have no idea how to achieve that

Also birdlover, why add that text?

[TheEt1234]

also, should the async_controller include extra stuff in it's environment like a safe version of loadstring and pcall?
and also some vector stuff too(?)

(The main issue with pcall was that the execution could continue after the hook got wiped)

And loadstring with a character limit and proper sandboxing (not allowing bytecode, disabling jit, setfenv, etc.) would most likely be safe and allow for cool things

[TheEt1234]

there are also a lot of things in the minetest namespace that are safe

also idea: maybe there should be a node, that when placed next to the async_controller will allow access to this fancy environment(?)

[birdlover32767]

Also birdlover, why add that text?

that was for... april foolz

also here is a list of safe mt stuff
would be good if you erased the stuff from the "gray zone" (like minetest.get_modnames)

• ItemStack
• ItemStackMetaRef
• PerlinNoise
• PerlinNoiseMap
• vector
• bit (yes, this exists)
• minetest.get_modnames
• minetest.get_game_info
• minetest.is_singleplayer
• minetest.features
• minetest.has_feature
• minetest.get_version
• minetest.sha1
• minetest.sha256
• minetest.colorspec_to_colorstring
• minetest.colorspec_to_bytes
• minetest.encode_png
• minetest.urlencode
• minetest.string_to_privs (probably with a priv digidevice)
• minetest.privs_to_string (probably with a priv digidevice)
• minetest.formspec_escape
• minetest.explode_table_event
• minetest.explode_textlist_event
• minetest.explode_scrollbar_event
• minetest.inventorycube
• minetest.dir_to_facedir
• minetest.facedir_to_dir
• minetest.dir_to_fourdir
• minetest.fourdir_to_dir
• minetest.dir_to_wallmounted
• minetest.wallmounted_to_dir
• minetest.dir_to_yaw
• minetest.yaw_to_dir
• minetest.is_colored_paramtype
• minetest.strip_param2_color
• minetest.player_exists
• minetest.serialize
• minetest.deserialize (?)
• minetest.compress
• minetest.decompress
• minetest.rgba
• minetest.encode_base64
• minetest.decode_base64
• minetest.global_exists

[TheEt1234]

oh wow that's a lot of things, i will simplify the list

things will update as i go and implement the things from the list

Also edit: everything was reworked somewhat, here is how the functions are edited:

• string sandbox gets escaped
• arguments get checked for string length, if it exceeds 64000 it will error
• basic pcall sandboxing is applied (if not debug.gethook() then error("the hook went poof",2))

"+" - will add fully (maybe still needs to be reworked)
"+/" - will have to be reworked or limited in a different way
"-" - won't be happening
"~" - i feel like that's useless

    - ItemStack (why)
    - ItemStackMetaRef (why v2)
    - PerlinNoise  (Due to async limitations i can't)
    - PerlinNoiseMap (same thing)
    - minetest.get_modnames
    +/ minetest.get_game_info (don't expose path, since some people have their real life name as their username for some reason?)
    + minetest.is_singleplayer
    + minetest.features
    ~ minetest.has_feature
    + minetest.get_version
    + minetest.sha1
    + minetest.sha256
    + minetest.colorspec_to_colorstring
    + minetest.colorspec_to_bytes
    + minetest.encode_png
    + minetest.urlencode
    - minetest.string_to_privs (no)
    - minetest.privs_to_string (no)
    + minetest.formspec_escape (could be useful for another project of mine....)
    + minetest.explode_table_event (useful)
    + minetest.explode_textlist_event (useful)
    + minetest.explode_scrollbar_event (useful)
    + minetest.inventorycube
    ~ minetest.dir_to_facedir
    ~ minetest.facedir_to_dir
    ~ minetest.dir_to_fourdir
    ~ minetest.fourdir_to_dir
    ~ minetest.dir_to_wallmounted
    ~ minetest.wallmounted_to_dir
    ~ minetest.dir_to_yaw
    ~ minetest.yaw_to_dir
    ~ minetest.is_colored_paramtype
    ~ minetest.strip_param2_color
    - minetest.player_exists
    + minetest.serialize (i don't know how theese work but i will assume they are just some sort of pcall(table))
    + minetest.deserialize (same thing)
    + minetest.compress
    + minetest.decompress
    + minetest.rgba
    + minetest.encode_base64
    + minetest.decode_base64
    - minetest.global_exists

outside of minetest:

+ pcall = safe.pcall
+ xpcall = safe.xpcall
+ loadstring = safe.get_loadstring(env)
+ bit = table.copy(bit)
+ vector = table.copy(vector)

[SwissalpS]

Some of those would work well as [mooncontroller] libraries.
I might c/p some of your code for https://github.com/SwissalpS/minetest_mooncontroller_libraries once you are done ;)

[SwissalpS]

ItemStack() could be interesting to more easily inspect an item's meta. Not sure though if it would work well constructing an ItemStack from info given by a lua- or detector-tube.

[TheEt1234]

ItemStack() could be interesting to more easily inspect an item's meta. Not sure though if it would work well constructing an ItemStack from info given by a lua- or detector-tube.

You would need to receive an item stack with that meta info right? (or maybe its the item string that contains it?)
digilines makes sending/receiving userdata impossible

(If luacontrollers could send something hard to serialize through digilines that would probably cause crash bugs on many digilines devices)

Maybe there could be a function to construct a table from an item string but i don't think a full ItemStack is possible because of async limitations

[TheEt1234]

so i just realised how horrid my code was (i was noob with async and didnt think register_async_dofile was worth the effort)

Expect a huge change soon (mostly just to make reading the code easier)

[TheEt1234]

actually no it would be way too much to basically rewrite this

[TheEt1234]

also i forgot to mention, minetest.sha256 is going away too as i think thats a 5.9.0 thing(?)

[S-S-X]

is a little concerning (takes 50 ms, i wonder how actually)

Is pcall hard requirement in async environment. Like I've no idea but it is possible it might not be... of course not having it would also mean dropping some good features.

[TheEt1234]

is a little concerning (takes 50 ms, i wonder how actually

Is pcall hard requirement in async environment. Like I've no idea but it is possible it might not be... of course not having it would also mean dropping some good features.

what do you mean by hard requirement and async environment

i just thought pcall would be a cool feature and i saw a mesecons pull request that added a safe version of it (the problem with unsafe pcall was that you could evade the "Code Timed Out" errors and the timeout hook would no longer work once that error got caught)

also now i am pretty sure that 50ms is caused by all of the safe pcalls error'ing because code timed out

and i am pretty sure it's fixable by not allowing you to basically chain pcalls infinitely

[S-S-X]

what do you mean by hard requirement and async environment

I mean main reason for pcall for luac programs is to not crash whole Lua VM (to bring down minetest server) with luac program error.
Async environment is completely separate and runs under its own VM so pcall might not be that hard requirement. Unless it somehow propagates errors to main Lua environment.

I was thinking would it be possible to disable all pcalls when code is about to get executed in async env / separate vm.

[TheEt1234]

what do you mean by hard requirement and async environment

I mean main reason for pcall for luac programs is to not crash whole Lua VM (to bring down minetest server) with luac program error.

The sandbox needs to use pcall for the user code

Async environment is completely separate and runs under its own VM so pcall might not be that hard requirement. Unless it somehow propagates errors to main Lua environment.

I was thinking would it be possible to disable all pcalls when code is about to get executed in async env / separate vm.

No minetest would crash if there's an error in the async environment

[TheEt1234]

also, running 200 async_controllers (yes literally 200) with the code:

interrupt(0)

<uhh this code would actually make the normal luacontroller OOM or freeze the server, but async_controller has a proper time limit alongside an instruction limit so it won't happen, i don't want to leak it here, if you want to know this see one of the mesecon issues i've linked>

My fps started jittering like crazy and cpu usage was like at 100% but max lag was still at 0.2~0.3

But the real problem could arise when serializing memory or doing massive amounts of interrupts/digiline messages because that gets done after the execution and isn't ratelimited

So the serializer abuse bug [that i cannot reproduce] could probably be used in the async_controller

[S-S-X]

Any interrupt stuff does not have to be fixed because there's configuration for its behavior.

Anything you can just push to queue can create issues, for interrupt you can disable queue so for 200 luacs it is always at most 200 active interrupt any given time.

[S-S-X]

For serialization issues make sure you validate asynchronously and only return verified good results that have been already serialized. That takes overhead away from main thread and amount of data can be counted toward possible per luac execution limits.

[TheEt1234]

Any interrupt stuff does not have to be fixed because there's configuration for its behavior.

Anything you can just push to queue can create issues, for interrupt you can disable queue so for 200 luacs it is always at most 200 active interrupt any given time.

the issue wasn't with interrupts but the fact that i have interrupts + laggy code

this issue can be solved by setting the execution time limit 5ms instead of 10ms for example, or like tracking down the lag machine (i have made a basic tool to track them down)

[TheEt1234]

For serialization issues make sure you validate asynchronously and only return verified good results that have been already serialized. That takes overhead away from main thread and amount of data can be counted toward possible per luac execution limits.

that's what the luacontroller does in the remove_functions function (functions cannot be serialized and will crash minetest)

the current problem is that the serialization can simply just take too much time, this problem can actually be solved by imposing a time limit on that as well

wait i have done that before, i just threw the solution out because i thought it was un-needed, oops :p

i don't know how to reproduce it because the steps to reproduce that issue are private

[S-S-X]

i have made a basic tool to track them down

Additionally it would be good to integrate reporting for mesecons_debug mod

[TheEt1234]

i have made a basic tool to track them down

Additionally it would be good to integrate reporting for mesecons_debug mod

I can't figure out how this mod works, but the only thing that my mod (and same with all the other controllers i think) seems to lack support for is their node timer override

also my "basic tool" is just logging where an event happened and how many ms it took

[SwissalpS]

Have you tested/considered how this node works when e.g. jumped with jd?

[S-S-X]

I can't figure out how this mod works, but the only thing that my mod (and same with all the other controllers i think) seems to lack support for is their node timer override

It actually collects data about any mesecons/digilines activity everywhere and adjusts delays for queues based on usage per mapblock. For this to work it somehow needs information about how long program execution actually took but it does not take into account that no main thread time were used.

For other synchronous devices / wires / whatever stuff it is easy for debug mod to gain this information, anything asynchronous however will hide this information so it can't really know anything about async luac unless there's some kind of reporting from async luac to debug mod.

[TheEt1234]

I can't figure out how this mod works, but the only thing that my mod (and same with all the other controllers i think) seems to lack support for is their node timer override

It actually collects data about any mesecons/digilines activity everywhere and adjusts delays for queues based on usage per mapblock. For this to work it somehow needs information about how long program execution actually took but it does not take into account that no main thread time were used.

For other synchronous devices / wires / whatever stuff it is easy for debug mod to gain this information, anything asynchronous however will hide this information so it can't really know anything about async luac unless there's some kind of reporting from async luac to debug mod.

When activated through the "execute" button, the mystery code i am not gonna show, but it seems there has been something done about it(?) since it says load:4: string length overflow but still caused a big lag spike, but mesecons_debug didn't seem to care (at least from /mesecons_hud)

BUT when activated by another luacontroller with the code:

if event.type=="program" then

digiline_send("","")
end

It noticed something was fishy and registered a usage of 110 440 us/s

So maybe it only monitors digilines and digiline/mesecons/luac related node timers

So... should i just add the mesecons_debug's node timer override if mesecons_debug is avaliable (mooncontroller might have the same issue)

[birdlover32767]

please trim the worldpath!!!!!!

[TheEt1234]

please trim the worldpath!!!!!!

that image is blank for me, but i think you were reffering to timeout errors... well... the mod path can be useful for identifying what exactly caused the timeout

[TheEt1234]

I kinda want to make something where a user can make a library, and an async_controller can import it

should i do this and how should i go about doing it, and where should the libraries be stored

[SwissalpS]

look at mooncontroller and the open issues.
Basically the libraries are mods that register themselves with the controller.

Re: wordlpath: everything before the mod-dir can be trimmed: /some/path/to/mods/mod/file.lua -> mod/file.lua

[SwissalpS]

or if you mean libraries like advtrain supports (user generated) modstorage would be a modern option for storing them. That could potentially allow all users to use each others code. Depending on the complexity you want to add, you could have a system with privs for reading and writing global code and you could also allow users to mark their code as private.

[birdlover32767]

look at mooncontroller and the open issues. Basically the libraries are mods that register themselves with the controller.

Re: wordlpath: everything before the mod-dir can be trimmed: /some/path/to/mods/mod/file.lua -> mod/file.lua

so basically trim everything before the 2nd-to-last /
that requires you don't use slashes as error messages

[TheEt1234]

so basically trim everything before the 2nd-to-last / that requires you don't use slashes as error messages

how do i do that and make it so that like... uhh.. when you do error("/funny/heh") it doesn't get edited

[SwissalpS]

one way is to use local mp = core.get_modpath(<yourmodname>) then local base = mp:sub(1, <total length minus length of your modname>) then replace occurances of base with ""

[TheEt1234]

one way is to use local mp = core.get_modpath(<yourmodname>) then local base = mp:sub(1, <total length minus length of your modname>) then replace occurances of base with ""

oh cool, that works

[TheEt1234]

also, now that i think about it adding user libraries might be a bad idea (because i feel it would be too complex to implement)

so i guess now i should focus on like... security and improving the source code

[TheEt1234]

So i've rewired/rewritten the async_controller mod a bit, namely to use minetest.register_async_dofile instead of passing around an async_env variable

Also breaking changes:

• env_plus is enabled by default
• the server table does not exist anymore, conf is now it's own thing and server.us_time has been moved to minetest.get_us_time in the env_plus

Debugging improvements: you can now see how much the callback took (aka the part that processes all the digiline_sends, interrupts and saving memory) if you have enabled it in the source code

uhh holdon saving memory appears to crash (?) maybe i broke something recently again nevermind just a really really dumb mistake, fixed

[TheEt1234]

(Need to fix a really silly crash bug, eta: tmrw)

fixed lol, why did i wait 3 weeks to post that update

[TheEt1234]

should i rewrite it to use my sandboxing mod

[TheEt1234]

looking back at this - i should probably rewrite it to use my sandboxing library but i am feeling lazy - so maybe later