Execute scripts in a work dir without copying them

[spine-o-bot]

In GitLab by @ererkka on Nov 17, 2020, 10:17

Sometimes a unique work directory might be needed for a script execution, but the script itself (nor any other files) need to to be copied there. Example is an externally developed Python program, which writes some outputs to files we cannot control (e.g. some hard-coded ouput.dat in there). In this case we merely would need to call the interpreter and the script in it’s original location from the created work dir. For a PythonToolInstance this would mean roughly speaking the following:

self.program = '/path/to/python'
self.args = [
    '/path/to/my_project/my_script.py',
    '/path/to/my_project/input.dat'
]
self.basedir = '/tmp/my_script__XXXXXXX__toolbox/'

This would be in a kind of hybrid solution between execution in work directory or source directory, and could of course only be applied to local execution. I still don’t know if this could replace one of the existing options. What do you think?

[spine-o-bot]

In GitLab by @manuelma on Nov 17, 2020, 10:26

I think that would be really easy to implement, because now we copy the file so we can call Popen("my_script.py", *args, workdir="workdir"), which works because my_script.py is in "workdir". If we didn't copy, then we'd need Popen("path/to/original/my_script.py", *args, workdir="workdir") and it would work all the same.

[spine-o-bot]

In GitLab by @ererkka on Nov 17, 2020, 10:39

Right, maybe biggest problem is how this is shown in the UI and what are the options we want to have. Benefits are less copying of files when it’s not needed.

[spine-o-bot]

In GitLab by @jkiviluo on Nov 17, 2020, 11:15

Something like this?

[spine-o-bot]

In GitLab by @jkiviluo on Nov 17, 2020, 11:18

Those blue things try to be buttons.

When you run in program directory, you cannot use persistent work directories or create new work directories, so they should be grayed out.

[spine-o-bot]

In GitLab by @jkiviluo on Nov 17, 2020, 11:21

There is a further issue when running in program directory - you can have the same tool twice in the workflow and they could try to run at the same time and mess each other (I already have this situation). We can leave this problem to the user, but there could also be a warning.

[spine-o-bot]

In GitLab by @manuelma on Nov 17, 2020, 11:25

I like that. We need to think how to make it compact, because some users won't need much of it. How often we'd be using tools that have hard-coded writing operations?

[spine-o-bot]

In GitLab by @ererkka on Nov 17, 2020, 11:32

Examples are e.g. TIMES and Balmorel as well as the AMIRIS agent-based modelling system (will be used in our TradeRES H2020 project)

[spine-o-bot]

In GitLab by @jkiviluo on Nov 17, 2020, 11:36

• 'program in source' could also be 'tool in source'
• 'copy program' could then be 'copy tool'

Keeping it smaller: you can display the three last things only when work directory is selected (and hide them when 'execute tool in program directory' is selected. But maybe it's still nicer to just gray them out.

[spine-o-bot]

In GitLab by @manuelma on Nov 17, 2020, 11:46

Examples are e.g. TIMES and Balmorel as well as the AMIRIS agent-based modelling system

So I guess quite often. It's surprising though, I don't think it's a very good practice.

[spine-o-bot]

In GitLab by @jkiviluo on Nov 17, 2020, 12:27

Fully agree, but they are venerable tools started when the systems were monolithic.

[spine-o-bot]

In GitLab by @manuelma on Nov 17, 2020, 12:34

You're right, nobody is safe from the future.

[spine-o-bot]

In GitLab by @PekkaSavolainen on Dec 22, 2020, 13:00

moved from spine-tools/spine-items#40

[DillonJ]

Seems like the discussion has become quite philosophical - converting to discussion for now.