Running a task on a particular dask worker

[rpitre]

I am using dask distributed scheduler to create workers on many machines. Each worker on each machine has resources attributes specifying for example the amount of RAM, the graphic card(s) installed, the number of cores, etc.. Is there a way to specify that a prefect task should only run on a worker with a particular set of resources ?

[cicdw]

Hi @rpitre - yes you can use Dask Worker Resources + Prefect Task tags to achieve this. See the comment in this doc here: https://docs.prefect.io/api/latest/engine/executors.html#daskexecutor

For posterity:

Note that if you have tasks with tags of the form "dask-resource:KEY=NUM" they will be parsed and passed as Worker Resources of the form {"KEY": float(NUM)} to the Dask Scheduler.

[asandeep]

Hi @cicdw

Just a query on top of this. Is it possible to apply tags during runtime?

To put in context:
I want to try a task with some resources eg. 100MB RAM. If during runtime an OutofMemory error is encountered, I want to increase the resource limit and retry the task with updated resource tag.

[cicdw]

At this moment tags are considered a part of the static configuration for a Flow, so unfortunately you cannot update their values on the fly.

[asandeep]

Got it. Thanks.

[rpitre]

Is NUM the only value type you can pass ?

[cicdw]

I believe Dask requires numerical values for resources, yes; but this shouldn't be restrictive.

[rpitre]

Maybe not restrictive but very inconvenient. For example I have tasks that must run on computers with a particular graphic card. With this restriction I cannot say resources={'GRAPHIC_CARD'='NVIDIA XXX'}, instead I have to arbitrarily attach a number to each card type ....

[cicdw]

I don't think it needs to arbitrary - could you do resources={"GRAPHIC_CARD_A"=1} because each worker has 1 card. You could pick that number to be the maximum number of tasks you want accessing GRAPHIC_CARD_A at any given time.

[rpitre]

Yes, I was a little quick in replying. Making the graphic card name the key will do it.