Distributed Learning with Queue and pytorch-lightning

[nicoloesch]

Hi everyone,

I am currently preparing my model and my data loading for DDP. Luckily, this is rather straight-forward in pytorch-lightning. However, I stumbled upon a mismatch between the functionality of pytorch-lightning and the Queue w.r.t. to DistributedSampler.

The documentation of pytorch-lightning mentions the following:

When running under a distributed strategy, Lightning handles the distributed sampler for you by default

# Don't do in Lightning...
data = MNIST(...)
sampler = DistributedSampler(data)
DataLoader(data, sampler=sampler)

# do this instead
data = MNIST(...)
DataLoader(data)

Under the hood, they reinstantiate the Dataloader with the DistributedSampler automatically meaning that the DataLoader has now the respective sampler and all other attributes as well.

However, in the documentation of the torchio.Queue. it is mentioned that the Queue requires the DistributedSampler in the case of DDP.

My question now is the following: Does the Queue require the DistributedSampler in the case of DDP, if the overarching DataLoader encompassing the torchio.Queue is already "equipped" with it? Or should I instantiate it twice (once myself and once automatically by pytorch-lightning), which I feel is probably not the right approach for it?

I tried to follow the path of the respective DistributedSamplers and DataLoaders but my knowledge about data loading reached its limit, so I am seeking for advise from someone who has more experience in this!

Thank you in advance for your help!
Nico

Edit

It appears that the pytorch_lightning.Trainer has the flag use_distributed_sampler, which determines if the respective DataLoader should be wrapped or not. As a result, I will continue with setting this to False and use the DistributedSampler in the torchio.Queue as required!

[fepegar]

Thanks @nicoloesch for opening the discussion and especially for sharing your findings in the note!

Pinging @hsyang1222 in case he has any comments.

[hsyang1222]

@nicoloesch

Q. Does the Queue require the DistributedSampler in the case of DDP, if the overarching DataLoader encompassing the torchio.Queue is already "equipped" with it?
A . Unfortunetly, no. Driving way of torchio.Queue is very speciallized to make patch efficiently in single GPU environment so that pytorch lighting with DDP could not used directly.

I am going to implement pytorch-lightining-support DDP dataloader in torchio.Queue. When I done, I will notice you.