[BUG] Fail to walk through a FlyteDirectory through os.listdir

[JiangJiaWei1103]

Describe the bug

Currently, FlyteDirectory doesn't support os.listdir with a remote S3 path (fortunately, a local path works!). The error is shown as follows:

Though FlyteDirectory provides listdir method itself, I just think os.listdir could provide a better UX.

Expected behavior

A FlyteDirectory with a remote S3 path can be walked through by os.listdir, as commented here

Additional context to reproduce

Create a demo dir on minio and add a text file first. Then, run the following script:

import os
from flytekit import task 
from flytekit.types.directory import FlyteDirectory 


S3_DIR = "s3://my-s3-bucket/demo


@task
def create_fd() -> FlyteDirectory:
    fd = FlyteDirectory(path=S3_DIR)

    entities = os.listdir(fd) 
    for e in entities:
        # Suppose there's no subdir
        with open(os.path.join(S3_DIR, e), "r") as f:
            print(f.read())

    return fd


if __name__ == "__main__":
    fd = create_fd()
    print(fd)

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[pingsutw]

People can use s3fs to list the files if they want: s3fs GitHub. I’m not sure if we need to support it, as it’s unclear whether we should return S3 paths or local paths when listing a Flyte directory

[JiangJiaWei1103]

That makes sense to me. Thanks for your quick reply!

[eapolinario]

@JiangJiaWei1103 , you should be able to run FlyteDirectory.listdir() instead of os.listdir. Is that enough for your use case?

[JiangJiaWei1103]

Hi @eapolinario,

Yes, FlyteDirectory.listdir() is fairly good. I just wondered whether supporting os.listdir would provide better UX or not.

Thanks so much for your reply and please feel free to close it!

[eapolinario]

It is a good question, but other libraries have specialized listdir to handle the case of remote files, e.g. fsspec.