index_files(path::S3Path) is very slow on large directories

[aisopous]

It is because isfile being run on each path returned by collect(walkpath) on this line.

One possible workaround would be to filter out paths which end in /. For a 3000 file path the difference is 0.8s vs 60s using isfile.

[nickrobinson251]

i wonder if we can trust the isdirectory::Bool field to tell us when an S3Path is not pointing to a file?
then we could maybe have something like

function index_files(dir::S3Path)
    isdir(dir) || throw(ArgumentError(dir, "Need an existing directory."))
    map(Iterators.filter(p -> !p.isdirectory, walkpath(dir))) do path
        return IndexEntry(path)
    end
end