mismi-s3: Multi-thread downloadRecursive

[erikd-ambiata]

This complies. Waiting to see if the tests pass.

Hard coded the number of forked threads to 10. Should that be a parameter?

[nhibberd]

Hard coded the number of forked threads to 10. Should that be a parameter?

Can we do similar to sync, expose a --fork with a default

[olorin]

Why the per-file concurrency on top of the per-chunk concurrency? Not sure if I missed a conversation somewhere, but I suspect this might actually be slower - increasing the concurrency by a factor of ten would make us more likely to hit rate-limiting.

[erikd-ambiata]

@olorin This is just a WIP. I think we probably need to collect file sizes as well and if the file sizes are small enough using multiple downloadSingle and for large files, use a single multipart download.

[olorin]

This is just a WIP. I think we probably need to collect file sizes as well and if the file sizes are small enough using multiple downloadSingle and for large files, use a single multipart download.

That sounds good. Would be good to have some empirical basis for whatever the split ends up being too.

[erikd-ambiata]

Might make sense to fix recursiveUpload first. More obvious how to do that.

[nhibberd]

As a first pass it would be fine to have a default of 1 with an option to override as well. That way we can start using it even if its not the most efficient at the moment

[erikd-ambiata]

When this does the right/smart thing, would we still want an option to specify the parallelism? If not it seems a bit weird to add it for now only to remove it later.