Open
Conversation
This PR change the download function for the flickr and bing search class. Was added a new way to multithreading the data URL's for a fast download increasing a way better performance with big images searchs.
This PR change the download function for the flickr and bing search class. Was added a new way to multithreading the data URL's for a fast download increasing a way better performance with big images searchs. To test this change just run the idt init with all needed parameters and run idt build at the end, download time will be more faster than usual.
Change functions to add a multithread way to increase the download speed and performance, change just the duckgo and bing classes, others classes whos need a API key was not changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR changes the download function for the Flickr and bing search class. Was added a new way to multithreading the data URL's for a fast download increasing a way better performance with big image searches.
To test the improvement, first Initializing the IDT and create all needed parameters, and after that build the dataset requested, the time will be faster than usual, by the multithreading all download subprocess will work to download the images.
In local tests, this was the result of some ways to download a sample of just one class and a sample of 100 images sample per search in the changed classes. The "normal" represents the time without the multithreading, and others with the tests max_workers of threads run simultaneously like 5 and 10.
duckgo -
5 threads - 1:36
10 threads - 1:48
Normal - 04:05
bing -
5 threads - 1:06
10 threads - 0:49
Normal - 03:52
before start the build
after start the build
the use of multithreading on a pc with 8gb RAM like mine, don't have negative effects, the tools work great!