Description
The Cloudflare crawler (iyp/crawlers/cloudflare/__init__.py) currently has a significant scalability issue that prevents it from processing HostName nodes. The code responsible for this is commented out with a TODO due to memory constraints, meaning the crawler is incomplete and missing valuable data.
The Problem
In the DnsTopCrawler class, initialization attempts to load all target node IDs into memory using Python dictionaries:
# iyp/crawlers/cloudflare/__init__.py
self.domain_names_id = {node['dname']: node['_id'] for node in existing_dn}
# self.host_names_id = {node['hname']: node['_id'] for node in existing_hn}Why this is critical
Memory Usage (OOM Risk):
Loading millions of HostName objects into a dictionary (self.host_names_id) consumes a massive amount of RAM (O(N) complexity). On large datasets, this will inevitably cause the process to crash with an Out-Of-Memory error.
Data Loss:
Because of this risk, the feature is currently disabled (commented out). This means the Internet Yellow Pages is completely missing Cloudflare ranking data for specific hostnames, limiting the dataset's value.
Code Location
File: iyp/crawlers/cloudflare/__init__.py
Lines: 43–50
# TODO Fetching data for HostName nodes does not scale at the moment.
self.host_names_id = dict()
# existing_hn = self.iyp.tx.run(...)
Suggested Solution
The crawler needs to be refactored to move away from pre-loading all IDs. Instead, it should:
1. Iterate in Batches
Use a generator or a paginated query to fetch HostName nodes from the database in small chunks (e.g., 1000 at a time).
2. Process & Release
Process each batch (query Cloudflare, save results), then immediately release memory before fetching the next batch.
Expected Result
This refactor would allow the crawler to run on datasets of any size while maintaining constant memory usage, making HostName crawling scalable and production-ready.
I would be happy to raise a pr that will fix this issue
Description
The Cloudflare crawler (
iyp/crawlers/cloudflare/__init__.py) currently has a significant scalability issue that prevents it from processingHostNamenodes. The code responsible for this is commented out with aTODOdue to memory constraints, meaning the crawler is incomplete and missing valuable data.The Problem
In the
DnsTopCrawlerclass, initialization attempts to load all target node IDs into memory using Python dictionaries:Why this is critical
Memory Usage (OOM Risk):
Loading millions of HostName objects into a dictionary (
self.host_names_id) consumes a massive amount of RAM (O(N) complexity). On large datasets, this will inevitably cause the process to crash with an Out-Of-Memory error.Data Loss:
Because of this risk, the feature is currently disabled (commented out). This means the Internet Yellow Pages is completely missing Cloudflare ranking data for specific hostnames, limiting the dataset's value.
Code Location
File:
iyp/crawlers/cloudflare/__init__.pyLines: 43–50
Suggested Solution
The crawler needs to be refactored to move away from pre-loading all IDs. Instead, it should:
1. Iterate in Batches
Use a generator or a paginated query to fetch HostName nodes from the database in small chunks (e.g., 1000 at a time).
2. Process & Release
Process each batch (query Cloudflare, save results), then immediately release memory before fetching the next batch.
Expected Result
This refactor would allow the crawler to run on datasets of any size while maintaining constant memory usage, making HostName crawling scalable and production-ready.
I would be happy to raise a pr that will fix this issue