README.md

Spider

Python website crawler

Overview

This is an open source, multi-threaded website crawler written in Python. There is still room for a lot of improvements, so feel free to help out with the development.

Note: For now, this tool gathers links only.