todo

Todo for spider!

• Main process handles how many threads it runs, and tunes it for maximum efficiency
• Individual spider processes download source, capture urls from it, and save those to a pool
• Indexer (WTF) i dunno how to implement (actually indexes content from the body, removes common words, etc)
• Database design (this is important)
  • pool for urls to spider
  • pool for spidered urls
  • indexes
• Database manipulation library