This thread has been moved to BuildYourOwnSearchEngine.net

I’ve been getting a ton of emails from people all over the world asking me to share how I built RSS Spider.com… several people even offered to “partner” with me if I would build them a version of RSS-Spider for their language. Well folks. Sorry… I don’t have the time to build everyone an RSS Spider, however, I will over the course of the next few months post the basics of a Do It Yourself RSS-Spider clone. But first lets list some requirements.

1. A good knowledge of PHP, MySql, Apache & Cron

2. A beefy server with at least 100 gigs of hard drive space. Those drives should also be uber fast

3. Two secret ingredients… one called MagPie RSS and the other called Sphinx Search.

4. An understating of how to compile source code

5. A butt load of free time.

I’d advise starting off by finding yourself a development server and installing Linux on it. I do not advise using Windows… I’m not saying you couldn’t use windows on it but everything I’ve done is on a linux box. It’s very easy to setup a bare bones Linux box running everything you need to build a spider. Windows has way to many services and other things running by default than I like. Plus it doesn’t come with any sort of compiler and your going to need that