RSS-Spider » Development

Answering the questions I get at least once a week… “How can I setup an RSS feed for my site?”

Dave — Thu, 14 Jun 2007 05:27:44 +0000

This step by step comes from Design World Online and is reprinted with permission. The orginal document is located at http://www.designworldonline.com/ftp/dmm_pdf/DesignWorld_HowToRSS.pdf
FEED / BLOG CREATION

STEP 1. Find Service or Application
Design World recommends TypePad (www.typepad.com). It is very easy to set up a free 30-day trial and get started with your communication. If getting your IT department involved is required, you can even point these services to your own domain for seamless integration. If you have a Web site, blog, audio/video content or even photos, you can offer a feed of your content as an option. If you are using a popular blogging platform or publishing tool like TypePad, Wordpress or Blogger, you likely publish a feed automatically. There are also tools on the market that can help transform traditional web content into the right format for distribution. Simply creating an XML version of your content allows Aggregators the ability to read, but this entails some knowledge of XML syntax. Another method is PC-based software that allows blogging with associated feeds to be automatically published to a specific website location.

STEP 2. Enter Data
The more frequent the better! Your readers and search engines like constantly updated content.

STEP 3. (Optional) Enhance your Feed
There are services like FeedBurner (www.feedburner.com) that allow you to track statistics on your feed that include subscribers, hits and other good stuff.

STEP 4. Required! Tell us about your Feed*
Once your up and running, go to http://www.rss-spider.com/fsb.php and submit your feed address so we can subscribe to your feed and keep apprised of your news automatically. You post and we redistribute immediately. You gain the exposure of the RSS-Spider with no hassle.

* Edited out Design World’s email address since 99.999% of user submitted feeds have no relation to design engineering.

DIY RSS-Spider clones

Dave — Sun, 28 Jan 2007 05:23:23 +0000

This thread has been moved to BuildYourOwnSearchEngine.net

I’ve been getting a ton of emails from people all over the world asking me to share how I built RSS Spider.com… several people even offered to “partner” with me if I would build them a version of RSS-Spider for their language. Well folks. Sorry… I don’t have the time to build everyone an RSS Spider, however, I will over the course of the next few months post the basics of a Do It Yourself RSS-Spider clone. But first lets list some requirements.

1. A good knowledge of PHP, MySql, Apache & Cron

2. A beefy server with at least 100 gigs of hard drive space. Those drives should also be uber fast

3. Two secret ingredients… one called MagPie RSS and the other called Sphinx Search.

4. An understating of how to compile source code

5. A butt load of free time.

I’d advise starting off by finding yourself a development server and installing Linux on it. I do not advise using Windows… I’m not saying you couldn’t use windows on it but everything I’ve done is on a linux box. It’s very easy to setup a bare bones Linux box running everything you need to build a spider. Windows has way to many services and other things running by default than I like. Plus it doesn’t come with any sort of compiler and your going to need that

Built for Speed… new Indexing engine goes online…

Dave — Sun, 12 Feb 2006 03:22:56 +0000

Over the past two weeks there was a major drop in the speed at which searches were being returned. The MySql database hit a line in the sand somewhere and once crossed search speed suffered. At the time of this writing the database has over 5 million articles pulled from various RSS. The FULL_TEXT search has collapsed and searches for simple one word searchs like Clevealnd were taking 300+ seconds to return. Frankly I’m still amazed at the amount of page views we were getting at this time but looking at the search log I can see many people came in from the same IP address 4 or 5 times within a minute looking for the same thing. This says to me that they thought the site was slow or didn’t accept their querey so they clicked search again only to have to wait 4+ minutes! BAH!

As of 2006-02-11 20:01:02 RSS-Spider is now being powered by a new Full Text Index server called Sphinx. Searches that once took 300+ seconds to do now take under a second! Sphinx was simple to install and I’m seriously impressed with the overall speed gain!!! http://www.shodan.ru/projects/sphinx for more information!

What was Hot Yesterday!

Dave — Sun, 29 Jan 2006 23:23:49 +0000

Yesterday marked the launch of the Hot Words section of RSS-Spider. What this section does is mash up all the posts from any given day, sort all the words from that day and count the number of times any specific word appears. From there it takes the top 100 words as they appear and rank them in font size order.Â So on January 25th the system processed all the documents in the database that had a pubdate of January 24 (this date comes from the RSS feed that the spider pulled) and found that in the top 100 terms used on that day Alito, Bush, Iraq and War all came up…

Clicking on any one of these terms will pull up all the articles stored in the database for that day with that term.

Currently we are only processing English language feeds, but our next step is to add a Hot Words for German users.

Top Twenty Things people from Google are looking for on RSS-Spider

Dave — Sun, 08 Jan 2006 04:22:05 +0000

I love Google. Every time I turn around they’ve got some new and cool tool for using the web. A few months ago they introduced Google Site maps which allows web managers to upload sitemaps to Google so it knows where to spider. Recently Google updated the stats page of this system to show the top twenty search terms users searched for and found your site (in this case www.RSS-Spider.com) and clicked on. This saves me tons of time not having to go though all my weblogs and pull up this stuff manually.

I have to admint that most of the terms below I know nothing about… Robot Rage keeps popping up all over the place on my logs. I did a little digging and it’s actually a neat little free Flash game where you can build a “robot” (more like a car since you control it) and battle other players in an areana. You battle for points and more points you get the more “upgrades” you can make to your machine. Very very similar to the Battle Bots idea except no cheesey host.

Anyhow.. the list…

1 robot rage
2 robot rage rearmed cheats
3 amanda wenk
4 secret santa poems
5 robot rage cheats
6 amelle berrabah
7 natasha marley
8 son of dork website
9 son of dork official website
10 son of dork pictures
11 viezone
12 robot rage rearmed
13 roadrunner united guitar tabs
14 2hot4blog
15 sugababes ugly chords
16 son of dork
17 cheats for robot rage
18 brent corrigan
19 robot rage hacks
20 my hunps