Built for Speed… new Indexing engine goes online…
Over the past two weeks there was a major drop in the speed at which searches were being returned. The MySql database hit a line in the sand somewhere and once crossed search speed suffered. At the time of this writing the database has over 5 million articles pulled from various RSS. The FULL_TEXT search has collapsed and searches for simple one word searchs like Clevealnd were taking 300+ seconds to return. Frankly I’m still amazed at the amount of page views we were getting at this time but looking at the search log I can see many people came in from the same IP address 4 or 5 times within a minute looking for the same thing. This says to me that they thought the site was slow or didn’t accept their querey so they clicked search again only to have to wait 4+ minutes! BAH!
As of 2006-02-11 20:01:02 RSS-Spider is now being powered by a new Full Text Index server called Sphinx. Searches that once took 300+ seconds to do now take under a second! Sphinx was simple to install and I’m seriously impressed with the overall speed gain!!! http://www.shodan.ru/projects/sphinx for more information!