RSS-Spider

Actually they all taste like burning…

Last night we experienced an 8 hour outage during our weekly database backup & purge.  The database of headlines has grown so massive that it took way too long to purge out any old & “spammy” headlines.  The last time we did this purge was back in December and we only experienced a 15 minute outtage.  I was up till 6 am waiting for everything to finish.  If you had visited the site during this time you would have been directed over to the my.rss-spider.com page which was incorrectly listed as our forwarding page.  My.RSS-Spider.com is a project we’re working on which is in it’s infancy.  In a nut shell what it will do is allow users to have their own webspace to aggrogate RSS feeds from RSS Spider.  So instead of coming and searching for something every time you can simply go to your my.RSS-Spider.com home page and view all the new feeds matching your search criteria.  Again.. this is something thats way down the road, however, we are planning to have a Beta release in late March.

I’ve been noticing that some times it takes upwards of a minute or more to return a search. This ain’t good. Looking into it I see that MySql is attemping to do UPDATES and SELECTS at the same time. The spider is updating the RSS feed links to current time (last polled) while people are using the database. I’ve bundled all the updates into a single querey and added PRIORITY_LOW to the updates. Remove all other updates which weren’t doing anything meaningful ie: number of times an item was viewed on a returning search page… bah… crap…