Seriously sad shape
Put some time into the spider tonight. It’s been in seriously sad shape for the last 10 or so months. Having a baby really put everything on the back burner. Had several big screw ups in coding which caused everyone signing up for a new account to encounter a 404 page when trying to validate themselves. The link was firing off to my development server which is behind a firewall. Fixed that, fixed a problem with password generation. However, this was nothing compared to the problem I found and fixed just now. About two weeks ago I noticed that the amount of content seemed to have peeked at around 15,000 and that the number of new feeds wasn’t sky rocketing like it had in the past and had also peeked at 62,000. Well winds up somewhere along the line I was testing the spider and FORGOT to pull out the text code… soooooooooooo the content spiders were scanning the same 500 websites.
ugh