Quote:
Originally Posted by Ryan Why block bots at all? Wouldn't it make more sense to just email the administrator so they can go in and clear the problem?
I would have thought that blocking bots would lead to a reduction in rankings anyway since it looks like you don't want your site indexed at all. |
Having a spider grab your site while it's got spam links on it can be problematic. Blocking the spider temporarily is better in the long run. The spider doesn't have to know, especially if you send back some sort of transient error code to it, like a 503 error. Then it will basically ignore the result and spider you again later, hopefully after you've cleaned up the mess.