I like the fact that verified Google crawlers have unlimited access to my sites and that I can block fake Google crawlers. Why can't I do the same for MSN bots? I'm constantly having to whitelist all their IPs. Can't even tell if they are fake msn bots either.
↧