Conversation
Notices
-
I am endlessly annoyed by all the lists of "bad bots" that should be "blocked" from your site. it seems everyone keeps copying everyone else, repeating in their list generic tools like wget, a python library or even 'pyhon' and ancient bad biot that haven't been used for years (often a decade or more) because everyone was blocking them anyway. I have not found a single page yet that actually validates each item on the list by stating what is "bad" about its behavior (if it even exists any more)... there used to be a page like that (but that was a decade ago, too). Of course there are a few "known" bad bots that ignore robots.txt or just hammer your site (Alexa's ia_archiver is one) - but endlessly growing lists of supposed bad bots will do nothing for your site except slow it down. #rant
-
@mk the only halfway useful reference site I've found is http://www.botsvsbrowsers.com/ - practically everything else turned up by searching is regurgitated lists of deceased bots that everyone keeps copying from each other. Anyway, I now have a much reduced list of bots to ban...
-
