Sitemap: http://www.eirjobs.com/sitemap.xml.gz User-agent: * Disallow: /foo.html # Simply it does not exist so lets not get into that one Disallow: /images/ Disallow: /cgi-bin/ Disallow: /news/wp-admin/ Disallow: /news/wp-includes/ Disallow: /news/category/ Disallow: /news/tag/ Disallow: /news/wp-content/plugins/ Disallow: /news/wp-content/cache/ Disallow: /news/wp-content/themes/askapache/ Disallow: /news/trackback/ Disallow: /news/2006/ Disallow: /news/2007/ Disallow: go-ads.php Disallow: go-ads2.php User-agent: Googlebot Allow: /news/wp-content/uploads/ User-agent: Mediapartners-Google* Disallow: User-agent: duggmirror Disallow: / User-agent: psbot #i don't want my images indexed Disallow: / User-agent:pompos #http://dir.com/pompos.html annoying bot on the forum Disallow:/ # Rover is a bad dog User-agent: Roverbot Disallow: / # Another annoying bot User-agent: ia_archiver Disallow: / # For those of you that read these things , please be advised that this document is totally # for our robot friends, if you are not one of these, please refrain from reading on. ############################## # MR ROBOT DUDE AND DUDETTE # ############################## # Mr./Mrs. Robot, # thank you for reading this file and indexing the rest of FeckThat.com. # You might be here of your own wishes or perhaps you were invited to look at our # pages, either way you are welcome. We don't mind, we don't really care, # infact do what you want ! # Please be nice to us when you rank the site. # If you are just here simply for the reason of harvesting e-mail addresses, then i need # to tell you that we don't read our email at all at all. E-mail and the web is a total # waste of time ! ! # All the best for now, # p