http://www.davidrisley.com/2007/01/03/robotstxt-the-often-forgotten-file/ Six Figure ProBlogger, Blog Marketing, Make Money Online, Blogging Mon, 15 Mar 2010 05:04:07 -0400 http://wordpress.org/?v=2.8.5 hourly 1 http://www.davidrisley.com/2007/01/03/robotstxt-the-often-forgotten-file/comment-page-1/#comment-5 Yahoo Slurp Spider Drives Forum Server Load Through the Roof! : WebbyOnline Fri, 19 Jan 2007 15:05:53 +0000 http://www.webbyonline.com/2007/01/03/robotstxt-the-often-forgotten-file/#comment-5 [...] Well, the mystery of it is that my server load was still relatively high even after this change in my server setup. Yes, there was an improvement, but the load was still not in the range I would have expected. Well, yesterday morning, I went over to the PCMech Forums as I do every morning to check things out. I noticed that there were over 900 guests on my forum! A couple weeks ago, I made a post here about the robots.txt file and how you can control spiders across your site. I put two and two together and decided to check out the listing of all people on the forums at that time. Sure enough, the Yahoo Slurp Spider was VERY active on my forums, so much so that there were probably at least 700-800 instances of Yahoo on my forums at the same time! [...] [...] Well, the mystery of it is that my server load was still relatively high even after this change in my server setup. Yes, there was an improvement, but the load was still not in the range I would have expected. Well, yesterday morning, I went over to the PCMech Forums as I do every morning to check things out. I noticed that there were over 900 guests on my forum! A couple weeks ago, I made a post here about the robots.txt file and how you can control spiders across your site. I put two and two together and decided to check out the listing of all people on the forums at that time. Sure enough, the Yahoo Slurp Spider was VERY active on my forums, so much so that there were probably at least 700-800 instances of Yahoo on my forums at the same time! [...]

]]>