

News Archive
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
Yahoo Slurps Somewhere Else
June 7, 2007, 9:44 amThe migration of the Slurp is complete, says Yahoo. Over the past few weeks, the search engine has been transitioning its crawler, dubbed (disgustingly) "Slurp," to a new address at crawl.yahoo.net. Adjust your server logs as necessary and join the curmudgeons who are unimpressed.
Too little (or too much, by some complaints), too late, it would seem.
The Yahoo Search Blog reads:
...all machines crawling as Slurp are now in crawl.yahoo.net. You can see this change in your web server logs, where the page accesses from inktomisearch.com are being fully replaced by crawl.yahoo.net contacts. Note that this does not cover other Yahoo! crawlers, such Yahoo! China, and other verticals, like Yahoo! Shopping, Yahoo! Travel, etc., which have their own user-agent.
Don't fret though; there is no need to change your robots.txt file because the crawler user-agent is still Yahoo! Slurp. If you use IP based filtering, there is no need to change that either, since the IP addresses from which we crawl remain the same. However, please ensure that your network or firewall setup does not keep crawl.yahoo.net out as we won't be able to include your content in our results.
Be sure to click that link to get more enumerated information.
Over at WebmasterWorld, the crowd is a bit mixed about it (but only a bit), the loudest complaint, from "IncrediBill," who notes not only is it the move a year-and-a-half too late, but has gone overboard.
Why do we need to allow an army of Yahoo spiders to redundantly abuse our servers?
Is it a conceptual problem that Yahoo can't share pages already downloaded?
When I posed that question to one of their engineers I was given a lame excuse that the various crawlers had different needs….
Funny, Google managed to make some of their crawlers share CACHE, so we know it can be done.
Negativity often rings louder and truer than other things, but there is at least one voice in that forum who thinks Yahoo's update is "a small evolutionary improvement above" Google.
Even if in our hearts, we know that's not true. :)




