Too many spiders!

[Lurker]

Note to Webmaster...

 

I noticed when I go to Who's Online there seems to be about 130 Yahoo Slurp spiders on this forum. I know this to be a common performance issue with vBulletin, not to mention the fact that it can consume all of your SQL database connections, rendering it difficult for other members to gain access to the forums at peak times. We had this problem at one time with our vBulletin site.

 

I was wondering if was ever a problem on this site, as I have noticed a recent slowdown while navigating the forums. Have you ever considered limiting the Yahoo Slurp spiders? It would give the site a performance boost.

 

FIX:

Add this line to your robots.txt file:

 

User-agent: Slurp Crawl-delay: 10

[kjones]

I took a look at the whos online page and the spiders seem to have a time delay on them already.

 

Note to Webmaster...

 

I noticed when I go to Who's Online there seems to be about 130 Yahoo Slurp spiders on this forum. <snip>

 

FIX:

Add this line to your robots.txt file:

 

User-agent: Slurp Crawl-delay: 10

[giz]

The list records activity over a fairly long time, so that isn't 130 accesses within seconds. It is not even within minutes. It is the last 10 or 15 minutes or so.

 

Spiders don't get fed a session ID, so those entries may well be one spider looking at multiple pages, and without a session ID each looks like a separate process.

[windharp]

Apart from that, I haven't noticed any performance issues lately, so I wouldn't see a terrible need to reduce traffic. At least while we don't reach the traffic limit, of opne should be in place.