CURLIE/DMOZ/ODP crawlers/bots

[Keef]

Hi,

 

Is there a list of DMOZ/ODP site crawlers or bots?

 

I ask because I am keen to avoid accidentally blocking them in my .htaccess

 

Many thanks,

Keef

 

 

PS. I hope I'm asking this in the right place - apologies if I'm not.

[pvgool]

As far as I know thare is not such a list.

There are a few cralwers who check for websites that have gone down.

Blocking them would cause the website to be seen as not available anymore. But such websites will always be checked by a human to be sure they are gpne. I a human sees the site is still available the crwaler will be overwritten.

Best not to block anything coming dmoz.org or any of its subdomains xxx.dmoz.org

[Keef]

Thanks for getting back to me about that.

 

To be on the safe side, I've had a good trawl through my .htaccess file and there's nothing there to cause any problem.

 

Oddly enough, there was a bot I saw mentioned elsewhere a little while ago which had added "dmoz" into its user agent title (if that's the correct term). I gather it had nothing to do with dmoz, but had presumably included/hijacked the name to appear more legitimate than it actually was - it didn't contain anything explicitly referring to dmoz.org though.

I'd post a reference but I can't remember where I read it.

[vlinkimmigration]

Hi I want to know how can I get my website crawled.? I have Express Entry website and it is not getting crawled.

[Elper]

Hi, our crawlers work with sites which are already listed in the directory in order to identify/flag the ones which need a re-review or removal.

To have a site listed in the directory you need to suggest it once to the closest category it should be listed in, then wait for a volunteer editor to review your suggestion.

You can read more about that process here.