Does DMOZ spider itself?

shawnjohn

Member
Joined
Mar 3, 2010
Messages
2
I've been wondering about this for a while.

My site is listed in this section:

http://www.dmoz.org/Regional/North_America/Canada/Ontario/Localities/O/Oshawa/Business_and_Economy/

But if you do a normal search in DMOZ for my business name or domain name, neither show up.

Why is that? Does this mean that the search function DMOZ uses is not yet finished spidering itself?
Is the category too deep?
Is the category set as noindex nofollow in some robots.txt or htaccess file?
Is the search function offered by some 3rd party, like Google? I can see G hasn't cached that page since Dec 27th.

Very strange. Can someone shed some light here?

Merci,
 

photofox

Curlie Admin
RZ Admin
Joined
Jun 9, 2010
Messages
2,092
Location
[Right here]
To see if a URL is listed in the directory you need to search for it using "example.com" rather than "http://www.example.com".

The search database is based on the RDF file we generate, so it's not updated in real-time, it can take up to a week for a site to appear in search (so long as the RDF generation process is working correctly).
 

hutcheson

Curlie Meta
Joined
Mar 23, 2002
Messages
19,136
The search function is internal--not Google or any other search engine. It's periodically rebuilt from internal databases; it doesn't use what you'd call a "spider" procedure, so doesn't have any spidering issues.

But dmoz search IS frequently out of date, as you notice. We volunteers don't control (or even know ahead of time) when the database will be re-indexed.
 

jimnoble

DMOZ Meta
Joined
Mar 26, 2002
Messages
18,915
Location
Southern England
I wouldn't worry about it. If you can see your website listed in the category, our search system will catch up eventually.
 
This site has been archived and is no longer accepting new content.
Top