Relation of ODP to Google, AltaVista, etc

I have a URL ( Alamax Consulting ) currently listed in a regional directory under dmoz.org, listed as Alamax Consulting Computer Services. When I type in a simple phrase like "alamax" into a Google or Alta-Vista search box, I don't come up.

I thought that those search engines utilized ODP/DMOZ directories for their own listings.

What is the nature of the realtionship between ODP and the popular search engines?

Do I need to do something else to get those guys to pick up my listing?
 
U

uzs980

Member
Joined
Jul 7, 2002
Messages
5,624
Your site has been listed in October. The data for external users like Google are a bit older (from the end of September) and could not be updated yet due to technical problems. But staff is working on that. So you - and we all - just have to be patient.
 

Thank you. I presume that means that Google, AltaVista and others are all somewhat linked via the ODP.
 
H

hutcheson

Curlie Meta
Joined
Mar 23, 2002
Messages
19,136
Remember that a search engine doesn't have to ask permission either to spider dmoz.org, or to download and parse the RDF. We know what Google is doing, because they publicize it -- good publicity for them, they musta thought. We don't know what AltaVista is doing with ODP data -- if they are doing anything special with it, they must think keeping it secret is a competitive advantage. Either way, their choice.

Only DIRECTORIES (e.g. directory.google.com) have to include ODP attribution.
 
R

richard123

I think it may be a bit older than end of September (??) My site was published earlier this year on September 19, 2002.

I am hoping for an early xmas present, but I'm not really that hopeful. If they can't fix it in 2 months, what's the chance they can do it in 3? Or 4 even.... Who knows? It may require a major hardware upgrade and that could easily take 6 months or more, I'd imagine.

Still... I live in hope <img src="/images/icons/smile.gif" alt="" />
 
R

richard123

I'd rather not say. It's adult oriented. But it appeared the very first time on September 19, 2002 and has been there every day since.
 
R

richard123

Thanks for the link! It looks like the last successful one happened on the one before Sep 22, because the the one on Sep 22 wasn't complete. Mayve that's when they discovered they had a problem. So... that would have been (??) September 17, 2002. That's the most recent "content.rdf.u8.gz".

Another thing is that my site still doesn't show up in "search" after all this time. I suppose that's also on the "to do" list <img src="/images/icons/smile.gif" alt="" />
 
B

beebware

Member
Joined
Mar 25, 2002
Messages
1,070
Yep, the ODP search engine normally updates around 2 days after the RDF dump has been produced (I believe staff have got the search running off a copy of the dump to try and relieve a bit of pressure on the main server). However, this does mean that when the RDF dump is out of date, search is too.

ODP staff members are more than aware of the issue and are working on resolving it as soon as possible (in fact, as I type, there is another attempt to produce the RDF dump going ahead - but it'll be a minimum of 24 hours before we know if the problem has been sorted).
 
R

richard123

This is what I don't get, really... I think the ODP is a great resource, but it's 2 months out of date with no credible signs of being "fixed" anytime soon.

Would it not be better to tell people how long before the update will happen? I have read in various places that the update is the "highest priority" and it really just gives credence to those critisizing the ODP for being slow to get things done. I mean: If the update, as a "high priority" takes over 2 months (and possibly 3 or 4??) then what hope have we got? I mean, really!

(Of course I realise computers can be finicky things, but there are limits as to just how poor time estimates are allowed to be <img src="/images/icons/smile.gif" alt="" /> )
 
windharp

windharp

Meta/kMeta
Curlie Meta
Joined
Apr 30, 2002
Messages
9,204
Some facts about ODP you might not know:

--&gt; staff programming "team" is one person.
--&gt; RDF dump generation takes about a week if it is performing normally - sometimes even longer if it crashes.
--&gt; A task that takes that long clearly has to be optimized for speed. That means less debug information and so on.
--&gt; DMOZ link database contains almost every kind of foreign characters (you ever had to implement latin languages and japanese in one database?), lots of different encodings and almost any stupid stuff you could imagine.

Combine all of these and you will realize that tracking down bugs in RDF generation is a very time consuming task, especially since the programmer has not done all the software herself, so has to gather knowledge first.

We cant tell you how long it will take because we simply do not know when it will be fixed.

Every software related project that grows rapidly - like the ODP - reaches a limit when they discover that the current software has bugs that show only under heavy load and/or under weired circumstances.
 
R

richard123

Thanks for the very informative post. I knew some of the stuff, but not other things (especially about it taking a week to generate an RDF dump).
My impression was that "running an update" took a couple of hours at most <img src="/images/icons/blush.gif" alt="" />
All the more reason for me to not hold my breath and keep wishing for an update before xmas!
 
S

stevesliva

Member
Joined
Mar 28, 2002
Messages
80
It's "only" been running for three or four days now. We've got our fingers crossed. Here is a example of the character set nightmare referred to above, although I think it's gotten a bit worse. I've read a lot about transitioning to UTF-8, whatever that is.
 
windharp

windharp

Meta/kMeta
Curlie Meta
Joined
Apr 30, 2002
Messages
9,204
... and now to some very basic information about "Unicode UTF-8" which may sound more familiar to some :)

Unicode is a type of encoding that can handle all (uhhhh... Say at least most <img src="/images/icons/wink.gif" alt="" /> ) of those chaotic charsets used around the world - so it would make everything easier for communities like the ODP. If it wasnt a bit more complicate than those simple charsets everybody used yet. <img src="/images/icons/smile.gif" alt="" />

Some further readings:

 
S

stevesliva

Member
Joined
Mar 28, 2002
Messages
80
Well, heck, if they meant Unicode, why didn't they say so? And why throw in -8 when the big deal about Unicode is the transition to 16-bit characters from 8?
 
windharp

windharp

Meta/kMeta
Curlie Meta
Joined
Apr 30, 2002
Messages
9,204
There is a so called "UTF-8" which I think makes Unicode somehow work on 8bit. More about this can be found in the links I mentioned above <img src="/images/icons/wink.gif" alt="" />

(And you could check the internal fora for some more information about the ODP and unicode if you like)
 

There are UTF-8 for 8-Bit Character and UTF-16 for 16-Bit Character like those called "Doublebyte-Characters" from east-asia. <img src="/images/icons/smile.gif" alt="" />
 
E

eqfan7v

windharp,
Very informative posts.

But I still don´t understand: ONE week to update a database the size of Dmoz's? When I remember that google processes millions of daily searches, over a much bigger database, and returns *ranked* results in fractions of a second, I think I have reasons to still be surprised, don´t you agree? <img src="/images/icons/confused.gif" alt="" />
 
This site has been archived and is no longer accepting new content.
Top