Relation of ODP to Google, AltaVista, etc

But I still don´t understand

Click to expand...

Well, Google was funded by venture capital (I know of one round of $25 Million, I don't know if there were other rounds or not), plus they make oodles of money from search agreements they've signed, AdWords, etc.

All that money allows them to buy a fair bit of processing horsepower. I know of 10 or 12 servers accessible to the public, and I don't doubt they have many times that for internal data massaging.

ODP has some very nice hardware, but nothing like that.

Indeed, the entire ODP runs on a mere 4 machines, if my memory serves my correctly. And at least 1 one of those was only added in the past 2 weeks or so.

And Google doesn't produce their index in seconds. They produce it in at least days, perhaps weeks. We don't know and they aren't telling, but their spider crawls over my own web site about once a month. It's the searches within that index that the user sees, and those are indeed blazingly fast.

But I still don´t understand: ONE week to update a database the size of Dmoz's? When I remember that google processes millions of daily searches, over a much bigger database, and returns *ranked* results in fractions of a second, [...]

Click to expand...

Google takes a long time to generate a complete update, too. They update their index towards the end of the month, based on data gathered during the "deep crawl" at the beginning of the month. On my server, the "deep crawler" showed up during 3-12 January. The actual update began around 26 January, so it took about two weeks to generate the new index.

This is a good read: http://www.internetwk.com/lead/lead060100.htm

>> This is a good read <<

Cool! Thanks for the link. And that was 2 1/2 years ago... when they just added a 3rd data center.

According to this FAQ...

Google Update FAQ

They now have several data centers with at least 10,000 individual computers (I believe there are 7 data centers). That's an awful lot of crunching power <img src="/images/icons/smile.gif" alt="" />