Search index refresh rate

Hello everybody,

My question doesn't fall perfectly under the "bugs and features" category, but it's the closest I could find.

I just submitted my site http://www.velosophy.com to the Regional: North America: United States: New York: Localities: C: Cooperstown: Business and Economy category yesterday, and the great news is that it was accepted today, with a small, but good edit of the description.

I've noticed however that it doesn't show up when I use the search feature. I'm guessing that it is a matter of the frequency of the index refreshing, and I'm curious how often that occurs if that is really the situation.

Thanks in advance, and cheers,

Cameron Booth

The search index is not refreshed on a continuous basis, but rather more sporadically. Although I personally don't know, other editors have said it to be around once a week.

It should be refreshed about twice a week, according to staff editor "dole" who is in charge of the search system. The last update was late on Sunday evening Pacific Time (about 8am GMT), so expect an update any time soon.

Bear in mind that Robozilla has been weaving his merry ways around, so it may take a little longer.

This sounds very similar to the current situation that our site is experiencing. We had our site added to the
Shopping: Home and Garden: Soft Furnishings: Cushions
and Decorative Pillows category on April 12th. Our thanks to the ODP editor here for that, and the listing is just fine. However, we notice that on using the search facility of the OD our site still cannot be found. We have noticed that once a week since then (usually on a Sunday/Monday) for a period of around 24 hours any search using the ODP search facility results in "Temporary unavailable" message being issued. Eventually when the Search index seems to have finally been updated - assumption made here is that when the "Temporary unavailable" message is no longer displayed and a search returns results, then the index update is complete. However, currently the last update date always reverts back to April 8th. It's as if once a week the "index update" program runs (which appears to take aroud 24 hours) during which time the search facility is unavailable (this all makes sense); but for the past 2 weekly attempts something has gone wrong and the update has been aborted and rolled back to the last succesfful update which was on April 8th. So my diagnosis is that any site added after April 8th will still not be searchable until a successful index update has been completed. Any of this ring any bells with anyone out there. Also if my rationale has any logic to it, why is the index update failing, and when will it be fixed so my site and others added after April 8th can be found during a search. Also, I guess, until the index update on the ODP is corrected any sites feeding from the ODP index will be stuck in an April 8th time warp.
Anyone any thoughts, or am I way off track. Help please.

Basically, the first thing to remember is that most people don't use the on-site ODP search. (Basically, editors, submitters, and users of sites which take live feeds of the ODP data).

Secondly, what's available in the search is not indicative of what is in the ODP data dump (RDF). The way the category itself looks is how the sites are listed.

The ODP search index is generally updated about twice a week; during this time the search function is normally available (just with the current, not new, index). Once the indexing is complete (which takes more than 24 hours, I've heard), the index is replaced with the new one.

In the past couple weeks, search has been undergoing many improvements "under the hood" and this has resulted in the search being 'unavailable' and also has prevented the search index from updating properly. Staff is working on this issue and it should be fixed soon.

So please be patient and our onsite search will be back and working better than ever soon. As long as your site is visible in a category, it will be updated when the time comes.

-sam
[Added: see also this thread]

samiam,

I have the same concern over the odp search. I have been wondering why Google did "not" pick up my business category listing on it's last update. If I go to http://dmoz.org/Business/Industries/Construction_and_Maintenance/Design/Kitchen_and_Bath/ and search by my Company name "only" in that category, I am not found. Why is this? Same answer you are telling other poster?

Thanks For any info,
Dave S.

This is actually two separate questions: one relates to the data dump and Google's use of it, the other to internal ODP search. The data dump (RDF) is completely independent from the ODP search index [however, note that they are currently both being affected by bugs stemming from work being done by staff]

I don't know when Google updates their data and any details relating to that. A side effect of the bug with ODP search has been that the most recent data dump (RDF) that Google could have picked up is from April 4, before your site was listed. That would explain why it is unavailable in the Google directory. (your site should be included in the next RDF dump, which should happen soon; I don't know when Google will pick it up).

The question on dmoz search is separate - it appears that your site is not in the current search index for that listing in that category. When the next index update rolls around, it should be added - patience is key.

Note that what I say is just what I've picked up over time and is accurate to the best of my belief, but I'm not staff, or even a Meta.

Just FYI, the RDF dump that Google used for its last update had to be from before March 23. I've been watching a couple of sites I added on and after that date to see what happened in Google with them, and Google hasn't picked them up yet.

samiam,

Thanks for the reply /images/icons/smile.gif
As always the information I receive from this forum helps me to understand what's "really" going on. I now understand why google picked up our "regional" listing around the tenth of the month but never picked up our "business" listing. It probably didn't matter anyways since google totally messed up when indexing our site this last time around and reverted to a very old cache of our site. Hopefully the month of May will be better for everyone in the same boat as we are in /images/icons/smile.gif

Thanks Again Sam,
Dave S.