Dead URLS and change in URL never incorporated

Hi ya!

This is my first post ! Look forward to be with you all!

I have observed that there are many bad URL links in DMOZ and when a company moves site or changes website address it is never incorporated.

In my last employment as webmaster I faced this problem big time.

Any idea why?

We tried to pass a law prohibiting website changes....... just kidding.

The Internet is a very dynamic place wit hundreds of millions of sites. URLs change, domains expires and some foolish webmasters change the entry points to their websites a couple of times a year because they don't know any better.

We have an automated link checker that runs either too often or not often enough, dpending upon one's perspective, and we have a thread elsewhere in this forum where we encourage the general public to tell us about those types of problems.

If you have a constructive suggestion as to something we could do better we'd be glad to hear it.

1. The link checker does not work, because many providers have non-standard 404-error pages which the checker cannot recognize. Why not let every user change his url for himself?

2. I tried the last year to get my page http://mlehmann.piranho.com/ out of the directory because an url-change does not work on category "personal pages", because nobody wants to become an editor for personal pages (of course, this is bullsh.... category). Once in the wrong category and you never come out again, although I think I have a professional page that offers some software.

3. Even worse, other directories like http://www.zomd.org/ take the the dead urls and spread them.....

4. Better no urls than wrong urls!!!!!

@1: As you already know, there is a link "update listing" at the top right of the categories where you can submit any changes you like. Of course they will have to be checked by an editor before going live. We all don't like people hijacking listings by changing the URL to something totally different, do we? Apart from that, there is a well defined way a server has to report that a link is 404, which has nothing to do with page layout whatsoever, only with a header field. If hosting companies choose to give their clients a reduced service by not reporting such links correctly, well, change the hosting company.

(I can imagine you saying: "But the person that has submitted the site should be able to..." - Definitely no. We really don't want those adult webmasters to submit random sites and change the URL once they are listed. There is no way to assure a change is valid but make an editor look at the site.)

@2: All categories can be edited from a) listed editors b) all editors listed in parent categories up to the top and c) quite a lot of editors with directorywide editing permissions.

Until recently, update requests did not stand out, so one had to look for them by hand. Since a short while now, we are able to spot them much easier. So you can expect better handling in the future, once people are used to that system. I already processed update requests in large areas that I rarely ever edit in, large areas that are now "update-request-free".

@3: Yes they do, thats neither new nor something we can change. If you spot such URLs, submit update requests so that editors know about them. If you spot a certain scheme that might help us detect them, submit an abuse report via http://www.report-abuse.dmoz.org/ including the details.

@4: That is a modified version of the main reason the ODP was created. Back then, all other directories failed to have their data current, because they simply lacked manpower.
I recently tried mining some other directories for links, and they were much worse than the ODP. Regardless of the efforts, a certain amount of rotten links can't be avoided, though. Sites go MIA all the time, and our voluntary editors time is limited.

0: Thanks for the detailed answer windharp.

@@1: >>We all don't like people hijacking listings by changing the URL...<< Nothing is 100% secure, but to build a password protected access for a user to change his url should be possible nowadays, otherwise all e-shops should be closed. I decided to use a redirector now, that is the same like changing the url by myself....

>>If hosting companies choose to give their clients a reduced service by not reporting such links correctly, well, change the hosting company.<<
Why should the hosting companies report that correctly if it's a great way to sell commercials? Maybe wrong urls are the only way to get clicks for the company.

Well- fact is that the link checker did not work on my page and others seem to have the same problem.


@@2: A category should be edited by someone with knowledge for the theme, otherwise you can take a robot for it. Personal categories should be thrown out. The advantage of an editor is, that he can rank a page, better than a machine. That is of no use, if the editor has no idea of the theme or has lot of work with updating urls.

>> Why should the hosting companies report that correctly if it's a great way to sell commercials? Maybe wrong urls are the only way to get clicks for the company.

Sure, it's up to them. It is still a reduced service for the customers, and it's up to the customers if they want to act on it or not.

A category should be edited by someone with knowledge for the theme, otherwise you can take a robot for it. Personal categories should be thrown out. The advantage of an editor is, that he can rank a page, better than a machine. That is of no use, if the editor has no idea of the theme or has lot of work with updating urls.

Click to expand...

Several misconceptions of what we do, I think (or else I'm just not understanding you - if so, I apologise.)

We have no interest in throwing out "personal categories" - why would we do that? They contain a lot of unique content, and that's what we are interested in.
It's always good if an editor has knowledge of the topic s/he edits in, if nothing else because it makes it more fun to edit. But it is not as important as knowledge about editing.
We don't rank pages. Sites that are listed, are listed alphabetically and the site descriptions are supposed to be objective (not all are, but that's a different matter).

saxophon, you have clearly mistaken us for an advertising banner farm.

The fact is, listings in the open directory do not belong to the webmaster, or to the submitter (as if we could tell whether they were the same person!), or even to the editor that added them.

Once listed, they belong to the public and are held in trust by AOL/Netscape.

So, you see, your concept simply has no meaning in our context.

I think that, despite your excellent intentions, there are a few salient points you don't understand about DMOZ (this does not make you a bad person )

Access to the DMOZ database is pretty restricted. Even within the editorial commjkunity, editors can only edit in those parts of the directory where they ahve demonstrated (past) proficiency and general knowledge. If I wanted to start editing, for example, in the Philippines, I would have to apply for that category, submit sample sites, and have some of the seniormost editors review my past editing history to see if I comprehend and follow the editing guidelines, if I comply with policy, and if I am performing in an ethical manner. I'm just glad that having a full head of hair is not a requirment.

If we are this tough (which is a very, very good thing) on our own editors, you can imagine that we are not going to allow non-editors access to the same data. The issue really isn't technology, it is philosophy.

Heck, we can't even get 75% of the posters in this forum to follow the rules -- can you imagine the chaos they could create within the directory?

While the link checker does an excellent job, we also rely on a couple of other things to improve the quality of the directory:

-- The Update URL process is not restricted to the site owner, anyone can suggest an update, and some recent programming chagnes have made these updates much more visible within the editing community, which means they are generally less likely to languish in some category unseen by an editor.
-- This fourm has a section where the public is encourage to identify and submit bad links. This section is manned by some of our most senior and trusted editors who aggressvively attack the bad links.
-- Editors routinely check links within their categories, looking for bad or expired URL. Editor tools are available that facilitate this process.

Hope this helps.

but to build a password protected access for a user to change his url should be possible nowadays, otherwise all e-shops should be closed.

Click to expand...

I guess you didn't understand my point. Most liklely, because I am concerned to mich with this topic, so my statements aren't understandable by others. I'll try again, imagine the following process (I can assure you that I have seen lots of similiar attempts with the Update URL function...):

a) Mr. X submits 2 really good sites. Not sites of his own, but he acts as if they are. We can't possibly check if those are his sites, so - following your advice - we would assume they are.
b) Some time later, an editor checks those sites, finds they are perfectly suited for the the categories they are suggested in, and lists them.
c) Mr. X immediately grabs the login data he provided in step A, starts giigling and changes the sites we just listed to mrxhispornsite.com.

Conclusion: We simply can't allow non-editors to do things like this.

A Shop needs a different approach. Which benefit would someone have to give false information in a shop? None, most likely. Which benefit would someone have to give false information at the ODP? Obviously a lot, since people try it all the time.

Great response!

Nea: Every directory does a kind of ranking. When editors put a site in a specific category, update an url or not, it's ranking. Even if you sort it alphabetically it's ranking... To speak philosophic (like spectregunner suggests): "If every information is available everywhere to every time, it's the same like having no information." So the sense of a directory is to give the chaos a direction, and that only works if you filter out or prefer things - ranking.

If someone wants to cheat he can do this right now windharp:
a) Make (or copy) a site with content "Jesus loves you".
b) Wait until an editor puts it in a category.
c) Change the content to "Jesus loves sex".
So I guess, if users can edit their url and site-description for themselves, you will not have more junk than you have right now. But most of the site-descriptions will be well defined, urls are up to date and editors get more time...

...more time to e.g. get unimportant things out, that you can find on e.g. most "personal pages". So editors should be much more restrictive and throw out pages like "my personal intrests, and my pets". Why do you want to be open for everybody? I told above, you do ranking! Why not doing it right, when you have expert-editors? Let the experts sort the pages. Don't be democratic, instead try to get a knowledge-base.

...looking foreward to some interesting replies!

So I guess, if users can edit their url and site-description for themselves, you will not have more junk than you have right now. But most of the site-descriptions will be well defined, urls are up to date and editors get more time...

Click to expand...

Given the horrific titles and descriptions that most site owners use when they suggest their sites, there is no way we'd be able to give site owners free reign over their listings and not have complete and utter rubbish result.

saxophon, it looks like you've invented something ... certainly not the ODP, and certainly not at all like the ODP, and not at all compatible with the ODP's mission, methods, or community. But something.

Let's see, we could call it -- oh, I know! We could call it a "free-for-all links farm."

If this were 1995, and your main purpose in life were to destroy the search engines of that era, you would be deep in the leguminous ground cover blossoms now.

As it is, if that's what you really want, there are tens of thousands of them out there on the net. We will not be offended if you take advantage of their services. And they do what they do better than we ever could, even if we wanted to -- which we don't.