Suggestion to help remove stale links from DMOZ

Hi:

I run a couple of sites, one of which has a targeted directory of subsets of DMOZ data. In the course of running my downsized version of DMOZ, I have noticed a LOT of stale links in the DMOZ data.

I am not too familiar with all the inner workings of DMOZ and all, so I posted my observations to a forum (www.webmasterworld.com) that I knew was frequented by DMOZ editors. In the course of our discussions, I learned the following.

1) DMOZ does have a spider that does regularly check for old links.
2) A LOT of links can "pass" the spider test, and be bad- custom 403 and 404 pages, a page that says, "We have moved..." etc.
3) DMOZ editors are a bit overworked, and do not always have time (or the inclination) to check existing links
4) DMOZ DOES put a priority on getting this sort of thing removed from their directory.

Well, in addition to pointing this out (which I am sure you are aware of anyway), I have a solution that should work well in the exisiting structure of DMOZ, AND NOT CAUSE A WHOLE LOT OF ADDITIONAL WORK BY THE EDITORS! Sure, it will cause SOME additional work, but it is not too time intensive. In practice, I have found I can review and deal with one bad URL in 10 seconds or so. But the MAJOR thing is that it at least brings these bad URL's to someones attention!

The solution is quite simple! Add a link to EVERY listing (dynamic, of course) that when clicked on by a user will take them to a page so they can report a bad link to the editor of that category. It would be run by a simple cgi script that is outside of existing DMOZ code, so should be able to be added quickly and easily, and yet work for the whole DMOZ system. (See a sample "Report a Bad Link" page from my site here: http://www.toon.com/links/cgi-bin/badlink.cgi?Title=The%20Cartoon%20Factory%20Animation%20Art%20Gallery&ID=7888&URL=http://www.cartoon-factory.com/&g=Report). Once reported by a user, my script would e-mail the editor of the category, tell him/her what site MIGHT be bad, the URL of that site (which in most e-mail systems is a clickable link, so click and decide right there!) and then also provide the editor RIGHT IN THE E-MAIL a link to either edit the URL (if the page has moved) or delete the entry entirely.

Yes, a very simple solution... and works very well for me. DMOZ might want to add some safeguards or something, again I do not know how you work. But this would really make cleaning up DMOZ a LOT easier, I think!

Yes, I am sure you will get some abuse, too. But as I said, it only takes 10 seconds to check each report, so the time impact is pretty minimal. But getting those stale links OUT of DMOZ has a tremendous benefit.

Anyway, I wanted to bring this to your attention. Hopefully, there is someone here who knows how to take this the next step (if you think it is a good idea!)

I have a basic script all written, and would be happy to GIVE it to DMOZ, gratis, no strings attached. (Kind of my way of paying DMOZ back)

If you think it is a good idea, and would like to see my script, and know who to go to to get this up and running, please feel free to contact me at admin-at-bcdb-dot-com

Thanks!

Dave

I must say I do like this idea of some sort of clickable "Report a Bad Link" on the user side. They already can effectively do this by clicking on the editor name and giving feedback. However, with a lot of child cats having no named editor, many cats don't even have that link. Users may not know to go up the tree to find an editor. With a visible "Report a Bad Link" to click on, then users will know exactly how to report such.

Re: Suggestion to help remove stale links from DMO

I'll be honest with you and say that, while it no doubt works really well where only one person (you) is responsible for the links, I don't think it would work very well for the ODP as described. The biggest impediment is that many categories have no listed editor and even those that do may have relatively inactive editors -- sending an email to a specific person, therefore, may be impossible or at the very least possibly very ineffective.

Each category page right now has an "Update URL" link that can easily be used by anyone who comes across a bad URL. The benefit to using "Update URL" rather than emailing someone is that the update goes into the queue where any editor traipsing by can see it and act on it.

Re: Suggestion to help remove stale links from DMO

Based on what motsa has written, and alternate possibility would be to have the "Report a bad link" lead to a page similar to the update URL one. The problem with that update URL link is the page says "If you have submitted an URL to the Open Directory Project, and it is listed in <cat name> you can use this form to make changes to your site's listing." This implies that it is intended for site owners. The public may not realize that it can also be used to report bad URLs.

Re: Suggestion to help remove stale links from DMO

rfgdxm:

Thanks for that! I was goiung to write the exact same thing. For those of us on the "outside", that does NOT imply that it is a place for me, as a user of DMOZ, to tell you something is bad.

In regard to the other comment... yes, it does work well for me, as one person editing one place. I would fully expect it would have to be modified to fit the DMOZ environment. Not being part of that environment, I really do not know how to tailor it more to be DMOZ-SYSTEM friendly- that would be up to you.

I would also suggest that in a system like yours, it might be better to have a site put on "hold" or something... once an editor sees a site os off-line or something, they could "suspend" a site for 10 days, or 30 days, or whatever... put it in a "re-review" queue. If the site remains inactive after whetever DMOZ considers a reasonable time, THEN it gets whacked! (Or possibly sent to a higher up for a final review).

The bottom line is that this can at least be a starting point for you to consider... toss the idea around, and see how it would work best for you... and then impliment it or not as appropriate.

dave

Re: Suggestion to help remove stale links from DMO

I think I have to come out and say, as a programmer and a long time editor, that I think this idea has a lot of merit and would work if done right. A simple link on the public side after each description that said "Report bad link" or something like that. Then clicking on it would mark it like robozilla does, only say orange instead of red.

The thing we would have to guard against is a malicious visitor who might begin clicking on everything (I even have a couple of ideas for that but won't go into it here).

I don't think I would suggest anything like this, though, until we have a couple more promised improvements that would relate to this; which we likely won't have until the new hardware arrives in about 5 months. Once the hardware does arrive, though, rest assured that I will suggest this internally. <img src="/images/icons/smile.gif" alt="" />

Re: Suggestion to help remove stale links from DMO

And don't forget you'd have to watch what sort of link it is - if it's a straight forward a href="reportbad.cgi?url=http://..." style one the first 'bot (such as Google) will come along and mark every site as bad!

Re: Suggestion to help remove stale links from DMO

Yes, good point. I was already thinking about a confirmation screen using POST, so that probably wouldn't be a problem. The confirmation screen would slow down a malicious visitor also.

Re: Suggestion to help remove stale links from DMO

theseeker:

&gt;And don't forget you'd have to watch what sort of link it is - if it's a straight forward a href="reportbad.cgi?url=http://..." style one the first 'bot (such as Google) will come along and mark every site as bad!

Good point- and one I had not thought of. I will have to look at this again on my site for just this reason! I could easily have this key (for me!) on the link ID number, and then "behind the scenes" grab the URL from my Database... Good point! Thanks!

Beebware:

Glad you see some merit in this... and I am glad you will bring it up at some point in the future! That is all I wanted to do! Thanks!

dave

Re: Suggestion to help remove stale links from DMO

Just wanted to clarify that I said "I don't think it would work very well for the ODP as described." (i.e. having it send an email to an editor), not that I thought the idea in general had no merit (just realized my original post wasn't perhaps as clear as it could have been).

Re: Suggestion to help remove stale links from DMO

One idea would be to have a link to a form similar to the Update URL one. This form when submitted would add the site in unreviewed like an Update URL does now, just with a tag like "Bad URL" or such.

Re: Suggestion to help remove stale links from DMO

I don't want to be seen to throw water on what is IMO a _most_ desirable feature, but:

-- You don't want to e-mail the editor--who may not be active anyway; you want to go into the queue with Robozilla's bad-URL alerts (which non-editors don't see, but which get higher priority treatment than either unreviewed sites or e-mails to editors. (That's a technical issue, probably a very minor one that Netscape Staff could easily fix.)

-- Where are you envisioning this link as integrated into dmoz.org? Netscape sees the ODP as producing the RDF, which people like Google can use so long as they put the "Add URL" and "Become an editor" links. I don't think even "Update URL" is required. So, assuming that most users who spot bad links will spot them at AOL or Google, who don't have to add anything to their screens, how could the ODP roll out this feature so that the people who need to see it (and I agree they are legion) will see it?

Perhaps a "test implementation" could do something like insert posts in a thread in these forums, which moderators could delete as they were checked?

Re: Suggestion to help remove stale links from DMO

&gt;So, assuming that most users who spot bad links will spot them at AOL or Google, who don't have to add anything to their screens, how could the ODP roll out this feature so that the people who need to see it (and I agree they are legion) will see it?

Even if just the dmoz.org users were able to do this, this would mean that at least some people would do it. In particular, it wouldn't surprise me that webmasters with sites listed in the cat wouldn't look for dead links and report them so there site would be easier to find. Definitely the best way to implement this is something like how Robozilla flags reds.

Re: Suggestion to help remove stale links from DMO

Robo-flagged Reds and Human-flagged Oranges then ? I like the idea.

Re: Suggestion to help remove stale links from DMO

Good! I am glad, that in some small way, I could help! <img src="/images/icons/smile.gif" alt="" />

Good luck to all at DMOZ!

dave