I hate to depress you guys... but

>Automate the management of the review process so that it becomes more equitable, transparent and efficient

There isn't any "management of the review process". None. There are no inequities in it. There is nothing opaque, or even translucent, in it. And there are no inefficiencies in it. None. What you see (which is nothing) is all that there is.

Any volunteer can review any site, any time, for any reason. There's no need for any automation: any management, any kind of management at all, would make the process less efficient (and make all sites wait longer for review.)

Every site listed is public -- anyone on earth may look and see whether those sites contain significant unique informational content, and therefore determine whether they are listable by ODP standards. And anyone on earth may report listings that shouldn't be there. And any editor can check on those listings; any editor with privileges in the category can change them. You can't get more transparent than that!

Note that there's no management on who will check on alleged bad listings, either. And you know what? There doesn't have to be! If the editors didn't care about bad listings, they are volunteers and no amount of management could make them do anything. And if the editors DO care about bad listings, any conceivable amount of management would just get in the way of the editors fixing whatever matters!

I think that also illustrates the radical efficiency of the editing process.

In addition, editor actions are logged, so other editors can see what has been done, and decide whether (or not) reduplicative work (for instance, checking for inequities!) seems warranted. And internal editor discussions allow editors to exchange techniques that enable them to find and review sites more efficiently, as well as refine the site listing criteria so that everyone is judging sites similarly.

All of this information is available to any editor -- and it's easy to become an editor. All you have to do is demonstrate (by your actions, on an application) that you understand and sympathize with the ODP mission.

For anyone who has demonstrated that understanding and sympathy, the details of all actions taken on sites (including rejections, in the special case of suggested sites) are an open book. That's about as transparent as one can get.

Now, people who try to use that transparency to promote their own selfish interests, lose access to that information. And that's equitable -- because that is abuse, pure and simple.

Now, why did YOU want information like that?

Hi Motsa,

Let me repeat again for the benefit of everyone... I think the ODP is not only a great concept and resource, but also that all the volunteer editors do a great job and ALL communication I have ever had with them on an individual level has been a good experience.

Yes, I do realize that ODP is human edited directory and that SHOULD NOT CHANGE, however I believe that proper tools and technologies can streamline the process, reducing the amount of drudge work required by editors. ie checking for affiliate sites,mirrors and other technical stuff that could be done automatically ( maybe you already have some of this?). The idea being that editors really only need to concentrate on their editing and that whatever else they need to do should be automated if it can be.

With regard to submissions, yes it is automated to a certain extent but I see no protection against machine input (which is likely to generate a huge amount of spam entries). Theres lots else I could suggest... but I don't want to get into a specific debate as I don't actually know what your behind the scenes systems are like. But what I do know is; as a submitter, you submit your site and it then disappears into a black hole. So what? I hear you say! Well as a submitter, like thousands of others, I care.

With regard to transparency, I'm sure you are right when you say things are very transparent for editors, if they go looking... But what systems are in place that self monitor or actually measure performance with a view to flagging if something is going wrong? You guys are busy, its simply not realistic to expect you to go around checking that things are working OK. The only other people that care that things are working properly are the submitters... and the system is not transparent for them.

With regards to me reapplying to be an editor, you make a fair point and perhaps I should have tried harder... but I go back to my very first post on this thread... I did everything "by the book", and that includes the category. In the editor application guidance on the dmoz site, tips and advice section, it states that you should select a category with less than 100 entries... which is exactly what I did. But got rejected for selecting a category that was too large. It just didnt make sense to me.

With regard to submissions, yes it is automated to a certain extent but I see no protection against machine input (which is likely to generate a huge amount of spam entries)

Click to expand...

Because you can't it, doesn't mean it doesn't exist.

You guys are busy, its simply not realistic to expect you to go around checking that things are working OK.

Click to expand...

And yet we manage to do just that in a reasonably effective, if not perfect (by any stretch of the imagination), way. Sure, things get missed but less than I think you might suspect.

You are right at so many levels! I love the simplicity of it all and I am strong believer in KISS (keep it simple silly).

What I suggested however was not straight-jacketing you into some management structure but rather; automating things where it would assist editors in their work or improve the ODP experience for everyone and that includes submitters because believe it or not... we are human too!

The reason as you well know that all of this matters at all is that ODP listing has become a very important thing for websites... and when things become important, they matter. Now I know that that is NOT the ODP agenda ( which is to build a great human-edited web calogue ), but I refuse to believe that you do not care about the frustration people experience when they believe their submissions have been abandoned. The transparancy that I would like to see would be sufficient for people to feel reassured that their submissions are not lost and will be dealt with in good time, in the knowledge that if something does go wrong... there are systems in place that will flag it up and correct it. If that were true, people could submit and forget.

There are no inequities in it

Click to expand...

I disagree... I do not think it is equitable that someone can submit a site tommorrow and have it listed within a week ( but I hope they do) and that others who submitted years ago are still waiting

I think that also illustrates the radical efficiency of the editing proces

Click to expand...

Chaos Theory is all well and good, but sometimes a little order can help things along.

Finally, as for MY AGENDA, I have none... I took all of your advice years ago and no longer worry about the submission I made, I do believe that it will eventually get reviewed and I don't lose any sleep over it. ( That does not mean that I don't care, because I do.)

As a fine word-smith, you have drawn me into this debate and I have given my honest opinion ( however misguided you may think it is ) and views where I think things can be improved. If you think everything is fine exactly as it is... well that's great

Life is change

You should be aware, that in a large group of very bright editors, there are many that have some very good computers skills and have developed many internal software tools that aid and semi-automate some of the things you talk about. You would be surprised the tricks that some web-masters get up to in order to try and sneak in duplicate content. We don't discuss these tools, since we would just as soon the bad webmasters not get any hints in how to get around this.

In the end however, it's the human touch that give us the edge. When I see a web site and know that I've seen the images before in some similar form, it's hard to imagine that any automated system can do the same.

Many editors also rely on intuition, I've often gone to review a site, and had a gut feeling there was something wrong, tagged the site and left it waiting for a another review. Monhts later, I will discover that I was right, and be able to document the reasons for deletion, and dump the site.

georgez said:

But what I do know is; as a submitter, you submit your site and it then disappears into a black hole. So what? I hear you say! Well as a submitter, like thousands of others, I care.

Click to expand...

Probably this is the cause of your confusion.
You can't submit a site to DMOZ at all.
You can only suggest a site. Any site, not only yours.
Maybe the difference between submit and suggest isn't clear but there is a big difference.
By suggesting a site you tell the editors "Hi, I think I have found a site that is worth listing in your directory." If an editor agrees (s)he will list it.
By submitting a site you tell the editor. "This is my site, please list it in the directory".
Do you see the subtle difference.

For editors it would be best if you could only suggest sites that you aren't affiliated with. Ofcourse this is technicaly not possible.

I do not think it is equitable that someone can submit a site tommorrow and have it listed within a week ( but I hope they do) and that others who submitted years ago are still waiting

Click to expand...

What you are proposing is some sort of seniority system where edtiros are required (somehow) to review the oldest suggestions regardless of merit, regardless of the editor's preference or interest.

The problem with what you propose presupposes that sites in the suggestion pool have some sort of intrinsic value to the average editor -- that by the virtue of making a suggestion, the website get put in some sort of hallowed grownd from which the editors must work.

I spent hours last night, in one tiny corner of the directory, doing nothing but moving grossly missuggested sites to much more relevant categories (knowing that a certain percentage of these will get moved again by editors with much more specific knowledge of what goes where inteh receivig categories). I may have listed a site or two in the process, but for the most part it was open the suggested site, try to figure out who they are and what they do (certainly the title and descriptions were little more than a mass of keywords and completely failed to describe the site content) and then try to figure out where they belonged. It was an essential task, one that could never be automated, I hated every minute of doing it, but it needed to be done. Later this week, I'll go play with airplane and military sites, scouring geocities and tripd to see if I can find some jewels. That will be fun.

George, I believe you are well-meaning and sincere. It is just very difficult for someone on the outside to critique our internal process when they are only able to see part of our total output.

It is not unlike walking into a Dell Computer store, looking at the laptops, and telling them they would have a better product if they changed the brand of surface mount capacitors they are using on the underside of the motherboard that you have never seen. you are probably growing frustrated because you are making these well-meaning suggestions, and they are getting shot down in nanoseconds -- in no uncertain terms. That does not reflect on our opinion of you, or even on the amountof thought you have put into your ideas. It is just that we have seen variations on these same themes many, many times.

Does that mean we don't want to hear from people like yourself? No, it doesn't. There have been anumber of occasions when people in this forum have come up with some truly novel ideas that have been carried into ODP internal discussions, to further evaluate their merit and if they could be implemented.

I suspect, but do not know, that the manner in which we internally identify update requests is a result of discussions we had here. So we really are looking for the next "really good idea."

Your suggestions on automation and tools is a good example. Among our active core of volunteer editors we hav some excellent tool makers who have voluntarily put together some exceptionally powerful tools that facilitate the editing process -- while still leaving the decisions in the hands of editors. Every tool they develop seems better than the one before. So we really do not face a situation where the lack of tools hinders our ability to edit.

The biggest challenge I think we face: willfully bad submission by malicious people who want to break our system for their own benefit.

human touch that give us the edge

Click to expand...

I totally agree!

As I stated in an earlier post, I don't know what systems you have behind the scenes, but when I was making the decision as to whether to apply to be an editor, I obviously reviewed on DMOZ what that entailed. It seemed to me that many expected tasks could be automated without impacting the "human editing task".

It seems that many with IT skills have done just that! But wasn't I reading in some earlier post that one of the primary pre-requisites to being an editor was to have librarian skills?

Having librarian skills is not a requirement. Thinking like a librarian is a huge benefit.

I disagree... I do not think it is equitable that someone can submit a site tommorrow and have it listed within a week ( but I hope they do) and that others who submitted years ago are still waiting

Click to expand...

I think it is completely equitable that when you suggest (or even *if* you suggest) a site should have no bearing on when your site might be reviewed. Is someone who suggested their site to us more worthy of a listing than someone who never suggested their site at all simply because the former found us and the latter didn't? No. A site is worthy of listing or not based on the site itself.

Hi spectrgunner,

I'm sorry if you thought I was making any specific proposals, I had tried to avoid doing that.

It is clear from several responses that there are more tools to help you guys than I; as an "outsider" know about. And I'm glad to hear it.

I think I was suggesting "problem areas" without being prescriptive about the solutions.

Your description of what you as an editor had to do and how much you hated it, does (to me anyway) start ringing warning bells. As a long term system analyst and architect, it is exactly these kind of process failures that we look for when trying to iron out process flow problems within a system. Put simply, if it ain't working, somethings wrong!

I don't understand your argument. How can you determine whether one site is more worthy than another unless you review both? In which case my argument still stands, I think...

Maybe this will help: Hypothetically - I'm working in a category that deals with information related to the town of Lacey, Washington. Someone suggested a site two years ago for the town of Bellingham, Washington and someone suggested a site two days ago for the town of Lacey. Assuming both have unique content, which one is more worthy of being listed in the category where I am working?

Your description of what you as an editor had to do and how much you hated it, does (to me anyway) start ringing warning bells. As a long term system analyst and architect, it is exactly these kind of process failures that we look for when trying to iron out process flow problems within a system. Put simply, if it ain't working, somethings wrong!

Click to expand...

Sure, but the only way to remove that particular problem would be to remove the possibility for the public to suggest sites for review. I kid you not, that is the only way we could make sure that sites are not mis-submitted.

And while doing that would be a huge relief in many ways, and would make our lives easier... it would not benefit the directory. After all, we can't find all the good sites on our own. At least, I don't think we could, and nor do the powers that be (or else the site suggestion feature wouldn't be there.)

Also, we don't really try to list the sites that are most worthy first, before the ones that are less worthy. We just try to list the sites that are listable.

Your description of what you as an editor had to do

Click to expand...

A point of clarification.

I did not hve to do that.

No one told me that I had to do it.

It needed to be done, and I did it. It was my choice. Nothing special, nothing heroic. It is the same choice that every editor makes the instant they log on:

"What am I going to do today to help the directory?"

georgez said:

I don't understand your argument. How can you determine whether one site is more worthy than another unless you review both? In which case my argument still stands, I think...

Click to expand...

We don't. A site must be worth a listing purely on its own. It is either yes or no. Nothing about is it worth more than any other site so it should be listed first. Reviewing sites would be impossible. I would have to look at all sites (suggested and not-suggested) to determine which one is worth most and list it. How many million sites would that be.

I'm not sure that helps... because my argument is based on the initial assumption that both "suggested sites" are in the correct category and offer unique content.

If someone had suggested a site two years ago for Lacey and another site for Lacey was also suggested two days ago... which one would be more worthy? And which one would you review first?

My argument was that you could not tell which was more worthy without reviewing both. I am however curious on which one you would review first?

I would review the first one I looked at first - how I decide that depends on what I feel like doing at that particular moment. Once I've decided to look at a site - whether it's been suggested, was a link on a site listed already, was on the trunk of a car that passed me on the freeway (true story ) - then that is the site most worthy of a review at that particular moment. If only two sites were suggested - and one of the descriptions looks easier than the other, I might chose that one first. Or I might look at both of the descriptions and decide I'm really not in the mood to work on Lacey right now and not review either of those sites and instead go work on knitting or cooking or watch TV or play with my Granddaughter. Or I might be in the mood to review all the sites waiting for review in Lacey at that particular moment and just go next, next, next. It all depends on what I feel like doing - since I'm the one who is working on that category at that moment. Another editor may feel like doing it differently at that particular moment - or they may not be interested in Lacey at all.

Well, I know submitters are interested in the ODP. And that's OK, they do provide us with a bit of help.

And ANYONE (submitter, editor, webmaster) may have an interest in the ODP based on the ulterior motive of promoting their own site, then ... again, that's understandable.

But, and this is the point, ANY ACTION TAKEN SOLELY BECAUSE OF THAT MOTIVE IS ABUSE!

It doesn't matter who does it. If an editor does it, it's not only abuse, it's also deception and treachery. If a webmaster does it, it's not only abuse, it's also rude spam.

Now, in our experience, submitters ARE getting all the information they need that has any honest, productive use. We tried giving out more information: you can read a few thousand threads, to get a feel for how much good it did; but the consensus of all involved, both editors and submitters, was that it did little or no good. We don't need opinions or speculation: we tried the experiment, and we know.

So there we are. When we discover a site is certainly not spam, we list it, and all the world can see it: even the submitter and webmaster. When we discover a site is certainly spam, we clam up: the submitter and webmaster do not need to know we're on to them.

When we make a mistake, anyone who's demonstrated an ability and willingness to do the work can check the work. Anyone else -- it doesn't matter how much they care, how much they want: it matters how much they CAN (do).

As for automated processes: we don't talk about them much, but quite a few editors have demonstrated talents as programmers, and you can write this down: if editors see a part of their activity that can be automated, and if (based on actual experience) any number of editors judge an automated process would save time, then ... someone will write that process and make it available for any editor who wants to use it.

Most speculation about automated processes is by people who haven't worked on the manual process (and, in any case, are more interested in nailing the editor down so their own spam can be aimed more lethally). The intent is not to HELP editors but to FORCE editors to use tools with known inadequacies that can be exploited. So public discussion of automated tools is counterproductive. And forced use of automated tools is extremely unlikely. If you like, you can think of editorial freedom NOT to use automation, as the essential human check on the quality of the automated tools.

don't understand your argument. How can you determine whether one site is more worthy than another unless you review both? In which case my argument still stands, I think...

Click to expand...

Not really. You said you thought it was unfair that a site suggested recently might get reviewed before a site suggested some time ago. My point was that *when* either site was suggested to us is irrelevant. Whether a site that is added was suggested to us yesterday, last year, or never is meaningless in the grand scheme of things (unless you're a webmaster who believes in FIFO).

georgez said:

I'm not sure that helps... because my argument is based on the initial assumption that both "suggested sites" are in the correct category and offer unique content.

If someone had suggested a site two years ago for Lacey and another site for Lacey was also suggested two days ago... which one would be more worthy? And which one would you review first?

My argument was that you could not tell which was more worthy without reviewing both. I am however curious on which one you would review first?

Click to expand...

OK, let I assume that these are the only 2 sites waiting review in that category and I only have time to review one. Both not very likely to happen but let's assume just to make things easier.
I would review the site which has a supplied title and description compliant with DMOZ guidelines. Why? This person showed to be interested in DMOZ and to have read our guidelines.