Software to grade Editors

If I understand what you are describing, we have a "sandbox" scheme. We call it "greenbusters". It started out as a scheme monitored by a few editalls, but proved popular enough that support for a slightly different implementation was added to the editor interface.

As for the sites actually in unreviewed: we cannot make that public. We don't want spammers to know at what stage of the process, or in what order, or in what connection, their garbage was thrown out. And it can't possibly help 99% of the legitimate submitters. I mean, they submit, they wait, the site gets reviewed, if it has content it nearly always gets added. At that point they see it. The 1% or so of accidentally deleted submittals can be most effectively handled by the standard submittal policy. Adding overhead to that process will cause more delays, not less.

Now for a point of honor. Your use of the word "backlog" has been deceptive, and I believe deliberately so. On the one hand, you denied that you meant "submittals" when you said backlog -- well, actually, you acted offended by the misunderstanding when you were understood to be referring to "submittals", but in your last post your usage of it can have, so far as I can tell, no other meaning.

You are welcome to apologize for the deceit, or you may carry the weight of it added to your reputation. It is your choice.

Also, the best way of dealing with spam is by making the suggest a url more tedious.

Make it so you have to open a user account and fill in lots of information. There are lots of things you can do which constantly suprises me that you do not.

Metrics for evaluating editors will not be discussed here. Rest assured that there is a method of quality control in place.

a) More automation means less time spent on editor evaluation and more time on applicant evaluation

Click to expand...

Some automation is in place, and as you have stated multiple times, this would not negate the necessity of a human to review the automated results. As hutcheson stated, applicant evaluation is not a huge backlog.

b) If someone applies for a small enough category, the standards could be decreased and acceptance could be more automatic(note that I said more, and not just automatic).

Click to expand...

The standards will not be decreased.

c) New applicant editting rights could be curtailed and automated evaluation could help determine when they are ready to take on more responsibilities.

Click to expand...

Thus the procedure of accepting new editors to relatively small categories. And thus the procedure of applying for additional editing permissions.

I'm still puzzled by where this whole discussion originated from on your part...are you "just" a concerned ODP user? webmaster/site owner who tried to submit? etc.?

Not library science. Computer Science. Yes, the concept is not unfamiliar. Minor in Math with focus on Numerical Methods. And teachers with war stories about unvalidated models in fields like operations research, program optimization, rocket science, literature, and experimental psychology. It's been ingrained deep in my hindbrain: check your model before you use it. "The job isn't complete until the error analysis is done." That's why the word "correlation" keeps popping out. You can measure lots of things -- the trick is to figure out what's useful to measure.

CTR is a good measure, if you're trying to see how well you attract people to links -- it is a standard marketroid technique. But we are really in a different world, if not a different universe than that. We don't care which links people follow. We have the Google directory links, so we can spot pages with unexpectedly low page rank (quite often that DOES indicate problems). And the snapshot links, again, proved their usefulness before they were added to the page. To the best of my knowledge, nobody is offering up similar information on CTR. If somebody did offer it, then some editors would probably look at it ... and again, if they claimed it was valuable, more editors would look.

But at this point, we have a long list of things we KNOW would be worthwhile; staff is working on some of them (You should, for instance, see "update URL" requests handled more quickly in the future.) and we'd like staff to work on more of them. coordinating collection of CTR data is not going to make the cut without an outside advocate and pioneer.

touted and publicized?

These are human beings we're talking about, not robotic morons.

Let me tell you a little story. There was once upon a time a secretary who was using a word processor to type up documents. The employer felt that characters per second was a good way to measure employees: and, miribile dictu, the word processor printed a cover sheet for every document with characters per second, um, displayed, touted, and publicized!

Secretary, remember, was human. (Micromanagers and control freaks forget that quite often.) All right, they wanted high CPS, they'd get high CPS. And, after all, they had a computer to help. So secretary established a new procedure: Open blank document, type up and proof everything, "edit/cut all", close document. Open new document, paste, print. Presto, umpteen thousand characters added in less than three seconds. After several weeks of superhuman achievement by one secretary (manager was not that bright, as witness the original notion of using an irrelevant metric), that metric was abandoned.

You think our editors are stupider than that? If we measure number of words per description, the worst editors will churn out descriptions with exactly 17 words, regardless of what gets chopped. If we measure size of "backlog", they'll just nuke the whole "backlog" (or worse yet, accept them all without reviewing.) If we measure "number of moves", they'll leave their mistakes where they lay, or just delete them instead. I don't even want to think about what perversities the random fluctuations of CTR would inflict on the directory!

Oh, as for my lack of concern about the "backlog", I should mention that I have heard it, emphatically, multiple times, from the editor-in-chief. It's proverbial: "We do NOT review submittals; we build a directory." We aren't going to do ANYTHING to give editors the impression that we care differently. This is not my opinion, or my representation of the community's opinion: this is as close to "official Netscape position" as I can get.

To further provide evidence about the value of heuristics and how they could be better applied to ODP, I have an interesting suggestion:


Why not have user accounts for people who wish to suggest URLs? You can even say something like "

1) If you have a user account, your url suggestion will be viewed more quickly.

2) If you have previously submitted more than one url successfullyl, then it is likely your suggestions will be viewed and added very quickly.

3) Please note, however, if your suggestions are for incorrect categories, are already in the directory, or you are spamming the directory than your ability to suggest urls quickly will be undermined."

The more successfull URLs someone suggests, the faster their future suggestions will be reviewed. I am sure a lot of SEO companies would love to have user accounts where they could make a suggestion immediately and have it included. They of course, would be very very careful to vet any suggestion they make so that their suggestions will be treated with a great deal of respect.

All of a sudden this is turning into submission issues instead of grading editors.

Just as an aside to all this fun, I was looking at some numbers and it looks like we have signed up approximately 500 (496) new editors since May 01, 2004. That was out of approx. 2000 applications processed. I'd have to say that's pretty good. I'm actually quite surprised that we found that many good applications - some of them are pretty scary sometimes

Sigh, not at all. If you use a bit of imagination, you can see how I have blurred the line between a 'submitter' and a 'editor' and developed a useful heuristic on measuring their ability to submit/edit, thus very naturally complementing the original concept of this post.

Suddenly, every 'submitter' becomes a potential 'editor'. If they are poorly writing up submissions, putting them into the wrong categories, etc, then their 'submission'/'editting' capabilities are reduced. Editors can graduate from submitters instead of having to be put right into editting capability immediately.

ie: you need to suggest a few urls successfully without a lot of changes/corrections first.

Also, the more suggestions you do correctly, the faster and more likely people will review/add your suggestions to the directory.

But I think we may be looking at the whole topic from widely different perspectives We're not seeing any problem with the way we are currently joining new editors, or the way we are processing submitted sites.

I think that one of the hardest things for people outside of the process to understand is the statement that hutcheson mentioned above - "We do NOT review submittals; we build a directory." Contrary to popular belief we really don't feel like there is a backlog. How can their be a backlog when this is an ongoing process?

This idea has been broached before, but at this point no proposed implementation has attained anything like the necessary support.

The discussion goes something like this:

"So why not just make them an editor?"
"Because we don't trust them that much, or they aren't that good."
"Then what's the point of giving their submittals preferential treatment?"
"To encourage them making more submittals?"
"Ah. We've got a definite shortage of submittals from incompetant professionals we don't quite trust, that's our critical problem, then, is it?"
"Um, no, not exactly."
"So then. What problem, exactly, does this solve for editors?"

And there it bogs down.

Wouldn't it be nice to rather have to weed through 800,000 submissions to get to the 1000 submissions immediately which have been submitted by high quality submitters?

Also, if submitters felt that their ability to submit in the future was determined by how well they did a submittal, don't you think the quality of those 800,000 submittals would increase?

Is there a particular reason why people who submit can't 'help build the directory' or why you can not encourage them to help build the directory?

Yes, well, this is where the heuristic comes in. By keeping a record of their successfull submissions and their unsuccessfull ones you quickly find out which submitters you can trust and which you can not.

Also, quite frankly, a lot of people are *extremely* competent, but don't really want to be an editor and all that entails.

tshephard said:

ie: you need to suggest a few urls successfully without a lot of changes/corrections first.

Also, the more suggestions you do correctly, the faster and more likely people will review/add your suggestions to the directory.

Click to expand...

How is this different from our current process with new editors? We ask new editors to provide us with examples and we judge those and either allow them to continue as an editor, or we deny the application and sometimes ask them to try again.

Once someone becomes an editor they are then free to edit their chosen category. Sometimes they are monitored by the person who joined them, or by any number of editors in the areas. Once they have gained some confidence and experience and have proven that they understand the process, they can then apply for more categorires. The expections are higher for each new category that someone asks for.

BTW - none of this is a secret. We do have a pretty extensive forum here called Becoming an Editor where we answer these questions and do give people information about how to become an editor. We also have extensive internal resources for people once they become an editor.

I think there are a lot of very smart people who have no desire to be an editor and have a lot of interests beyond that of one category and have no desire to review the work of others.

Also, the fact is, you have 800,000 submissions. Breaking that up on a prioritized basis of who is doing the submitting seems like a pretty natural thing to do..

Actually, it is not a very good thing to do.

By creating special classes of submitters, you create a group of second class submitters.

When an editor reviews a submission, the identity of the submitter is irrelevant. Anyone can submit any website. You can't get any more open than that.

I believe that any scheme that places the "average" submitter at a disadvantage violates the spirit of our social contract.

Well tell me that the next time someone registers 100 domains, 100 PHP templates and then gets some guy from india off of elance to write up content for each website. After doing some clever inter domain linking from his other network of websites, he gets PR 5/6 and then submits them all to dmoz.org with different email addresses, from different IP addresses.

So, here you are, thinking "social contract" and reviewing clever spam while Professor Joe and Librarian Larry are getting ignored as they just submit general information websites that they spend a lot of time on and spent considerable time making sure that they are in the right category. And it's not even their websites.

I agree with spectregunner.

And Professor Joe and Librarian Larry aren't getting ignored. Believe me, when descriptions are concise and information, and sites are a good fit for the category, they stand out and get reviewed more quickly. I won't speak for everyone else, but when I edit, it is much easier to get those types of sites added quickly.

So, here you are, thinking "social contract"

Click to expand...

You can mock it all you want -- but that is exactly what we think.

The social contract is the very essence of ODP.

To not be thinking of the social contract would be a major failure on our part.

To try and develop plans, programs, concepts, grading systems, and things like CTRs without giving heavy weight to the responsibilities we assume when we buy in to the social contract, would be (at a minimum) irresponsible on our part. Others might choose stronger/harsher words.

And, as an outsider, until you understand that, you understand nothing about ODP.

It would seem to be very difficult, bordering on impossible, to give suggestions for general tools to solve problems internal to an organization without knowing in advance what those problems are. There really isn't a single sort of quality control software that is useful to companies and organizations across the board. Starting with problems and then brainstorming ways to address them would be one thing, but you seem to be saying "Why not develop some software to solve your problems, and if you don't think the example problems I'm suggesting are important, substitute some of your own." That's a fairly unproductive approach... and as human nature will go, unproductive requests for someone else's self-improvement don't tend to be met warmly. I know what would happen if my husband were to tell me one morning "Say, have you ever thought about keeping some sort of daily log to work on whatever personality problems of yours you think are most critical right now?" It probably wouldn't be friendly. *sheepish grin* Whereas I'm quite happy to entertain "Darn it, you forgot to do the dishes again last night. How can we get it so you remember in the future?"

In your case, most of the examples you've given are things we don't consider problems (approving a site with a low click-through rate, for example--why in the world would we penalize something like that? We are just as happy to index sites on obscure and unpopular topics as sites on trendy and well-known topics.) So it's going to be hard to give a constructive answer to your suggestion that we implement new software measures to deal with issues that not only don't make our top ten list of problems we'd like to see solved, but may not even be considered bad from our perspective at all.

Well tell me that the next time someone registers 100 domains, 100 PHP templates and then gets some guy from india off of elance to write up content for each website. After doing some clever inter domain linking from his other network of websites, he gets PR 5/6 and then submits them all to dmoz.org with different email addresses, from different IP addresses.

Click to expand...

That kind of thing will happen no matter how many precautions are in place. Spammers are determined to have an edge and will do anything to get it. An automated system will not stop them.

I understand that you are trying to be helpful but you need to understand that what you think will help may not be at all helpful to the ODP editors or it's mission.

I have 155 listings in one of my categories (down from 240 or so when I started). I believe that at least half are spam (spam as determined by the ODP guidelines) but I can't find the real source of content for most of them. I doubt an automated system can do a better job of weeding these spammy sites out of the directory. And most were approved by a meta and if they got past him/her then they would most certainly have gotten past an automated system.

Opening the door to less qualified editors will not improve the quality of the ODP and cause more problems than it's worth. It takes a long time to become proficient at editing and it takes a lot of proficient editors to monitor/mentor new editors. There's no easy way to make it faster or better.

Just my 2 cents from a fairly new ODP editor.

Perhaps you should join and learn first hand how the system works.