Software to grade Editors

How do we tell if an ODP editor is doing a good job?

Editor adds a site.
Good Job!

Editor deletes a site.
Good Job!

Editor moves a site.
Good Job!

Any time an editor does one of those three things, even a single time, I say. "Keep up the good work."

I think there's a conversation to be had about heuristically looking at *categories* to find neglected or abused ones (and we do, although not as comprehensively or automatically as might be ideal.)

Heuristically examining editors edits is not a terrible idea, but developing correspondence (as Hutcheson discussed) is problematic. What does a good editor look like, heuristically speaking? I personally have no idea.

>What does a good editor look like, heuristically speaking?

Because of the high degree of flexibility editors have in their work, I think it would be very hard to get an algorithm to pin this down. I'm no computer scientist, but I'm pretty sure that the more different exemplars you have, the harder it is to automatically sort 'good' from 'bad.' An editor who spends most of his time diligently reviewing submissions and publishing many is good. An editor who completely ignores submitted sites and spends most of his time adding brand-new sites he looked up himself is good. An editor who spends most of his time clearing spam and duplicate submissions out of the queues so other editors can work in them is good. And an editor who spends most of his time cruising the directory looking for problematic sites and deleting them is good. These four example types of editors (there are surely many more, and many editors who further complicate matters by switching back and forth between modes and tasks periodically) have dramatically different editing patterns as far as stats go (number of edits vs. number of sites added, number of submissions accepted vs. number of submissions rejected, backlog in categories tended, etcetera). Any algorithm that tried to sort editors based on the aforementioned stats would almost necessarily be hosed from the get-go.

And I think algorithmically sorting out trusted submitters would be a lot of work for little return. Most people *only* submit their own sites, so spending energy analyzing them wouldn't be useful. Almost any librarian who wanted to submit multiple sites on a topic would apply to become an editor after the first few (that's what I did). Who would want to spend so much time filling in endless submission forms when one could become an editor and add them more easily and without delay? So the only potential value to this would be assigning value judgments to the small amount of dedicated SEO's who submit each new client's site--they're really the only ones who keep submitting new sites regularly. I really don't think there's enough of a potential benefit there to justify the amount of work it would take to implement and work through the invariable errors.

There are some other ideas I have for elicting more help from non-editors with useful contributions to make but insufficient skills to actually be editors; none of them involve software, though.

"And Professor Joe and Librarian Larry aren't getting ignored. Believe me, when descriptions are concise and information, and sites are a good fit for the category, they stand out and get reviewed more quickly. I won't speak for everyone else, but when I edit, it is much easier to get those types of sites added quickly."

I think there is a deep misunderstanding and under-appreciation for the skills, talents, and cleverness of spammers.

Writing clear and concise descriptions and making good fits for categories are a fairly trivial thing to do for spammers. These people make a living based on fooling ODP editors and search engines, their profit is often directly in proportional to their ability to do so. I am not saying that fooling you completely will be possible, but they will force you to spend and extra five minutes checking and verifying the link.

You may volunteer to do this, but spammers work *full time*, 10 hours a day, 7 days a week doing this. They usually get paid very very well, have zero ethics, and are extremely street smart.

Unless you have a more sophisticated mechanism for dealing with spam then yes, you will have to start ignoring submissions in their entirety and simply only add websites on a search and find basis.

This is truly unfortunate, as there are literally thousands of individuals who have good intentions and would probably like to submit a URL to ODP now or sometime in the future for reasons other than spam.

But it seems like some people are under the impression that the "social contract" is to give spammers the same amount of time and attention as you do the ones with honest intentions. I am sure there is a "social contract" but I would really hate to think that it is treating spammer submissions the same as you'd treat any other submission is what it is meant for.

Therefore the question is: how can you seperate the wheat from the chaff? My recommendation is to blur the lines between editors and submitters (after all a good editor is partially someone who goes out and submits sites to their own categories!) and keep a record of how well their submissions perform. Is it already in the directory? Did they submit spam? Did they read carefully the description of the category they submitted in? Was it just a deeplink from a bigger site? Etc etc.

This record can be a performance grade which in turn is a heuristic to find the submissions that need immediate attention.

When you go about selecting URLs to check from the record you should check first the ones that are being submitted by those who have a reputation for reading the instructions and following the rules.

This would work especially well with SEO companies. They would develop these user accounts and carefully vet anything they submit via them as they would not want to undermine their ability to submit in the future.

This would also encourage people to submit more carefully (ie: read the instructions!) because they might want to submit again sometime in the future.

Flicker, you bring up an interesting point. Of the 800K submissions, how many unique email addresses are there?

Also bluring the line between submitter and editor is a way to get people more involved with the ODP without having to bring them all the way in. Sometimes, when people read things like "application process" or "editor", they think - "woah, this is waay too much".

But if they become a submitter simply by submitting a URL, there is a very small threshold to becoming a part of the community.

But it seems like some people are under the impression that the "social contract" is to give spammers the same amount of time and attention as you do the ones with honest intentions. I am sure there is a "social contract" but I would really hate to think that it is treating spammer submissions the same as you'd treat any other submission is what it is meant for.

Click to expand...

Perhaps if you took a minute to actually read the social contract, rather than denigrating it, you might be taken a bit more seriously.

This would work especially well with SEO companies.

Click to expand...

And I was beginning to think that you didn't have a sense of humor....

I think there is a deep misunderstanding and under-appreciation for the skills, talents, and cleverness of spammers.

Writing clear and concise descriptions and making good fits for categories are a fairly trivial thing to do for spammers. These people make a living based on fooling ODP editors and search engines, their profit is often directly in proportional to their ability to do so. I am not saying that fooling you completely will be possible, but they will force you to spend and extra five minutes checking and verifying the link.

Click to expand...

I sure wish writing a clear and concise description was a trivial thing for spammers... again, without you understanding the inner workings of things, it is difficult for you to speak intelligently about what it is that editors do or don't do.

Quite clearly, you feel there is improvement to be made. Quite clearly, several posters have given you some feedback concerning your suggestions. You can continue to argue the value of heuristics and software, but without an understanding of what already goes on (which you would gain by becoming an editor), you're speaking from an uninformed stance. The bottom line for me is that your suggestions really don't make sense in light of some things that editors already do, and the basic philosophies of ODP.

Perhaps a specific example is in order:

Let's say we have spammer sue. She purchases 1000 domains from many different registrars for and registers their domain at with po boxes or addresses from 1000 different regional areas around the US.

She then buys 100s of different templates that can be stylized dynamically (so it's not clear that they even were originally a template).

She then buys a directory of hotels, resturants, places to go, things to do, etc.

She then plugs all of this data into a mysql database and hire a 5$/hour guy off of elance to write content based on that database in small, confined chunks that can be arranged dynamically.

Then she creates around 30-40 different 'styles' of regional websites which gather database from the database and arrange it in a way that makes it pretty hard to tell that it is all coming from the same Spammer Sue.

She interlinks with their websites and other websites in a way that even Google has a hard time telling them apart and gives them pr5 to pr6 rankings.

She then host these websites on 1000s of different hosting services. This is trivial in the extreme. They don't even need PHP hosting, they can just write a script to FTP up static webpages generated from PHP web pages.

They wait a few months and then start submitting to directories around the internet.


One of the directories is ODP. Now they submit, each with their own email address, each with their own style of 'concise' clear descriptions and titles to 1000s of regional subsections.

Spammer sue does not submit all at once. She submits all from different IP addresses, with proxies unknown to anyone (base requirement for any good spammer).

And let me tell you - I am not a professional spammer. But even I can figure this much out. Imagine how more sophisticated a professional spammer would be.


Now take 100s of people all over the web who craft original content about their home town. They write lovingly about the local make out point, or the best mexican restaurant in town, etc etc. It's not spam they are generating, but a website with original, well thought out content.

They submit it to the search engines - but guess what! No where to be found. The fact is, their SEO skills are terrible and though the content is *fantastic*, they are not getting the recognition from the search engines that they deserve.

They then submit it to ODP.

This 'social contract' that you talk about. Are you going to tell me that it means that you should treat the submission of original content websites the same way you would treat the submittal of spam?

Wouldn't it be better that these people could go to someone they know who has a good history of submitting to ODP? Or go to an SEO and say, look, could you submit this website for me?

"I heard that if I ask someone who has a good reputation to look at my website and submit it for me it will get into ODP a lot faster"

The people with good reputations will not submit spam, because they have no desire to lose that reputation.

The point being made by the others here is that you're offering critiques and suggestions based on having zero knowledge of what we already have in place. Some of that information you might gain if you were an editor, much of it you wouldn't know unless you were an editall or meta editor.

Since this thread has gone from a simple suggestion of how the ODP could improve to a debate of how the ODP is run (which is against our posting guidelines), I'm going to play the heavy here and close this thread.