Software to grade Editors

The value of ODP is directly proportional to how good the Editors are.

If the Editors are misplacing submissions by either ignoring them or by putting them in the wrong categories, then the directory becomes less and less useful.

Therefore, I believe it is important that senior editors have a way of determining who the weaker editors are so that they can help educate and improve how they are handling submissions.

What sort of measurements do you think could be made to determine whether or not an Editor is doing a good job?

Examples:

a) How often a submission they made is switched to another category
b) How often a complaint is made about a particular category they are editing
c) How often a correction (grammatical, factual, etc) is made to a submission that they have accepted and added to the directory
d) The CTR on the submissions they accept versus other submissions in the same category
e) The amount of backlog in the categories they edit

Others? Please suggest.

Editors could see their own grade and even get a breakdown of how they earned their grade. This information should only be shared with a select few editors responsible for evaluations.

Obviously, and I hope this doesn't need repeating, the software will *never* be the final judge and jury and it will *never* be sufficient for finding all of the weak editors.

However, it can and should be one more heuristic in finding out who these editors are.

The measurements you suggest seem far too focused on submissions. One can be an excellent editor without ever looking at submissions.

But, even if we accept that such software would only assess how well submissions are handled, I don't see how it could produce a meaningful score.

a) How do you assign a score to this? The software isn't going to be able to tell if the submission belongs in the category it was moved to or not, so doesn't know whether to assign a negative or a positive number.

b) A lot of complaints may indicate that the editor is doing a good job weeding out spammers, or it may mean the editor is doing a bad job finding sites. Again, no software can know which way to score this.

d) What's a CTR? Is it something that matters to the directory?

e) So, what do we do with editors who have a large backlog? Fire them, thus creating an even larger backlog? Or just cut their pay in half?

It's a heuristic and certainly isn't meant to be definitive. Like life, this doesn't have to be a black/white issue but something that can assist and can be tuned over time.

Also I'm sure there are more measurements .. I was hoping people would use their imagination and determine what those measurements might be rather than criticizing me for not being all knowing.

a) Well, first let me ask you - how would you score it? One potential method is by letting editors rank one categories / submissions. Give the editors one vote per day/week/month whatever, and if you get a bunch of votes that a category created by an editor is a dumb category then maybe that editor's work should be reviewed further. This is just one of many many different possibilities.

b) When I talk about complaints I don't refer to people outside of ODP, I refer to editors. Who cares what people outside of ODP think (j/k)

c) CTR is click-through ratio. Ie, the number of clicks / the number of impressions(ie: page views for that page only). If, for example, you have a category where every link except for one has a 5% CTR and the one that doesn't has a .01% ctr, then it means all the other links are getting clicked on when this one isn't.

To make the example more clear, let's say we had a "blue widgets" category and someone mistakenly but "green daffodils" link in the "blue widgets" category. All the blue widget links will get clicked on, but the green daffodils will not .. because people are not interested in green daffodils when looking at that category.

Therefore if the CTR for a link is low compared to it's peers it is likely that that link probably doesn't belong in that category very well. Again, and I hate to sound repetitive but I get the sense it might be necessary, it's a just a heuristic and is not meant to be a definitive absolute hard and fast rule.

d) You probably don't have very expert people volunteering for stuff because they imagine that certain categories they are interested in are already being maintained. If the editor is just sitting there and backlogging, you may miss out on a lot of great volunteers because they're too polite to want to move in on someone's turf.

Maybe you can keep them on, they could just move from officially handling a category to unofficially handling it.

If the scoring system becomes sophisticated enough then you could probably accelerate editor acceptance to a certain degree and let the system help assist in weeding out the weaker editors.

Meta-editors make suggestions to staff about ways of automating our work reviewing editors, and meta-editors have access to a server to write their own processes if they wish.

You have an interesting idea, but the examples that you give are most telling. Not a single one of those things are considered the least bit relevant by those who evaluate editors' work.

CTR? Our distributors download the whole directory! We dunno what they do with it, or where they post it, or where they click it. (Not, of course, that we would care one whit. Yours may be the only handcrafted Silesian ceramic widget site in this spiral arm, and there may be only two people on the planet who can stand to look at the things without losing their lunch. If it's unique content, and we thank the editor who finds it and categorizes it properly.) This certainly wins the award for "worst idea from a technological perspective."

Corrections? It has been a meta proverb for years: "I don't care how many times it takes you to get it right. I just care that you care that it be gotten right." Call it the "Best way to promote the CYA mentality" award.

Backlog? Our backlog is always the whole internet! The better the editor, the bigger the backlog we entrust to them. And the more enthusiastic an editor, the more backlog they take on. (Submittals? they aren't backlog, they're just the sites non-editors have attempted to categorize.) This is a prime contender for the "That isn't right ... that isn't even wrong" prize.

Site moves? Happens all the time where an editor is really actively growing a category. And meta-editors have foolishly been telling editors for years to build new categories where they can, and then request that ALL their sites be moved elsewhere. Our guidelines and categories are constantly being improved; so this definitely gets the "ex-post-facto" achievement award.

Complaints? Meta-editors aren't allowed, I suppose, to check or validate them first, just count them? This is a real change from our current approach, where it only takes one validated complaint to get an editor removed, but where a zillion spammers' complaints get an editor nothing but an "attaboy!". I think, on consideration, I like the current approach better.

You should also note that editors are not just running wild all over the directory. For the most part we do have a pretty comphrehensive system of checks and balances for keeping track of what's going on. Meta editors are in constant communication about who needs some mentoring and who needs to be shown the door

I'm intrigued by this entire discussion.

We have someone who is not an editor suggesting ways to evaluate editors.

Perhaps we can then get some non-doctors to discuss ways to evaluate brain surgeons.

Perhaps I would treat this more seriously if it were an internal discussion among people who know what editors do and do not do, and who had first identified a specific problem that needed to be fixed and the people leading the discussion had risen through the ranks of editors, and thus understood what editors do and why they do it. The idea that someone is goint to create a software program that evaluates how much initiative and creativity I bring to my volunteerism is, in my opinion, ludicrous and not worthy of a serious response.

Sigh.

This conversation and some of the contributions are a bit demoralizing.

It wasn't my intention to give you all the answers immediately, but rather form a characterization of the possibilties and prompt the community to use their imagination about what are the appropiate heuristics for evaluating editors. I apologize for not being omniscient.

The question should not be - are my examples the authoritative answer to all things regarding evaluating editors, but rather, it should be what do YOU think are the appropiate ways to measure and manage the 62K editors that are developing the greatest directory of knowledge in the history of mankind.

For dmoz.org to scale to all the web (at least, all of the mostly stable web that deserves to be in a directory), all languages and all knowledge, we will need far more than 62K editors.

A good basic course that some of the Meta Editors might want to take is
library science's "Information Systems Effectiveness". There are some really great sections on how to measure and manage a large and complex directory of information.

Also, for the brain surgeon comment, I sure hope you have a master's in library science or otherwise your comment is going to be rather ironic.

Hi tshephard,

I find your suggestions interesting, but I also feel that there is a pretty good system of checks and balances in place, as donaldb said. The problem is truly that you are making suggestions without the knowledge of what systems already exist.

I guess my question is: what prompted you to take a look at this particular aspect of the ODP... do you feel that there are editors slipping through the cracks? Please don't refer to specific editors in this thread, but I'm more curious at what problem you perceive there to be.

Thanks,

I think there is a great deal of frustration with dmoz that is constantly being expressed. For whatever reasons, the editors of dmoz have turned a deaf ear towards it.

We have all seen and heard many a complaint about

a) links clearly misplaced
b) obvious fraud
c) categories which simply increase the complexity and not the effectiveness of dmoz (ie: if you have two categories which are similar but it's not obvious which one a links goes into, this is not always a good thing)
d) categories missing critical links
e) backlog backlog backlog

And you could measure something other than simply the editors; better tools at measuring the directory itself are also important.

For the CTR data, I suspect you could ask Google (or someone) nicely to send their CTR information for the directory and they would be happy to provide it to you.

You may profit by reading a definition of heuristic:

1. Of or relating to a usually speculative formulation serving as a guide in the investigation or solution of a problem: “The historian discovers the past by the judicious use of such a heuristic device as the ‘ideal type’” (Karl J. Weintraub).

The numbers could be tuned to serve as a guide and guide only.

Hi again,

I guess I see it differently. I don't see a deaf ear at all, but rather often, an answer is given that is not satisfactory.

a) links clearly misplaced

Click to expand...

If they are, let us know through the correct ODP channels and they will be fixed. Often, though, a site is categorized correctly by our listing requirements, and that is not where a site owner or webmaster wants it listed.

b) obvious fraud

Click to expand...

Never seen a case of this that was not resolved and taken in a serious manner. There is a system in place to deal with abuse that, in my opinion, works quite well.

c) categories which simply increase the complexity and not the effectiveness of dmoz (ie: if you have two categories which are similar but it's not obvious which one a links goes into, this is not always a good thing)

Click to expand...

Effectiveness is a subjective thing to some extent. In many instances a category that might seem to just add to the complexity is put there to allow linking from another part of the directory...thereby enhancing that part of the directory.

d) categories missing critical links

Click to expand...

Site links or links to other categories?

e) backlog backlog backlog

Click to expand...

Honestly, backlog of unreviewed sites is only one thing that an editor concerns his or herself with. Editors actively seek websites outside of the unreviewed queue for a category. It's not as if the category is necessarily hurting by allowing sites to sit in unreviewed. Again, website owners, etc. are the most bothered by backlog. Backlog is an issue when updates to existing listings sit in unreviewed, and other such instances.

So, my responses might make it seem like I'm turning a deaf ear, but in reality, I really think that the perceived "troubles of ODP" are very different from the difficulties editors might face. It sounds mostly like the software solution you are proposing most addresses B) listed above.

As for real quality problems (which of course have nothing to do with backlogs of any kind) we have ways of reporting them, and we have people reviewing those reports who are in a good position to determine what metrics actually correlate with quality. And when we determine an effective metric, we implement it. Most genuine quality problems are addressed very quickly. General rising-tide quality issues are resolved more slowly, but I'd estimate that a hundred thousand links have been removed from the ODP this year.

As for frustration: you haven't thought this through at all. If you had, you'd have realized that frustration is a null datum.

It is our pride and our passion to frustrate some people -- those who are recycling copied content with the attribution stripped, in order to spam the search engines. That is something that humans do better, and it is desperately needed. A healthy level of frustration is a sign that we're doing it well.

On the other hand, we know there are many good sites out there we haven't reviewed yet. And I'll say this again, because it seems not to have been heard -- it has nothing to do with submittals! The unsubmitted sites are ethically just as valuable as the submitted ones. In fact, in our experience they are usually more valuable. Now, we can feel frustration at the real backlog (not, of course, some irrelevant metric) or we can do something about it. We can curse the darkness, or we can light candles. Or, I suppose, we could go around criticizing how other people supervise the real candle-lighters' way of trimming their wicks.

I am at a bit of a disadvantage here obviously as I do not know what is already in place and obviously no one seems to be volunteering to educate me on this so I can only continue to stumble around in the dark. If my posts seem ignorant, please realise that the posters here have had plenty of opportunity to tell me what is in place, so they are somewhat responsible for that ignorance as well.

However, let me continue to stumble. Hopefully a kind soul will step up and provide specifics rather than philosophy at some point.

a) An implicit motivation behind these ideas are "an ounce of prevention is worth a pound of cure" .. rather than waiting for complaints and corrections, a guide to pinpointing the individuals who are generating the frustration is much more appropiate than simply waiting for someone to come along and complain. Any network engineer knows that you simply do not wait for someone to complain about the network, you have a sophisticated monitoring system in place which lets you know when problems are arising.

b) Effectiveness is subjective. Such is the nature of heuristics, they allow room for subjectivity. They provide a guide and are not meant to be a black/white judge/jury but rather a number which helps you make a more informed decision.

c) Both.

d) "Honestly, backlog of unreviewed sites is only one thing that an editor concerns his or herself with." This is a straw man argument.

You imply that this is my thesis and because it's clearly wrong (yes, I do know it's only one thing that an editor concerncs his / herself with, please see statement c.) therefore my argument is wrong. Trust me, your statement is not my thesis.

I was only trying to say that backlog is one problem. One solution (there are many) is to increase the number of editors at a more rapid rate. Utilize automated scoring systems to help ensure that when you bring on a significant increase in editors the quality of the directory does not decline.

Hi,

We're not giving specifics because the details of how we investigate abuse/fraud, and how we do other things are confidential. It's not to slight you in the least, but I'm afraid you probably won't get too much more info on this.

As far as the backlog goes, I didn't say it was your thesis, but it's a tired argument from everywhere. Why are you concerned with backlog? Why don't we start there.

As far as increasing the number of editors, we try to do that every day, but there are standards for accepting new applications. Yes, heuristics, and statistical analyses of quality vs. editor number might help, the fact remains that having a standard for editor quality is necessary. I know you're not specifically addressing editor quality, but accepting "more editors" leads ultimately to discussions of how strict the standards are.

Seriously, I'm not trying to be combative here, but the fact is... many of the problems you mention are already being addressed in a positive and productive manner. Is there room for improvement? Sure. What can you do? Apply and become an editor.

"As for frustration: you haven't thought this through at all. If you had, you'd have realized that frustration is a null datum."

Frustration is a null datum? Yikes! Frustration by individuals who spam the engine is a null datum, perhaps, but to make a generalized statement like that concerns me greatly considering that you are a meta-editor..

"And I'll say this again, because it seems not to have been heard -- it has nothing to do with submittals!"

Though your replies have certainly undermined my confidence in a way I feel irrelevant to the thesis at hand, I still continue to read your replies carefully. Where did I say it was all about submittals? You see an easy way to avoid thinking about the problem by characterizing my argument as about something it isn't.

Let me be clear: I am discussing the advantages of automating the evaluation of editors and trying to provide some rough examples of how you might do this so that other, more sophisticated ideas might be put forward.

I am not implying that backlog is everything, I am not implying that ODP is all about submittals. I am not implying that the moon is made of blue cheese.

A consequence of the automation, I should say, is that more editors could be accepted at a quicker rate. Thus increasing the number of active editors and increasing the coverage of the directory.

Sorry, I think I am confused now. Automation of..... editor evaluation? I'm not sure how that will lead to accepting more editors.

"We're not giving specifics because the details of how we investigate abuse/fraud, and how we do other things are confidential. It's not to slight you in the least, but I'm afraid you probably won't get too much more info on this"

Perhaps not for fraud, but the metrics for evaluating editors should not only be shared, they should touted and publicized. Understanding the many measurements of a strong editor is useful information for anyone and is a great starting point to knowing what it is that you need to do in order to improve.

There's no way to distinguish spammer frustration with any other kind in mass. All you can do is relieve it individually by ... reviewing sites. So there's no point in measuring it in mass. It's like trying to measure someone's wealth by weighing their pocketbook. There is no correlation. It gives no information! And therefore there is no reason to measure it. That's simple logic.

As for accepting more editors, the editor application backlog is quite short -- all apps are looked at within a day or three. Simple ones processed quickly, borderline cases left for more thorough review. The simple fact is, this isn't a bottleneck.

No, we are absolutely not going to tell you what abuse procedures we have in place. However, we are very very willing for you to tell us what has slipped through our procedures. And we are willing to build "heuristics" that have been demonstrated useful. You are free to attempt such a demonstration: Find a pattern of fraud. Then tell us how you did it.

But so long as you are waving nothing but contraindicated statistics, we can't respond with anything but commonsensical platitudes -- I hardly think this discussion has quite risen to the dignity of "philosophical".

"Automation of..... editor evaluation? I'm not sure how that will lead to accepting more editors."

It's a fair question, I can see three ways immediately, though I am sure there are others:

a) More automation means less time spent on editor evaluation and more time on applicant evaluation
b) If someone applies for a small enough category, the standards could be decreased and acceptance could be more automatic(note that I said more, and not just automatic).
c) New applicant editting rights could be curtailed and automated evaluation could help determine when they are ready to take on more responsibilities


One thing I am surprised by is that there isn't a sandbox notion for ODP. Rather than keeping the backlog out of site, the backlog should be completely visible to users and then 'graduated' into the category proper when an editor reviews it. Junior editors could be responsible for accepting links into the sanbox and making suggestions as to which ones should be "graduated".

I, personally, would love to be able to see all of the backlog for a particular category. ODP Downloaders have the option (or not!) of getting the backlog.

"But so long as you are waving nothing but contraindicated statistics, we can't respond with anything but commonsensical platitudes "

Contraindicated statistics?

CTR, for example, is anything but contraindicated. It is probably one of the most important metrics for the position of links on web pages all throughout the internet.

If on average you are constantly putting links in places with low CTR relative to its peers, then you are likely doing something wrong. If, however, you are placing links with high CTR than you are likely putting links in places that are appropiate for them.

Hutcheson, I thought I had read somewhere that you had a degree in library science. Measurement and metrics play a very large part of that field of study, do they not?