Does the editor reviewing my submission work for our competition?

On the subject of submissions...

A real concern for me about the submission/approval process is the honesty and integrity of the editors. I'm sure the majority are honest and have the Internet community's best interests at heart, but how do we know there isn't a conflict of interest?

I'm waiting on a submission to a very small (~30 entry) sub-directory that has been in the queue now for over a month. The directory hasn't been updated in that time, nor has my submission been approved or rejected.

Since the particular directory obviously doesn't have much traffic associated with it, I can't understand why the editor can't spare 30mins to work on the queue in the last 30 days.

This has me thinking that perhaps the editor isn't exactly - shall we say - not influenced by the outcome. Perhaps he/she would benefit from my submisssion not appearing in the directory. Perhaps the editor infact works for our one of our competitors.

How does the ODP police against this?

I notice that the editor's identities are kept anonymous. Understandable. But it there isn't even a mention of who they work for.

This seems to be a real flaw in the ODP model.

Any editors/administrators there have any thoughts about this. I'm sure its been raised before, but the problem evidently still persists - along with its undesirable consequences.

Actually there are measure in place to head off abuse but you are always welcome to report any abuse if you actually have any evidence of such http://resource-zone.com/forum/index.php?showtopic=5455

It is not against guidelines for an editor to be associated with a site that is listed in the category they edit in as long as they adhere to the guidelines.

By the way, there could be a number of reasons why your site has not been listed. It may not meet guidelines, it has not been reviewed, etc. Although you may feel that a month is a long time, many submitted sites can wait up to 3 or more years.

Well, that was one of my points: How can I gather evidence of abuse when the editor's identity is kept completely secret?

I'm not suggesting that editors not be allowed to be associated with sites in the category. However, the is the potential for conflict of interest.

Perhaps the ODP should be using double-blind reviews, where there are two reviewers for each category who do not know how the other voted. And when the two don't agree, it gets passed upto a more senior editor.

That is a very typical process in situations where conflict of interest may exist. Since the ODP have taken the route of keeping the editors away from public scruitiny, I think it is up to the ODP to take on the responsibility of actively preventing this type of problem.


PS. On the status of my submission - if it had been rejected wouldn't I have received an email about it? And what excuse is there for a submission taking 3 or more years !?!

The ODP has grown to become a relatively important part of the Internet community. But I don't believe its attitude has grown up to reflect that. To me, it is still operating as though it were all fun and games (viz. a 3 year approval time). Why can't the ODP set limits on approval delays just as it has set other limits in its guidelines? I don't think its too much to ask of an organization to respond to its customers and partrons with some degree of timeliness. Is the ODP saying 1 month review cycles is too short?

>How can I gather evidence of abuse when the editor's identity is kept completely secret?

Abuse is abuse no matter who does it. If it isn't evident that some particular site (or family of sites) has been favored, then it isn't MUCH of an abuse, now, is it?

>Perhaps the ODP should be using double-blind reviews, where there are two reviewers for each category who do not know how the other voted. And when the two don't agree, it gets passed upto a more senior editor.

Well, do you REALLY want reviews to take twice as long, or not? make up your mind! A more rational attitude would be: since we obviously have more submitting energy than submittal-processing energy, it's a no-brainer decision to maximize editor efficiency, at all costs -- and submitter efficiency is an abundant, low-value coin, so spend it like water to save editor time! (NEVER the other way 'round!)

>Since the ODP have taken the route of keeping the editors away from public scruitiny, I think it is up to the ODP to take on the responsibility of actively preventing this type of problem.

Every positive action an editor does is under complete public scrutiny. But, more importantly, every action an editor does is under the complete scrutiny of all INTERESTED parties (the ones who actually show an interest by doing something, and who show an interest in the right things by doing something compatible with the ODP mission. In other words, editors.) Who else has earned the right to scrutinize or oversee?

>PS. On the status of my submission - if it had been rejected wouldn't I have received an email about it?

Of course not. Why ever would we do that? If a site is rejected, what interest do we have in it further?

>And what excuse is there for a submission taking 3 or more years !?!

There is no excuse. And none is needed. Nobody, repeat nobody, is answerable to you for who does what when. If we had all done what you have done, the ODP wouldn't exist at all. By your standards, nothing would have been done. We do better than that.

>The ODP has grown to become a relatively important part of the Internet community. But I don't believe its attitude has grown up to reflect that. To me, it is still operating as though it were all fun and games (viz. a 3 year approval time).

It is still operating in the amateur spirit. this is the spirit that MADE it a relatively important part of the internet community.

>Why can't the ODP set limits on approval delays just as it has set other limits in its guidelines?

There are no time limits in the guidelines. Again, it should be obvious that arbitrary numerical constraints hamper editor efficiency. (I don't see how so many people can fail to see this: but ANYTHING that reduces the efficiency of editors logically MUST make the average delay until a review LONGER!)

>I don't think its too much to ask of an organization to respond to its customers and partrons with some degree of timeliness.

Submitters are not "customers," and never will be. (Surfers are our customers.)

Nor are they "patrons". (AOL is our patron.)

In actuality, they are simply a class of volunteers from whom not much of skill or integrity is required, and to whom (in exchange) not much is offered.

You might say (with justice) that we could do more for our real customers. Since that is true, it would be foolish indeed to take on another mission (webmaster services), for which we are not equipped, and in which we are not interested

>Is the ODP saying 1 month review cycles is too short?

There is no synchronization, at any level of volunteer effort. What I did today, partly depends on the expectation that someone else will do something more later. But I do not know who or when. And there is no way for me to make anyone do anything else, on any time scale.

I am saying that any kind of deadline is radically incompatible with the process, the mission, and the community that built the ODP.

For the ODP to offer even a "FIVE YEAR" cycle would be to write a check on the efforts of volunteers five years hence, and we simply don't have access to their account. In other words, such an offering would be a fraud and a delusion.

If that is what you want, you can find many people willing to provide it. I am not.

Keyser Soze said:

The ODP has grown to become a relatively important part of the Internet community. But I don't believe its attitude has grown up to reflect that. To me, it is still operating as though it were all fun and games (viz. a 3 year approval time).

Click to expand...

And for the editors (atleast the ones I know) it still is about "fun and games". We do this as a hobby. And as such we are volunteering as much of our time as we personaly want for this project and in categories we personaly want. With over 600,000 categories and just over 7000 editors changes of some categories getting more and other less attention at a certain time is very big.

First, I want to remark that of all the enquiries I've read in these forums, which has been a few, the vast majority of submitters have been very polite and professional despite their legitimate frustration with the whole process - which typically is beyond their control. If anyone had a right to pound their fist on the table, figuratively speaking, in these threads, it is them. The tone you take in your reply (here specifically, but also in your other postings) is unprofessional, rude and shows a real disrespect for people you are communicating with.

And now back to the discussion at hand.

1. I questioned the ODP's position on keeping editor's affliations secret while also doing nothing to prevent a conflict of interest. I suggested using the double-blind review process. I've seen it work successfully in peer reviewing of scientific publications and in other technical publication fields. Yes, it does add a little more to the workload, but in these other areas that is an acceptable cost to ensure the quality of the work they release.

2. You claim that every action an editor takes is under complete scruitiny. But complete scruitiny would be public scruitiny. Since that isn't the case here, your assertion is false. This is why I suggested the double-blind process, so that the ODP could maintain a level of privacy in its process, yet provide some assurance of being unbiased.

3. Notifying submitters of the ODP's decision just makes sense doesn't it? I can't believe I have to make this point, but since you dispute it, I will. Notifying the submitter indicates that the ball is now in their court as far as the next action is concerned. That may involve resubmitting under a more appropriate category or changing the submission itself or whatever. But without such notification the submitter must simply decide themselves when to resubmit. If this is done before earlier submissions have been looked at then it adds unnecessarily to the workload of the editors (not to mention the submitters who are equally busy themselves). I am surprised that you take the opposing view on this.

4. Setting time limits for reviews also reduces the workload for the same reason. The argument that imposing maximum timelimits decreases efficiency is only valid when no work is being done. In that case, setting a timelimit would force something to be done and therefore require effort and therefore is less efficient than not doing anything which is supremely efficient in a singular sort of way. But that is a bizarre argument. I'm surprised you are using it to justify the ODP's position.

5. You mention that this is a volunteer effort and therefore demands can not be made of them w.r.t. timelines. Well, as I mentioned before, I've volunteered for many a peer-review (in both scientific and comp.sci fields) for organizations of world-class repute and in every case they made it abundantely clear that submissions must be reviewed within a certain timeframe (typically for publication reasons). That seemed quite reasonable to me! Are you suggesting that ODP editor's can assume that they have 5 years or more to review a submission and not think this is too long? If you do then I don't think those editors are doing ODP or the Internet community any favors.

One last point. As you can see, I've put a lot of energy into this discussion - ironically arguing in part about ODP editor's energy. And less you think that I am making demands of others that I wouldn't accept myself, I have infact previously volueentered as an editor of a category; prepared to help the ODP get upto date in that area because it is running behind and out of date. However, my application was declined for reasons undiscernable from the generic reply. I thought that was a real pity that the ODP, if I may be so humble, is turning down qualified, experienced and capable volunteers.


respectfully,
Me.

The first paragraph of my last reply is ofcourse referring to hutcheson's reply not pvgool's.

Since the particular directory obviously doesn't have much traffic associated with it, I can't understand why the editor can't spare 30mins to work on the queue in the last 30 days.

Click to expand...

There is no correlation between the traffic a category might get and when an editor might get around to reviewing suggested sites there. There are over 590,000 categories and only a few hundred editors who can edit in any given category, most of whom have editing privileges in all 590,000+ categories. Imagine the chances of any one of those editors randomly choosing to edit in that particular category. The chances might increase if there is an actual listed editor in a particular category but there is still no guarantee that that editor, when they choose to edit, will choose to review your site within a specific timeframe.

I notice that the editor's identities are kept anonymous. Understandable. But it there isn't even a mention of who they work for.

Click to expand...

An editor's level of public anonymity is their own choice. Some choose to be as anonymous as they can be. Others are more open about their identities and ties. Considering the amount of harrassment that editors can be subjected to, I'm not surprised that most choose to be as anonymous as they can.

This has me thinking that perhaps the editor isn't exactly - shall we say - not influenced by the outcome. Perhaps he/she would benefit from my submisssion not appearing in the directory. Perhaps the editor infact works for our one of our competitors.

Click to expand...

The usual answer the question "Why hasn't my site been reviewed yet?" is "Because no one has had a chance to review it yet." It's that simple. Abuse, while it does happen, is rare.

Perhaps the ODP should be using double-blind reviews, where there are two reviewers for each category who do not know how the other voted. And when the two don't agree, it gets passed upto a more senior editor.

Click to expand...

We could do that. It could very easily triple the amount of it takes for a suggested site to be reviewed, though, since each site would have to be reviewed not once but two or three times. Are you willing to potentially wait 9 years for your site to be reviewed? Seems unlikely given how upset you are that you've had to wait a whole month so far.

That is a very typical process in situations where conflict of interest may exist. Since the ODP have taken the route of keeping the editors away from public scruitiny, I think it is up to the ODP to take on the responsibility of actively preventing this type of problem.

PS. On the status of my submission - if it had been rejected wouldn't I have received an email about it?

Click to expand...

E-mails are not sent to Website owners regarding the status of their suggestions, regardless of whether the site is accepted or rejected.

And what excuse is there for a submission taking 3 or more years !?!

Click to expand...

If no editor has suffiicient interest in the subject to slog through the suggestions in any particular category, then they sit there waiting for someone to come along who does.

The ODP has grown to become a relatively important part of the Internet community. But I don't believe its attitude has grown up to reflect that. To me, it is still operating as though it were all fun and games (viz. a 3 year approval time). Why can't the ODP set limits on approval delays just as it has set other limits in its guidelines?

Click to expand...

Without introducing paid editorial staff (which won't happen), it would be impossible to guarantee review timeframes because volunteers are free to edit where they choose. If you're paying someone to edit (e.g. at Yahoo), you can tell them where to edit. With volunteers, you can try to tell them where to edit but that's likely just going to gain you a lot of inactive or former editors.

I don't think its too much to ask of an organization to respond to its customers and partrons with some degree of timeliness.

Click to expand...

The biggest disconnect between ODP editors and Website owners is that Website owners are not our customers.

Is the ODP saying 1 month review cycles is too short?

Click to expand...

We're saying that we do not promise to review a suggested site within a specific timeframe. Some may be reviewed in a few hours. Some may be reviewed within a few years. Most fall somewhere in between. And keep in mind that reviewing is not the same as listing. Just as we do not promise a review timeframe, we do not guarantee any site a listing.

1. I questioned the ODP's position on keeping editor's affliations secret while also doing nothing to prevent a conflict of interest. I suggested using the double-blind review process. I've seen it work successfully in peer reviewing of scientific publications and in other technical publication fields. Yes, it does add a little more to the workload, but in these other areas that is an acceptable cost to ensure the quality of the work they release.

Click to expand...

The addition to the workload would make any growth in the directory to slow to a crawl. IMO that would be an unacceptable cost.

3. Notifying submitters of the ODP's decision just makes sense doesn't it?

Click to expand...

Not to us it doesn't.

I can't believe I have to make this point, but since you dispute it, I will. Notifying the submitter indicates that the ball is now in their court as far as the next action is concerned. That may involve resubmitting under a more appropriate category or changing the submission itself or whatever.

Click to expand...

Sites aren't rejected because they were suggested to the wrong category or had poor titles or descriptions. Sites are rejected because they are unlistable. If a site is unlistable, why would we feel a need to enter into a dialogue with the site owner?

4. Setting time limits for reviews also reduces the workload for the same reason.

Click to expand...

How does imposing a time limit reduce the workload? Let's say we have a million suggested sites awaiting review. There are only around 7000 active editors across the entire directory, only a few hundred of which are able to edit in all of the categories where those suggested sites are awaiting review. Many of those few hundred have a lot of other duties besides reviewing suggested sites. By imposing an arbitrary time limit for a suggested site review, you'd be forcing those few hundred editors (who are already among the most active in the directory) to become factory workers, slaving away through the suggested site pool that isn't even our top priority. How is that reducing our workload?

5. You mention that this is a volunteer effort and therefore demands can not be made of them w.r.t. timelines. Well, as I mentioned before, I've volunteered for many a peer-review (in both scientific and comp.sci fields) for organizations of world-class repute and in every case they made it abundantely clear that submissions must be reviewed within a certain timeframe (typically for publication reasons). That seemed quite reasonable to me! Are you suggesting that ODP editor's can assume that they have 5 years or more to review a submission and not think this is too long? If you do then I don't think those editors are doing ODP or the Internet community any favors.

Click to expand...

We're suggesting that ODP editors can assume that they have no deadline whatsoever for reviewing any particular suggested site. As long as an editor is contributing to the net growth of the directory, the fact that any particular site is not reviewed is irrelevant in the grand scheme of things. People like to compare the ODP to other volunteer organizations that they are familiar with but the fact of the matter is that the ODP doesn't work the way a peer-reviewed journal or a volunteer fire department (a favourite comparison used by detractors) functions. We do not require that editors dip into the suggested sites pool at all, let alone that they review all suggested sites in any given category's pool within a specific timeframe.

I am the newsletter editor for a couple of non-profit organizations that I belong to. When I took on those roles, I did so with the understanding that I was expected to adhere to publication deadlines. Those roles are completely different from the ODP editor role. The ODP editor role is more casual, more fluid. We impose no deadlines on editors, though editors are free to impose deadlines on themselves, and we impose no minimum activity level on editors.

One last point. As you can see, I've put a lot of energy into this discussion - ironically arguing in part about ODP editor's energy. And less you think that I am making demands of others that I wouldn't accept myself, I have infact previously volueentered as an editor of a category; prepared to help the ODP get upto date in that area because it is running behind and out of date. However, my application was declined for reasons undiscernable from the generic reply. I thought that was a real pity that the ODP, if I may be so humble, is turning down qualified, experienced and capable volunteers.

Click to expand...

The last time you applied, you were sent a personal reply from the reviewing meta editor.

Keyser Soze, it seems that you view editors as people who have volunteered to review submissions. The problem is that view is erroneous. Editors have volunteered to build a directory. Suggestions are one source of listings, but they are far from the only source and in many categories they are the poorest source of listings. ODP is not, and has never promised to be a listing service via submissions/suggestions.

Keyser Soze said:

2. You claim that every action an editor takes is under complete scruitiny. But complete scruitiny would be public scruitiny. Since that isn't the case here, your assertion is false. This is why I suggested the double-blind process, so that the ODP could maintain a level of privacy in its process, yet provide some assurance of being unbiased.

Click to expand...

All editors can view everyone elses edits. Every edit I make can be viewed at any time by any editor anywhere in the directory, whether they have access to the category in question or not. If an editor makes a mistake, error in judgment, or blatant abusive edits-everyone can see them. Senior editors reqularly correct errrors made by others editors and category editors regularly report errors to senior editors for correction. These are taken as priority-along the lines of update requests and usually taken even more importantly. If that isn't scrutiny, I am not sure what is.

Keyser Soze said:

5. You mention that this is a volunteer effort and therefore demands can not be made of them w.r.t. timelines. Well, as I mentioned before, I've volunteered for many a peer-review (in both scientific and comp.sci fields) for organizations of world-class repute and in every case they made it abundantely clear that submissions must be reviewed within a certain timeframe (typically for publication reasons). That seemed quite reasonable to me! Are you suggesting that ODP editor's can assume that they have 5 years or more to review a submission and not think this is too long? If you do then I don't think those editors are doing ODP or the Internet community any favors.

Click to expand...

Thats great, it would appear that those organizations you volunteered your time to were focused on reviewing submissions. The ODP is not. The ODP gladly accepts suggestions, but its editors are under no obligation to review them in any time frame. In many categories the site suggestions are the poorest source of new listings. Editors routinely seek out sites on their own, and just as many new listings are added that way as there are listed from reviewing suggestions.

Keyser mentioned one method, that, in his mind, would "prevent" conflict of interest. I won't describe how malicious webmasters could get around that method: I leave it as a problem for the reader. (In fact, it's trivial: some pretty stupid webmasters have figured it out for themselves, although they didn't get past OUR way of dealing with the problem.)

>> Notifying the submitter indicates that the ball is now in their court as far as the next action is concerned. That may involve resubmitting under a more appropriate category or changing the submission itself or whatever. But without such notification the submitter must simply decide themselves when to resubmit. <<

You missed the point that for 99.9% of all the sites that get rejected we never want to hear from that site, that submitter, or any of their friends, relations, and business partners, ever again. To tell them which piece of sneaky spam has been detected is to weaken our defence against the massive wall of malicious site submissions that spammers send us. Submit once, then forget it, really is the best answer.

If we haven't looked at it yet, then it does not need to be resubmitted. If we have already rejected it then it does not need resubmitting. In both cases it does not need resubmitting. So, there is no point in saying anything other than "submit once, then forget it".