Site acceptance or rejection?

First up -- totally, I know my point is mostly moot from any practical perspective. I completely understand the obectives, and I understand there isn't a conspiracy!

My point (whats left of it anyway!), is if there is no benefit to search engine robots not being blocked, then why don't you block them? If you did, then you gain an additional counter argument to anyone claiming the importance of their being listed. No?

Anyway -- zero practical difference, but a 30-second robots.txt editting session and a [albeit, very] minor gain. Thats my £0.02's worth for today!!

All the best,

Peter

>is if there is no benefit to search engine robots not being blocked, then why don't you block them?

Since there IS a benefit, and we've all agreed there is a benefit, why do you keep asking hypothetical questions about situations that cannot happen?

In any case, the policy of the project, from the beginning, has been, "let the users make of it what they will." With that in mind, there is only one conceivable policy for search engine spiders -- "let them come, so long as they do the ODP servers no harm." (IIRC, some rude spiders have been blocked.) That means we don't have to ask who or what might be benefited. We accept that we probably will not know, and certainly will not be allowed to control, all the beneficial effects. Because the ODP social contract says nothing, NOTHING, about anyone having to show a social benefit BEFORE using ODP information.

And we simply aren't that desperate for counter-arguments about sites being listed. No sites are guaranteed a listing -- it says so right in the submittal policy.

And there aren't really any material arguments to counter. What the ODP has, is the links that thousands of people thought important for some reason or another. That constitutes the largest collection of sentient-reviewed links in known space. The presumption is, that's useful. Somehow, sometime, to someone.

There is no presumption that it's complete. There's a welcome for people who want to work on it, in places where it seems (from the point of the user) most incomplete. And that's all.

There's nobody that has standing to argue. The webmaster and his hirelings simply don't get a vote. The ODP is not for their benefit, and if it GAVE them no benefit at all, that would not be a concern at all.

There's nobody to argue about listings with. There is no argument that can be made for a listing, that the site itself doesn't make best. Every other source of argumentation is counterproductive -- at best.

And one should note that many users of DMOZ data, in particular the Google directory, do not use robots to grab the information from the DMOZ.ORG site. Instead they periodically download and parse the DMOZ RDF files and use that to propagate their directories/search engines.

In those cases the use of robots.txt would have no effect.

Thanks for all your answers!!!