Jump to content

An Outsider's Report of ODP Multiple Listings


Recommended Posts

Posted

We have been researching the ODP Multiple Listing problem. Just browsing through we see tons of spammy sites with hundreds of listings. We personally don't have the time to report them all, so we are letting you know that we are making this resource available to all.

 

DMOZ Top Listeds Domains

 

We hope it helps in the fight against abuse. <img src="/images/icons/smile.gif" alt="" />

Posted

Re: Multiple Listings in sorted order

 

Thanks for the list - it's fantastic!

 

I already caught a really slick hotel reservation affiliate outfit. Would it be possible for you to update the list on any regular basis? Secondly would it be possible for you to drop the inclusion threshold to 5 sites, instead of 10. It's really the sites with 5-20 listings where abusive sites are most likely to lurk.

  • Meta
Posted

Re: Multiple Listings in sorted order

 

If you generate the list by perl or PHP we have some editor-owned servers we could have it run on, if you want to have less work with it <img src="/images/icons/smile.gif" alt="" />

Curlie Meta/kMeta Editor windharp

 

d9aaee9797988d021d7c863cef1d0327.gif

Posted

Re: Multiple Listings in sorted order

 

Very useful list. Thanks for making it available. <img src="/images/icons/laugh.gif" alt="" />

Posted

Re: Multiple Listings in sorted order

 

Excellent resource, thanks very much.

 

You might want to include some method for approving certain sites. Most of the top fifty are hosts or information sites and are obvious candidates for multiple listings (like Geocities, Tripod, AOL, etc.). You also suffer from the same disease that caused SpamCop to block all mail from the UK one day, you only go down to second level domains. This means that at number 17 you have edu.au, which is of course the umbrella domain for all educational establishments in Australia. At number 60 you have sch.uk, which is where all UK school sites live. I notice that co.uk (the umbrella for all UK companies) is missing, so either the scan was broken, or they are already catered for in some way.

Posted

Re: Multiple Listings in sorted order

 

Yes, We plan on updating the list every time a RDF is available. Also the second level domain problem should get better over time, we will be correcting that and getting better in the future. We just need to add these extensions to our lists as roots. So hopefully you will not see the "edu.au" type problems after a new RDF dump.

 

Yes, we can make the list go down to 5 listings. Done. <img src="/images/icons/smile.gif" alt="" />

We are happy to help. We just want to see the abuse stop.

 

>> You might want to include some method for approving

>> certain sites. Most of the top fifty are hosts or information

>> sites and are obvious candidates for multiple listings (like

>> Geocities, Tripod, AOL, etc.).

 

I don’t believe these sites are pure either. Geocities has 98,373 listings. I believe a script that went through and checked those 98,373 for the “This account has been terminated message” would cut those listings in half. If a meta would use that list, I would be glad to crawl these bulk free hosting places! I think DMOZ could be really clean itself if more attention was given to automating the checking of the bulk top 100 hosts. Check for 404s on those hosts.

  • Meta
Posted

Re: Multiple Listings in sorted order

 

>If a meta would use that list, I would be glad to crawl these bulk free hosting places! I think DMOZ could be really clean itself if more attention was given to automating the checking of the bulk top 100 hosts. Check for 404s on those hosts.

 

We check for 404s, but I believe you are right in thinking that a check for known "terminated account" strings would catch a lot more dead "free sites."

 

If you produced this, I believe I can guarantee that it would be used (and not just by metas, this is prime editall-permissions territory.)

 

Please try it out and let us know what you find.

Posted

Re: Multiple Listings in sorted order

 

>> the second level domain problem should get better over time, we will be correcting that and getting better in the future <<

 

Great stuff, thanks.

 

 

>> I believe a script that went through and checked ... for the “This account has been terminated message” would cut those listings in half. <<

 

That's much better than just 'approving' sites to get them removed from the listings. Thanks once again. I'm looking forward to the RDF problem being fixed so that we can get another run with this information available.

Posted

Re: Multiple Listings in sorted order

 

Hi nameintel. Great resource. ;-)

 

When looking at for example ...

 

http://www.whois.sc/geocities.com

GEOCITIES.COM

Website Title: Yahoo! GeoCities

DMOZ: 98373 listings

Website Status: Active

Web server hosts: 7 other websites hosted

IP Address: 66.218.77.68

Visit Website: www.geocities.com

Name Server:

ICANN Registrar:

<no nslookup info?>

 

and

 

http://www.whois.sc/bcsports.net

BCSPORTS.NET

Website Title: BC Gaming

Server Type: Microsoft-IIS/5.0

DMOZ: 2 listings

Website Status: Active

Web server hosts: 2 other websites hosted

IP Address: 196.40.39.253

Visit Website: www.bcsports.net

Name Server: NS1.DIGSOLUTIONS.NET NS2.DIGSOLUTIONS.NET

ICANN Registrar: NETWORK SOLUTIONS, INC.

<and rest of nslookup info>

 

... can you make that "7" and "2 other websites hosted" clickable to a list of the domains hosted on that same IP?

 

I think that would make your resource useful even while reviewing a new domain submission not yet listed in DMOZ.

Posted

Re: Multiple Listings in sorted order

 

 

 

Is there any merit in having the list that is currently sorted in numeric order by the number of published listings, to be also made available as a straight alphabetical sort by domain, ignoring (but still quoting) the numbers? I can think of a few uses for that, looking for similar domain names, but where one may be listed 10 times and the other some other number of times which currently makes it harder to find.

Posted

Re: Multiple Listings in sorted order

 

sabre23t,

Thanks for pointing that out, we just corrected that bug, multiple domains listed at Verisign on that one. These people that list hosts that are the same as domains caused a bug, but we have fixed this. Whew!

 

GEOCITIES.COM.JCHOLLOWAY.COM

GEOCITIES.COM

 

>> ... can you make that "7" and "2 other websites hosted" clickable

>> to a list of the domains hosted on that same IP?

 

Yes, we can make that clickable. We are still deciding on the best way to do this.

 

>> straight alphabetical sort by domain

 

Looks like you have been peeking over our shoulders, something like that is coming very soon. Bot that plain though.

  • 3 months later...
Posted

Re: Multiple Listings in sorted order

 

Bumping up. Now that we've had a handful of new database updates, can we have a new list of multiple domains? <img src="/images/icons/smile.gif" alt="" />

  • 1 month later...
Posted

Re: Multiple Listings in sorted order

 

The DMOZ database on our site will now be automatically updated within 24 hours of a new RDF. Hope that helps.

 

BTW, we have also added Yahoo Directory Listings. So it is possible to see how many listings there are in other directories as well now.

Posted

Re: Multiple Listings in sorted order

 

Very cool. This should maybe be reposted in a more appropriate forum. Which one would be best?

Posted

Re: Multiple Listings in sorted order

 

It was all there a few days ago. Hmm...

  • Meta
Posted

Re: Multiple Listings in sorted order

 

Well, mon, I suppose you're not still wondering whether anyone ever uses it.

 

Thanks again.

Posted

Re: Multiple Listings in sorted order

 

Great work nameintel. <img src="/images/icons/grin.gif" alt="" />

 

I especially like the new http://www.whois.sc/members/reverse-ip.html that is clickable from any domain showing

Reverse IP: Web server hosts 223 websites (reverse ip tool requires free login).

 

<< Very cool. This should maybe be reposted in a more appropriate forum. Which one would be best? >>

 

Perhaps, even in the "General ODP Issues" forum, referring to whois.sc tools such as ...

  • [*]
http://www.whois.sc/dmoz/

[*] http://www.whois.sc/internet-statistics/dmoz-listings.html

[*] and features clickable from whois records such as http://www.whois.sc/about.com (including DMOZ listings and Reverse IP), possibly this whois.sc records may even be worthwhile to be clickable from addurl.cgi on DMOZ (ala links to google cache).[/list:u]

These DMOZ related features would be useful for DMOZ users and editors too.

  • 3 weeks later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...