Duplicate Site Entires

Following on from the discussion at http://resource-zone.com/forum/index.php?showtopic=18993 and as a result of looking at some of the porn out of cat it occurred to me that while all that site content was sitting on my hard drives it would be easy enough to look for duplicate sites. So I did. There are a large number of mirrors to be seen.

Is there anyone interested in following up on this or is it a relatively low priority?

I take by the term "mirror" that we're talking about two different listed URLs showing the same content?

Mirrors

Yes, different URLs *identical* content. I have read bits here and there about the general one site one listing ethos. Having pulled all the site content for the porn scan (and also having found a lot of sites going to the same content vis a vis porn) I thought before I delete it why not do a quick MD5 hash of the content for every site, sort the results, group them, and see how many sites have identical content. There are quite a few. To be specific 59,287 dmoz listings go to only 5,316 actual bits of unique content, where at least 3 differnent urls are hitting identical content. Because this was a quickie scan with MD5 we are talking absolutely identical down to the last character.

I would imagine many of these sites would be of interest. They all returned 200 OK status codes in the HTTP headers. The top categories have content like this:

'' ie shooting blanks, 200 OK but simply no content
Website Disabled
403 Error - File Not Found
Tipus de consulta incorrete
Page not found
Evangelical [Denomination] in [Country] CGI
Some Weather CGI
Website Unavailable
Metarefresh to static .gov content
404 Page Not Found
That Weather CGI again
A Forum
Under Construction
Domain Registration Parking Page
CPanel default page
403 File Not Found
newxxx/hotsite
Error Page Not Found
Free Sex Pics Of The Day
A movie CGI

And so on.

Yes, absolutely this is also something we'd be very interested in.

Dups

I will post the dups in the same place as the porny stuff. You should have the link.