So when should one report dead links (cutoff date)

[yum-jelly]

Hi

 

Just one question...

 

I keep a running count of links that are reported dead. Before I do that, I put them in my DNS cache and perform random lookup (different DNS(s)) for the first 28 days. After I try to connect for 21 more days, 4 times a day. The last week I go back to 8 random DNS checks. After that I remove them from the cache.

 

Now I do this because I don't want to kill any user reported dead listings that may be because of the users name servers not finding the site.So how should I report these, I have over 40,000, that have been unresponsive for 60 days, and over 90,000, that have been unresponsive for over 90 days.

 

I have them in this format, but I can format them any way you would like!

 

These are all still in the April 06, 2006 - dump

 

www.robustos.com','http://www.robustos.com/','Robusto\'s Cigar Humidors','Suppliers of wholesale direct storage solutions to consumers and the industry.','Top/Shopping/Tobacco/Cigars/Humidors
www.puroreycigars.com','http://www.puroreycigars.com/','Puro Rey Cigars','Cuban cigars direct from the Caribbean - Habanos, SA authorized dealer.','Top/Shopping/Tobacco/Cigars/Cuban
www.purosimple.com','http://www.purosimple.com/','Puro & Simple','Based in Montreal Canada and featuring a selection of cigars and accessories.','Top/Shopping/Tobacco/Cigars/Cuban
www.echallis.com','http://www.echallis.com/','eChallis','Collection of sarees, dupattas, salwar suits, pretlines, hipsters, skirts, capris and fashion accessories.','Top/Shopping/Clothing/Ethnic and Regional/Asian/Indian
www.fareastbatik.com','http://www.fareastbatik.com/','Far East Batik','Indonesian Batik clothing, including pants, dresses, shirts and robes.','Top/Shopping/Clothing/Ethnic and Regional/Asian/Batik
www.desertimpressions.com','http://www.desertimpressions.com/','Desert Impressions','Custom and preprinted Southwest and Indian motifs, an assortment of our regional wildlife, hot air balloons,water sports, and children\'s t-shirts.','Top/Shopping/Clothing/Ethnic and Regional/North American
www.weefurbished.com','http://www.weefurbished.com/','A Weefurbished Womb','Clothes in sizes newborn to pre-teen, diapers, gifts, toys and maternity wear.','Top/Shopping/Clothing/Children\'s/Baby
www.PanHandleBabies.com','http://www.PanHandleBabies.com/','PanHandleBabies','Handmade baby lounge wear for the newborns up to 12 pounds.','Top/Shopping/Clothing/Children\'s/Baby
preemies2kidsclothing.com','http://preemies2kidsclothing.com/','Sweet Child of Mine','Specializing in clothing for preemies 4 pounds and up, along with infant and toddler.','Top/Shopping/Clothing/Children\'s/Baby
perikitafashion.com','http://perikitafashion.com/','Perikita','Offers Brazilian cut bikinis for both retail and wholesale markets.','Top/Shopping/Clothing/Swimwear/Bikinis
www.scrubsprn.com','http://www.scrubsprn.com/','Scrubs prn','Offers scrubs in a variety of colors and prints. Also offers shoes, and health care products.','Top/Shopping/Clothing/Uniforms/Medical
www.trendyscrubs.com','http://www.trendyscrubs.com/','Hampton Scrubs','Offering medical uniforms, scrubs for men and women.','Top/Shopping/Clothing/Uniforms/Medical
www.rkuniforms.com','http://www.rkuniforms.com/','RK Uniforms','Authorized dealer of uniforms and accessories for the New York Police Department.','Top/Shopping/Clothing/Uniforms/Public Safety
www.countrygoers.com','http://www.countrygoers.com/','Countrygoers','British designer collection of original clothes in natural fabrics.','Top/Shopping/Clothing/Women\'s
www.pinkc.com','http://www.pinkc.com/','Pink C','Clothes and accessories.','Top/Shopping/Clothing/Women\'s
www.scallywag.uk.com','http://www.scallywag.uk.com/','Scallywag','Features tie dye velvet clothing in sizes from small through to extra large.','Top/Shopping/Clothing/Women\'s
www.expectingmaternitywear.co.uk','http://www.expectingmaternitywear.co.uk/','Expecting Maternity Wear','European fashions for relaxing, working, socializing, and partying. Also carries maternity lingerie.','Top/Shopping/Clothing/Women\'s/Motherhood/Maternity
www.funkymamamaternity.com','http://www.funkymamamaternity.com/','Funky Mama','Specializing in youthful maternity clothes in unusual designs.','Top/Shopping/Clothing/Women\'s/Motherhood/Maternity
www.magpieshoard.com','http://www.magpieshoard.com/','Aunty Magpie\'s Shining Hoard','Designer of coin belts, bra covers, drapes, turban decorations and accessories. Scrunchies and falls made from cowrie shells are also offered.','Top/Shopping/Clothing/Niche/Dance/Middle Eastern/Tribal
www.costumesfromtheattic.com','http://www.costumesfromtheattic.com/','Costumes From The Attic','Specializes in tribal belly dance tassel belts and jewelry.','Top/Shopping/Clothing/Niche/Dance/Middle Eastern/Tribal
www.lvclubwear.com','http://www.lvclubwear.com/','Las Vegas Clubwear','Clubwear for men and women. Dresses, pants, shirts and pimpwear.','Top/Shopping/Clothing/Niche/Genre and Subculture/Clubwear
www.urban-imports.co.uk','http://www.urban-imports.co.uk/','Urban Imports','UK based company stocking imported urban clothing brands including G-Unit, Ecko Unlimited, Rocawear, Phat Farm and Enyce.','Top/Shopping/Clothing/Niche/Genre and Subculture/Hip Hop
blisshandbags.com','http://blisshandbags.com/','Bliss','Handbags and accessories made from natural fibers, vintage, and novelty fabrics.','Top/Shopping/Clothing/Accessories/Handbags and Purses/Fabric and Tapestry/Custom and Handmade
www.kattskreations.com','http://www.kattskreations.com/','Katts Kreations','Handmade handbags, purses, totes and accessories.','Top/Shopping/Clothing/Accessories/Handbags and Purses/Fabric and Tapestry/Custom and Handmade
www.exquisite-styles.com','http://www.exquisite-styles.com/','Exquisite Styles','Retailer of replica handbags and watches inspired by designers.','Top/Shopping/Clothing/Accessories/Handbags and Purses/Designer Inspired
www.getknottedties.com','http://www.getknottedties.com/','Getknottedties.com','A range of formal shirts and ties based in the UK.','Top/Shopping/Clothing/Formal Wear/Men\'s
www.fashiondish.com','http://www.fashiondish.com/','FashionDish','"Shopping" area of this web-site gives you the chance to buy from up-and-coming designers.','Top/Shopping/Clothing/Formal Wear/Women\'s/Accessories
www.kidsmoda.com','http://www.kidsmoda.com/','Kidsmoda','Used and new items for infants and toddlers.','Top/Shopping/Clothing/Used and Secondhand/Children
www.angelswardrobe.com','http://www.angelswardrobe.com/','Angels Wardrobe','Offering baby, toddler and older children\'s clothing.','Top/Shopping/Clothing/Used and Secondhand/Children
www.cateschildrenscostumes.com','http://www.cateschildrenscostumes.com/','Cate\'s Children\'s Costumes','Handmade to individual size and specifications. Australia based site offers photos, description, customer comments, and ordering information.','Top/Shopping/Clothing/Costumes/Children\'s
www.sweatersdirect.com','http://www.sweatersdirect.com/','Irish Sweaters Direct','Luxury Irish knitwear: an exclusive collection of handknit fashion, aran sweaters and cardigans direct from Ireland\'s finest manufacturers men\'s, women\'s and children\'s.','Top/Shopping/Clothing/Natural Fiber
www.kiwi-collections.com','http://www.kiwi-collections.com/','Kiwi-Collections','Supplier of woollen apparel and shearling footwear.','Top/Shopping/Clothing/Natural Fiber/Wool
www.softsidegirl.com','http://www.softsidegirl.com/','SoftSideGirl.com','Private-label underwear and camisoles for young women.','Top/Shopping/Clothing/Undergarments/Basics
www.fancy-feet.com','http://www.fancy-feet.com/','Fancy Feet','Socks, tights, legwear and accessories for women and girls.','Top/Shopping/Clothing/Undergarments/Socks and Hosiery
www.sockboys.com','http://www.sockboys.com/','Sock Boys','Socks for all occasions.','Top/Shopping/Clothing/Undergarments/Socks and Hosiery
www.poutt.com','http://www.poutt.com/','Poutt.com','Offers glamourous and sexy sleepwear and bras, panties, and robes.','Top/Shopping/Clothing/Undergarments/Lingerie/Exotic
www.uptowngirllingerie.com','http://www.uptowngirllingerie.com/','Uptown Girl Lingerie','Specializes in complete selection of lingerie such as fantasy, bridal, costumes, leather and teddies.','Top/Shopping/Clothing/Undergarments/Lingerie/Exotic
www.jessiejean.com','http://www.jessiejean.com/','Jessie Jean','Features a collection of lingerie, intimate apparel and women\'s accessories.','Top/Shopping/Clothing/Undergarments/Lingerie/Exotic
wildmountainvision.com','http://wildmountainvision.com/','Wild Mountain Vision','Sweaters, active sports wear, and clothing accessories such as Dale of Norway, Icelandic Design, Tundra Canada, Sierra Designs, and Patagonia.','Top/Shopping/Clothing/Casual
www.glitterheadz.com','http://www.glitterheadz.com/','Glitterheadz','Glitter and rhinestone fashions for girls.','Top/Shopping/Clothing/Casual/T-Shirts

 

 

yj!

[jeanmanco]

My goodness! I think this needs a QC tool expert. I will draw attention to your list internally.

[jeanmanco]

Our QC experts suggest waiting for the next run by Robozilla, our link checker. I notice that you reported a lot of dead links last November. Those should have been cleared out by Robozilla in December. You can see the effect of Robozilla in the ODP report for 2005 at http://research.dmoz.org/publish/chris2001/odp_reports/report_2005.htm

[plantrob]

Thanks for the report, Yum-Jelly! I've sent you a private message at this board by way of followup.

[tschild]

Thanks for the assistance, yum-jelly. I cannot parse "over 40,000, that have been unresponsive for 60 days, and over 90,000, that have been unresponsive for over 90 days." (the first number of sites would have to be be larger, wouldn't it?).

 

What exactly are "dead links" or "unresponsive sites" in your numbers? Sites that have no DNS entry or are 403s, 404s etc. included?

[yum-jelly]

Thanks for the assistance' date=' [b']yum-jelly[/b]. I cannot parse "over 40,000, that have been unresponsive for 60 days, and over 90,000, that have been unresponsive for over 90 days." (the first number of sites would have to be be larger, wouldn't it?).

 

What exactly are "dead links" or "unresponsive sites" in your numbers? Sites that have no DNS entry or are 403s, 404s etc. included?

 

I have (8) servers in different parts of the world, each have 12 different DNS lookup servers that are updated more than most of the other well known DNS servers. So in this case, a dead link means the URL/Domain does not reply to a request from 96 different DNS servers handling the (host to ip) conversion. So in essence, when user tries to visit the URL/link they get no reply! I handle other HTTP/1.0,1.1 (401,404,..) differently, so they are never in the dead link list!

 

As for the (60) days list, not having more than the (90) day list...

 

Each list is maintained separately. Items/domain/URL(s), in the (60) day list are only placed in the (90) day list after they have reached that period of inactivity. So as of today, reported, there are some 150,000 different links that are dead! Granted it might be only 140,000 because some links that are in the April 06, 2006 dump are not listed in the DMOZ. I was going to check each one, but I don't want to overload your servers, doing a scan on each link, to see if it exists in the DMOZ live directory. I think if I did that my IP range would be denied access real quick!

 

 

c, ya...

 

yj!

[yum-jelly]

Thanks for the report, Yum-Jelly! I've sent you a private message at this board by way of followup.

 

I set you a PM, with links to the data requested...

 

Take Care

 

yj!