block IP to access to search.dmoz.org

[hploh]

Hello DMOZ support,

 

I am using odp/dmoz script in my script, however, as tested, it seem not working.

 

When tested on telnet to search.dmoz.org with port: 80 , I get the result as below:

 

C:\Documents and Settings\ebl2>telnet search.dmoz.org 80

Connecting To search.dmoz.org...Could not open connection to the host, on port 80: Connect failed

 

I suspected that my server IP have been blocked.

 

* I have tested from another server (server B), with the same command (telnet search.dmoz.org 80), it was able to connect and the script working fine on server B.

 

Kindly advise.

 

Thanks

[chaos127]

Have you been doing anything that might cause you to be blocked? Does your script obey the robots exclusion protocol before making any requests?

 

Bottom line: while you may get away with some requests, http://search.dmoz.org/robots.txt explicitly says that automated scripts (robots) shouldn't be making requests to that server. If you want to offer your users a search of ODP data, then the 'correct' approach is for you to download the XML data file and process it yourself on your own server.

[Callimachus]

Of course it could be that there is no telnet access as it would be a rather obvious security concern.

[EinsamSoldat]

Dear Community members,

 

I am one of the members which is working with HP Loh on this issue.

 

I have export the script that we are using to another server(different public IP), and the script works fine.

 

I am wondering if search.dmoz.org have blacklisted our server IP and who should we consult on this issue.

 

A pointer about how server/IP is blacklisted will be appreciated.

 

What we have done so far is using different server and using the same scripts, on different server and it works.

 

The telnet testing is we are accessing the port 80 explicitly instead of the default telnet port. Hence we are trying to connect to the http service instead of the telnet service. The rejection shows sign that our current server IP is blacklisted or being denied the http service.

 

I hope that you can provide us the contact for the adminstrator which is blacklisting external host from being able to access search.dmoz.org.

 

Thanks in advanced.

[chaos127]

A pointer about how server/IP is blacklisted will be appreciated.

Unfortunately I don't think that information is publicly available.

 

However, note that the search server is designed to allow individual members of the public to perform searches of the ODP database. If you're attempting to do something that could degrade server performance for other users and has a usage pattern that is not consistent with the purpose of the server, it's not at all unreasonable that your access would be blocked.

 

To examine the problem any further, some more information is probably required: What exactly is it that your script is trying to achieve? How frequent / numerous are the requests it's generating? If this script is being sold / given away, how many people do you anticipate may end up using it simultaneously?

[EinsamSoldat]

Unfortunately I don't think that information is publicly available.

I apologize should our project has causes your server overloaded, would it be possible if we can pm you our IP address so that you can confirm if the IP is blacklisted. and to find out what causes the IP to be blacklisted.

 

To examine the problem any further, some more information is probably required: What exactly is it that your script is trying to achieve? How frequent / numerous are the requests it's generating? If this script is being sold / given away, how many people do you anticipate may end up using it simultaneously?

At the moment, we have one site which is running the script at production stage in which its IP is being blacklisted, based on the wireshark result that we are getting.

 

On our traffic from report Generated by AWStats on Nov 2007, we have about 2500 unique visitors each month and about 260 request were made to search.dmoz.org.

[jimnoble]

We don't discuss the generalities our defence systems and we certainly aren't going to talk about the specifics. Please do as chaos127 recommended - download a copy of our database and work from that.

[EinsamSoldat]

We don't discuss the generalities our defence systems and we certainly aren't going to talk about the specifics. Please do as chaos127 recommended - download a copy of our database and work from that.

Understood, I will not ask about the security is implemented in your systems.

 

However, I need to know from your end should our IP address is blacklisted. If yes, is there any way to have it delisted. Our current main concern is about to get confirmation if our IP is blacklisted.

[informator]

Obviously that is not public information.

 

Please read chaos127´s first reply here again.