up vote 0 down vote favorite
share [g+] share [fb]

Is it recommended to disable curl and wget to disallow anyone from accessing my websites using those commands?

Or is it needed for some search engines to access my websites?

What are your recommendations regarding curl and wget?

link|improve this question
77% accept rate
2  
There is no security concern associated with using curl or wget. Methinks you are misinformed. – gWaldo Oct 18 '10 at 13:07
Besides, I could poll, walk, or pull your website with any number of tools that aren't a web browser, including just about every popular scripting language out there. Here is a 10-year-old book that is still applicable: oreilly.com/openbook/webclient – gWaldo Oct 18 '10 at 13:08
feedback

5 Answers

up vote 1 down vote accepted

Depends what you want to do, and why - you could implement a ban based on user-agent, that's the easiest way, but it won't stop a determined "scraper", but then little will.

It may be that enforcing referrer checks could achieve what you want as well.

Google certainly doesn't scrape with a curl-ush or wget-ish useragent, so it shouldn't affect that.

What web server are you using? Why are you trying to achieve this? These answers whill help us help you :)

link|improve this answer
i am using apache...i am not system admin,,, but i am worrying about security of curl and wget.... do they have any security concern?? – Alaa Alomari Oct 18 '10 at 12:24
1  
Why would they be? They do nothing that your web server wouldn't allow or anything a web browser is capable of doing, really. – Bart Silverstrim Oct 18 '10 at 12:49
1  
@User56166: I don't know where you got "security concern" in regards to wget and curl, but whatever that source was you should throw it out. They are just clients that follow the same protocols as browsers do for accessing your content which YOU are publishing. Trying to block them is like being an artist but saying anybody wearing glasses is not allowed to view your paintings. It just doesn't make any sense. – Caleb Oct 18 '10 at 17:54
3  
Ah - I'd presumed you had some uh.. "premium content" you were looking to block access to for those trying to download lots of it at once. If that's not the case, and it really is "will someone hack my site with wget/curl" then the short answer is: if they are going to do, this sort of thing won't stop them - you'd need to be looking at the underlying weaknesses. – Tom Newton Oct 19 '10 at 9:20
Thanks,, in fact this was my concern... our system admin recommended to disable them to avoid downloading the whole website using such commands and thus making load on the server!! – Alaa Alomari Mar 11 '11 at 22:28
feedback
up vote 4 down vote

As everybody has noted, doing this is nearly impossible and trying gains you nothing. Don't waste your time. You will only hurt yourself by trying to limit access methods.

If your goal is to distribute content, welcome to the web. If it's not, fighting it is a losing battle. If you have subscription content, consider proper methods including user authentication.

Edit: Since the questioner here has reiterated that this is about "security concerns", I thought it might be helpful to explain the pieces.

The tools wget and curl are just that, tools for looking at your content. They follow the same protocols as regular browsers, they just happen to have different interfaces for the user. In fact there are dozen, even hundreds, of different tools people use to retrieve content off the web. Your job as a server administrator is to manage what happens ON YOUR SERVER and the software there, not micro-manage your content-consumers lives.

Think of yourself as an artist painting a picture. Your job is to get the picture right, not limit your viewers by saying anybody wearing glasses is not allowed to view your paintings. It just doesn't make sense.

link|improve this answer
+1 for "welcome to the web..." – gWaldo Oct 18 '10 at 17:29
feedback
up vote 3 down vote

There is no security concern associated with using curl or wget. Methinks you are misinformed.

A web browser's (Be it IE, Firefox, Chrome, Safari, Webkit, etc.) entire purpose is to download and present the contents of web pages from a http server. curl's and wget's purpose is to download files from a http server. That's the major difference; the command-line tools just don't display your page graphically...)

As others have said, you could block by userAgent, but it's a losing battle... It's not actually going to solve anything.

link|improve this answer
feedback
up vote 2 down vote

The only way to block a particular HTTP client is to identify it from the agent string it provides in the HTTP request headers, and as this can easily be faked there is little point - anyone wanting to use wget or curl to scrape data from your site will do so by providing a different agent string (perhaps telling it to imitate Internet Explorer).

link|improve this answer
feedback
up vote 0 down vote

disable curl, I think it would cannot reading your website by search engines like google .. because the curl functions to read, view and retrieve content. correct this if wrong

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.