0

Is it recommended to disable curl and wget to disallow anyone from accessing my websites using those commands?

Or is it needed for some search engines to access my websites?

What are your recommendations regarding curl and wget?

2
  • 2
    There is no security concern associated with using curl or wget. Methinks you are misinformed.
    – gWaldo
    Oct 18, 2010 at 13:07
  • Besides, I could poll, walk, or pull your website with any number of tools that aren't a web browser, including just about every popular scripting language out there. Here is a 10-year-old book that is still applicable: oreilly.com/openbook/webclient
    – gWaldo
    Oct 18, 2010 at 13:08

6 Answers 6

1

Depends what you want to do, and why - you could implement a ban based on user-agent, that's the easiest way, but it won't stop a determined "scraper", but then little will.

It may be that enforcing referrer checks could achieve what you want as well.

Google certainly doesn't scrape with a curl-ush or wget-ish useragent, so it shouldn't affect that.

What web server are you using? Why are you trying to achieve this? These answers whill help us help you :)

5
  • i am using apache...i am not system admin,,, but i am worrying about security of curl and wget.... do they have any security concern?? Oct 18, 2010 at 12:24
  • 1
    Why would they be? They do nothing that your web server wouldn't allow or anything a web browser is capable of doing, really. Oct 18, 2010 at 12:49
  • 1
    @User56166: I don't know where you got "security concern" in regards to wget and curl, but whatever that source was you should throw it out. They are just clients that follow the same protocols as browsers do for accessing your content which YOU are publishing. Trying to block them is like being an artist but saying anybody wearing glasses is not allowed to view your paintings. It just doesn't make any sense.
    – Caleb
    Oct 18, 2010 at 17:54
  • 3
    Ah - I'd presumed you had some uh.. "premium content" you were looking to block access to for those trying to download lots of it at once. If that's not the case, and it really is "will someone hack my site with wget/curl" then the short answer is: if they are going to do, this sort of thing won't stop them - you'd need to be looking at the underlying weaknesses.
    – Tom Newton
    Oct 19, 2010 at 9:20
  • Thanks,, in fact this was my concern... our system admin recommended to disable them to avoid downloading the whole website using such commands and thus making load on the server!! Mar 11, 2011 at 22:28
4

As everybody has noted, doing this is nearly impossible and trying gains you nothing. Don't waste your time. You will only hurt yourself by trying to limit access methods.

If your goal is to distribute content, welcome to the web. If it's not, fighting it is a losing battle. If you have subscription content, consider proper methods including user authentication.

Edit: Since the questioner here has reiterated that this is about "security concerns", I thought it might be helpful to explain the pieces.

The tools wget and curl are just that, tools for looking at your content. They follow the same protocols as regular browsers, they just happen to have different interfaces for the user. In fact there are dozen, even hundreds, of different tools people use to retrieve content off the web. Your job as a server administrator is to manage what happens ON YOUR SERVER and the software there, not micro-manage your content-consumers lives.

Think of yourself as an artist painting a picture. Your job is to get the picture right, not limit your viewers by saying anybody wearing glasses is not allowed to view your paintings. It just doesn't make sense.

1
  • +1 for "welcome to the web..."
    – gWaldo
    Oct 18, 2010 at 17:29
3

There is no security concern associated with using curl or wget. Methinks you are misinformed.

A web browser's (Be it IE, Firefox, Chrome, Safari, Webkit, etc.) entire purpose is to download and present the contents of web pages from a http server. curl's and wget's purpose is to download files from a http server. That's the major difference; the command-line tools just don't display your page graphically...)

As others have said, you could block by userAgent, but it's a losing battle... It's not actually going to solve anything.

2

The only way to block a particular HTTP client is to identify it from the agent string it provides in the HTTP request headers, and as this can easily be faked there is little point - anyone wanting to use wget or curl to scrape data from your site will do so by providing a different agent string (perhaps telling it to imitate Internet Explorer).

0

just in case anyone does come across this, I have seen curl used effectively in hacks in order to gain access to data that has an underlying weakness (for example php related exploits), while the posters above are correct that wget and curl can be considered normal functions, why make it that much easier for someone to exploit the data? Site scraping is still an issue but worse still is when these functions can be used to bypass settings or access applications in default folders (of course things should not be placed in default folders anyway but that doesn't stop the loads of fools still not only doing it but also leaving the default usernames and passwords.) You can restrict these by setting flags or via the .htaccess file in Apache. To get to the point, things like libcurl are exploitable in their own right - personally I would ensure that your system is patched and up to date first and foremost while ensuring directories and passwords are non default, then look at restricting certain agents from running if applicable (but as gWaldo and DSpillett both said these can be easily spoofed and there are thousands of other ways to access the data)

1
  • This doesn't actually answer the question. The question is "Is there a problem in disabling curl and wget?"
    – Ladadadada
    Sep 25, 2013 at 13:36
-1

disable curl, I think it would cannot reading your website by search engines like google .. because the curl functions to read, view and retrieve content. correct this if wrong

1
  • I do not understand this response.
    – mdpc
    Dec 16, 2012 at 6:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .