Why should I firewall servers?

Question

PLEASE NOTE: I'm not interested in making this into a flame war! I understand that many people have strongly-held beliefs about this subject, in no small part because they've put a lot of effort into their firewalling solutions, and also because they've been indoctrinated to believe in their necessity.

However, I'm looking for answers from people who are experts in security. I believe that this is an important question, and the answer will benefit more than just myself and the company I work for. I've been running our server network for several years without a compromise, without any firewalling at all. None of the security compromises that we have had could have been prevented with a firewall.

Edited to add: I guess I've been working here too long, because when I say "servers", I always mean "services offered to the public" not "secret internal billing databases". As such, any rules we would have in any firewalls would have to allow access to the whole internet. Also, our public-access servers are all in a dedicated datacenter separate from our office.

Someone else asked a similar question, and my answer was voted into negative numbers. This leads me to believe that either the people voting it down didn't really understand my answer, or I don't understand security enough to be doing what I'm currently doing.

This is my approach to server security:

1. Follow my operating system's security guidelines before connecting my server to the internet.

2. Use TCP wrappers to restrict access to SSH (and other management services) to a small number of IP addresses.

3. Monitor the state of this server with Munin. And fix the egregious security problems inherent to Munin-node in its default configuration.

4. Nmap my new server (also before connecting my server to the internet). If I were to firewall this server, this should be the exact set of ports incoming connections should be restricted to.

5. Install the server in the server room and give it a public IP address.

6. Keep the system secure by using my operating system's security updates feature.

My philosophy (and the basis of the question) is that strong host-based security removes the necessity of a firewall. Overall security philosophy says that strong host-based security is still required even if you have a firewall (see security guidelines). The reason for this is that a firewall that forwards public services to a server enables an attacker just as much as no firewall at all. It is the service itself that is vulnerable, and since offering that service to the entire internet is a requirement of its operation, restricting access to it is not the point.

If there are ports available on the server that do not need to be accessed by the whole internet, then that software needed to be shut down in step 1, and was verified by step 4. Should an attacker successfully break into the server through vulnerable software and open a port themselves, the attacker can (and do) just as easily defeat any firewall by making an outbound connection on a random port instead. The point of security isn't to defend yourself after a successful attack - that's already proven to be impossible - it's to keep the attackers out in the first place.

It's been suggested that there are other security considerations besides open ports - but to me that just sounds like defending one's faith. Any operating system/TCP stack vulnerabilities should be equally vulnerable whether or not a firewall exists - based on the fact that ports are being forwarded directly to that operating system/TCP stack. Likewise, running your firewall on the server itself as opposed to having it on the router (or worse, in both places) seems to be adding unnecessary layers of complexity. I understand the philosophy "security comes in layers" but there comes a point where it's like building a roof by stacking X number of layers of plywood on top of each other and then drilling a hole through all of them. Another layer of plywood isn't going to stop the leaks through that hole you're making on purpose.

To be honest, the only way I see a firewall being any use for servers is if it has dynamic rules preventing all connections to all servers from known attackers - like the RBLs for spam (which coincidentally, is pretty much what our mail server does). Unfortunately, I can't find any firewalls that do that. The next best thing is an IDS server, but that assumes that the attacker doesn't attack your real servers first, and that attackers bother to probe your entire network before attacking. Besides, these have been known to produce large numbers of false positives.

Answer 1

Advantages of firewall:

1. You can filter outbound traffic.
2. Layer 7 firewalls (IPS) can protect against known application vulnerabilities.
3. You can block certain IP range and/or port centrally rather than trying to ensure that there is no service listening on that port on each individual machine or denying access using TCP/Wrappers.
4. Firewalls can help if you have to deal with less security aware users/administrators as they would provide second line of defence. Without them one has to be absolutely sure that hosts are secure, which requires good security understanding from all administrators.
5. Firewall logs would provide central logs and help in detecting vertical scans. Firewall logs can help in determing whether some user/client is trying to connect to same port of all your servers periodically. To do this without firewall one would have to combine logs from various servers/hosts to get centralized view.
6. Firewalls also come with anti-spam / anti-virus modules which also add to protection.
7. OS independent security. Based on host OS different techniques / methods are required to make host secure. For example TCP Wrappers may not be available on Windows machines.

Above all this if you do not have firewall and system is compromised then how would you detect it? Trying to run some command 'ps', 'netstat', etc. on local system cant be trusted as those binaries can be replaced. 'nmap' from remote system is not guaranteed protection as attacker can ensure that root-kit accepts connections only from selected source IP(s) at selected times.

Hardware firewalls help in such scenarios as it is extremely difficult to change firewall OS/files as compared to host OS/files.

Disadvantages of firewall:

1. People feel that firewall will take care of security and do not update systems regularly and stop unwanted services.
2. They cost. Sometimes yearly license fee needs to be paid. Especially if firewall has anti-virus and anti-spam modules.
3. Additional single point of failure. If all traffic passes through firewall and firewall fails then network would stop. We can have redundant firewalls but then previous point on cost gets further amplified.

Answer 2

TCP Wrappers could be arguably called a host-based firewall implementation; you're filtering network traffic.

For the point on an attacker making outbound connections on an arbitrary port, a firewall would provide a means of controlling outgoing traffic as well; a properly configured firewall manages ingress and egress in a way which is appropriate to the risk related to the system.

On the point about how any TCP vulnerability isn't mitigated by a firewall, you're nt familiar with how firewalls work. Cisco has a whole bunch of rules available for download which identify packets constructed in a way that would cause particular operating systems problems. If you grab Snort and start running it with the right rule set, you will also get alerted on this kind of thing. And of course, Linux iptables can filter out malicious packets.

Basically, a firewall is proactive protection. The farther you get away from being proactive, the most likely that you'll find yourself in a situation where you're reacting to a problem rather than preventing the problem. Concentrating your protection at the border, as with a dedicated firewall, makes things easier to manage because you have a central choke point rather than duplicating rules everywhere.

But no single thing is necessarily a final solution. A good security solution generally is multi-layer, where you have a firewall at the border, TCP wrappers at the device, and probably some rules on internal routers as well. You should usually protect the network from the Internet, and protect the nodes from each other. This multi-layer approach isn't like drilling a hole through multiple sheets of plywood, it's more like putting up a pair of doors so an intruder has two locks to break instead of just one; this is called a man trap in physical security, and most every building has one for a reason. :)

Answer 3

(You may want to read "Life without Firewalls")

Now: What about having a legacy system for which no patches get published anymore? What about not being able to apply the patches to N-machines at the time you need to do so, while at the same time you can apply them in fewer nodes in the network (firewalls)?

There's no point in debating the firewall's existence or need. What really matters is that you have to implement a security policy. To do so you will use whatever tools will implement it and help you manage, expand and evolve it. If firewalls are needed to do so, that's fine. If they are not needed that's fine too. What really matters is having a working and verifiable implementation of your security policy.

Answer 4

Most of your explanations seem to refute a need for a firewall, but what I don't see is a con to having one, other than the small amount of time to set one up.

Few things are a "necessity " in a strict meaning of the word. Security is more about setting up all the blockades you can. The more work needed to break into your server means less chance of successful attack. You want to make it more work to break into your machines than somewhere else. Adding a firewall makes more work.

I think a key use is redundancy in security. Another plus of firewalls is you can simply drop attempts to connect to any port rather than responding to rejected requests - this will make nmapping a little more inconvenient for an attacker.

Most important to me on the practical note of your question is you can lock SSH, ICMP, and other internal services down to local subnets as well as rate limit incoming connections to help alleviate DOS attacks.

"The point of security isn't to defend yourself after a successful attack - that's already proven to be impossible - it's to keep the attackers out in the first place."

I disagree. Limiting damages can be just as important. (under this ideal why hash passwords? or stick your database software on a different server than your web applications?) I think the old saying "Don't stick all of your eggs in one basket" is applicable here.

Answer 5

A firewall is a tool. It doesn't make things secure in and of itself, but it can make a contribution as a layer in a secure network. That doesn't mean you need one, and I certainly worry about people who blindly say "I've got to get a firewall" without understanding why they think that way and who don't understand the strengths and weaknesses of firewalls.

There are a lot of tools we can say we don't need... Is it possible to run a Windows computer without Antivirus? Yes it is... but it's a nice layer of insurance to have one.

I'd say the same about firewalls - whatever else you can say about them, they are a nice level of insurance. They are not a substitute for patching, for locking machines down, for disabling services you don't use, for logging, etc... but they can be a useful supplement.

I'd also suggest that the equation changes somewhat depending on whether or not you're talking about placing a firewall in front of a group of carefully tended servers, as you seem to be, or a typical LAN with a mix of workstations and servers, some of which might be running some pretty hairy stuff despite the best efforts and wishes of the IT team.

One more thing to consider is the the benefit of creating an obviously hardened target. Visible security, whether it's bright lights, heavy locks and an obvious alarm box on a building; or an obvious firewall on a business' range of IP addresses can deter the casual intruder - they'll go looking for easier prey. This won't deter the determined intruder who knows you have information they want and is determined to get it, but deterring the casual intruders is still worthwhile - especially if you know that any intruder whose probes persists past the deterrent needs to be taken especially seriously.

Answer 6

Should I firewall my server? Good question. It would seem that there is little point to slapping a firewall on top of a network stack that already rejects connection attempts to all but the handful of ports that are legitimately open. If there is a vulnerability in the OS that allows maliciously crafted packets to disrupt/exploit a host, would a firewall running on that same host prevent the exploit? Well, maybe ...

And that is probably the strongest reason to run a firewall on every host: A firewall might prevent a network stack vulnerability from being exploited. Is that a strong enough reason? I don't know, but I suppose one could say, "No one ever got fired for installing a firewall."

Another reason to run a firewall on a server is to decouple these two otherwise strongly correlated concerns:

1. From where, and to what ports, do I accept connections on?
2. Which services are running and listening for connections?

Without a firewall, the set of services running (along with the configurations for tcpwrappers and such) completely determines the set of ports the server will have open, and from whom connections will be accepted. A host-based firewall gives the admin additional flexibility to install and test new services in a controlled way before making them more widely available. If such flexibility is not required, then there is less reason to install a firewall on a server.

On a final note, there is one item not mentioned in your security checklist that I always add, and that is a host-based intrusion detection system (HIDS), such as AIDE or samhain. A good HIDS makes it extremely difficult for an intruder to make unwanted changes to the system and remain undetected. I believe all servers should be running some kind of HIDS.

Answer 7

All great questions. BUT - I'm very surprised PERFORMANCE is not been brought to the table.

For highly (CPU-wise) used Web front ends, local firewall really degrades performance, period. Try a load test and see. I saw this tons of times. Turning off firewall increased performance (request per-sec) by 70% or more.

This trade-off must be considered.

Answer 8

A firewall is additional protection. Three particular scenarios it protects against are network stack attacks (i.e. your server OS has a vulnerability to on specially crafted packets that never reach the level of ports), a successful intrusion making a connection to "phone home" (or send spam, or whatever), or a successful intrusion opening a server port or, less detectable, watch for a port knocking sequence before opening a port. Granted, the last two have to do with mitigating the damage of an intrusion rather than preventing it, but that doesn't mean it's useless. Remember that security is not an all-or-nothing proposition; one takes a layered approach, belt and suspenders, to achieve a level of security that is sufficient for your needs.

Answer 9

I'm no security expert by any means, but it sounds as though you are firewall'ed. It seems as though you've taken some of the core functionality of a firewall and made it part of your policies and procedures. No, you don't need a firewall if you are going to do the same job as a firewall yourself. As for myself, I'd rather do the best I can in keeping up security, but have a firewall looking over my shoulder, but at some point when you can do everything the firewall is doing, it starts to become irrelevant.

Answer 10

A firewall is certainly not needed for smaller setups. If you have one or two servers, software firewalls are maintainable. With that said, we don't run without dedicated firewalls, and there are a few reasons why I maintain this philosophy:

Separation of Roles

Servers are for applications. Firewalls are for packet inspection, filtering, and policies. A web server should worry about serving web pages, and that's it. Putting both roles in one device is like asking your accountant to also be your security guard.

Software is a moving target

The software on the host is always changing. Applications can create their own firewall exceptions. The OS is updated and patched. Servers are a high-traffic "admin" area, and your firewall policies/security policies are often way more important to security than your application configurations. In a Windows environment, suppose somebody makes a mistake at some Group Policy level and turns Windows Firewall off on the desktops PCs, and doesn't realize that it's going to get applied to the servers. You're wide open in a matter of clicks.

Just speaking to updates, firewall firmware updates generally come out once or twice a year, while OS and service updates are a constant stream.

Reusable services/policies/rules, manageability

If I set up a service/policy called "Web Server" once (say TCP 80 and TCP 443), and apply it to the "web servers group" at the firewall level, that is much more efficient (a couple configuration changes) and exponentially less prone to human error than setting up firewall services on 10 boxes, and opening up 2 ports x 10 times. When that policy needs to change, it's 1 change vs. 10.

I can still manage a firewall during an attack, or after a compromise

Say my host-based firewall + application server is getting attacked and CPU is off the chart. To even start to figure out what's happening, I am at the mercy of my load being less than the attacker's to even get in and look at it.

An actual experience - I once messed up a firewall rule (left ports to ANY instead of a specific one, and server had a vulnerable service), attacker actually had a live Remote Desktop session to the box. Every time I'd start to get a session, the attacker would kill off/disconnect my session. If it wasn't for being able to shut down that attack from an independent firewall device, that could have been a lot worse.

Independent Monitoring

The logging in dedicated firewall units is usually far superior to host-based software firewalls. Some are good enough that you don't even need external SNMP / NetFlow monitoring software to get an accurate picture.

IPv4 Conservation

No reason to have 2 IPs if one is for web and one is for mail. Keep the services on separate boxes, route the ports appropriately via the device designed to do that.

Answer 11

Blockquote Well, you're right, I didn't put any cons in there. Cons: increased network complexity, single point of failure, single network interface through which bandwidth is bottlenecked. Likewise, administrative mistakes made on one firewall can kill your entire network. And gods forbid that you lock yourself out of it in the meantime when it's a 20 minute trip to the server room.

First, adding at most one additional routed hop through your network is not complex. Second, no firewall solution implemented with any single point of failure is completely useless. Just like you cluster your important server or services and use bonded NICs, you implement highly available firewalls. Not doing so, or not recognizing and knowing that you'd do so is very short sighted. Simply stating that there is a single interface does not automatically make something a bottleneck. That assertion shows that you have no idea how to properly plan and deploy a firewall sized to handle the traffic that would flow through your network. You are correct in saying that a mistake in policy can cause harm to your entire network, but I'd argue that maintaining individual policies on all your servers is far more error prone that a single place.

As for the argument that you keep up with security patching and follow security guides; that's a shaky argument at best. Typically security patches aren't available until after a vulnerability is discovered. That means that the entire time you're running publicly addressable servers they are vulnerable until they are patched. As others have pointed out, IPS systems can help prevent compromises from such vulnerabilities.

If you think your systems are as safe as they can be, that's a good confidence to have. However, I'd recommend you have a professional security audit performed on your network. It may just open your eyes.

Answer 12

Have to say Ernie that whilst you seem do a lot to harden your servers and mitigate against attacks and vulnerabilities, I don't agree with your stance on firewalls.

I treat security a bit like an onion in that ideally you have layers that you have to get through before you get to the core, and personally I think it's grossly misguided not to have some form of hardware firewall at your network perimeter.

I'll admit I'm coming at this from the "what I'm used to" angle, but I know that every single month I get a nice little list of that months patches from Microsoft, many of them being exploited in the wild. I'd imagine the same happens for pretty much any OS and set of applications you care to think of.

Now I'm not suggesting firewalls do away with this, nor are firewalls immune to having bugs/vulnerabilities, obviously a "hardware" firewall is just software running on a hardware stack.

That said, I sleep a lot safer knowing that if I have a server that needs only port 443 to be accessible from the web, my perimeter Juniper ensures that only port 443 is accessible from the web. Not only that but my Palo Alto ensures that the traffic coming in is decrypted, and inspected, and logged, and scanned for IPS/IDS - not that it does away with the need to keep the server(s) secure and up to date, but why would you NOT find that a benefit given that it can mitigate for zero-day exploits and good old human error?

Answer 13

First of all, by your talk about port forwarding, I think you're conflating firewalling with NATing. While these days NATs very often function as de facto firewalls, that it is not the purpose for which they were designed. NAT is a routing tool. Firewalls are for regulating access. It's important for clarity of thought to keep these concepts distinct.

It is of course true that putting a server behind a NAT box and then configuring the NAT to forward anything to that server, is no more secure than putting the server directly on the internet. I don't think anyone would argue with this.

Similarly, a firewall configured to allow all traffic is no firewall at all.

But, is "allow all traffic" really the policy you want? Does anyone ever have a need to ssh to any of yours servers from a Russian IP address? While you're tinkering with the configuration of some new experimental network daemon, does the whole world really need access to it?

The power of a firewall is that it lets you keep open only those services that you know need to be open, and maintain a single point of control for implementing this policy.

Answer 14

Why do all of your servers need a public address?

Install the server in the server room and give it a public IP address.

Out of 14 or so servers that I run on a regular basis, only 2 have publicly accessible interfaces.

Edited to add: In other networks that I've had a hand in managing, we've had the ability to turn off/on services at will, whereas we didn't have access to manage the firewall. I cannot even begin to tell you how many times, inadvertently of course, an unneeded service (SMTP) got turned on and left on and the only thing saving us from becoming a spam dump and getting blacklisted in the process, was a firewall.

Also, is all of the traffic that passes between servers, fully encrypted?

You can certainly drive a car 100 mph with no seatbelts, no airbags and bald tires, but why would you??

Answer 15

In the physical world, people secure valuables by putting them in safes. But there is no safe that cannot be broken into. Safes, or security containers, are rated in terms of how long it takes to force entry. The purpose of the safe is to delay the attacker long enough that they are detected and active measures then stop the attack.

Similarly, the proper security assumption is that your exposed machines will, eventually, be compromised. Firewalls and bastion hosts are not set up to prevent your server (with your valuable data) from compromise, but to force an attacker to compromise them first and allow you to detect (and deter) the attack before the valuables are lost.

Note that neither firewalls nor bank vaults protect against insider threats. That's one reason for bank accountants to take two weeks leave consecutively, and for servers to have full internal security precautions even though protected by bastion hosts.

You seem to imply in the original post that you are forwarding "outside world" packets through your firewall directly to your server. In that case, yes, your firewall isn't doing very much. A better perimeter defense is done with two firewalls and a bastion host, where there is no direct logical connection from outside to inside -- every connection terminates in the DMZ bastion host; every packet is examined appropriately (and possibly parsed out) before forwarding.

Answer 16

Vulnerabilities are hard to predict. It is practically impossible to predict which way your infrastructure is going to get exploited. Having a firewall "raises the bar" for an attacker wanting to exploit a vulnerability, and this is the important part, BEFORE you know what that vulnerability is. Additionally, the ramifications of the firewall can be easily understood in advance, so you are unlikely to CAUSE problems with a firewall which are more severe than the problems you are likely to avoid.

This is why "nobody ever got fired for installing a firewall". Their implementation is very low risk, and has a HIGH likelihood of either preventing or mitigating an exploit. Also, most really nasty vulnerabilities end up being exploited by automation, so anything "unusual" will end up breaking a bot, or at least get it to skip you in favor of an easier target.

Side note; firewalls are not for the internet only. Your accounting dept. doesn't need ssh/whatever to your ldap server, so don't give it to them. Compartmentalizing internal services can make a big difference to the cleanup job in the event that something DOES breach the castle walls.

Answer 17

Firewalls can prevent system users from opening up network-accessible services that the administrators aren't aware of, or doing port forwarding to other machines. By making use of the hashlimit module, firewalls can also rate-limit abusers based on the remote IP.

A firewall is another safety net that ensures your policies are adhered to. Sure, don't run services you don't expect to.

I definitely recommend that software updates are applied in a timely manner, for example, but I also recommend firewalls on all machines. It's like when I drive: Sure I try to avoid obstacles and other cars, but I also wear a seatbelt and have airbags just in case the unexpected happens.

Answer 18

You may not be realizing how much you are benefiting from firewalls simply because everybody else is using them. In a day when literally everyone, down to home DSL users have firewalls in place port sniffing has been all but given up as a feasible attack vector. A decent hacker isn't going to waste their time checking for such things.

Answer 19

I believe the main reason in having firewalls is that, the people managing the servers don't have to be the people managing the security. I am a student at the university and we (all the departments etc) run many servers from inside the campus which are visible on outside especially those servers which provide computational services to people of other universities. SSH sessions are required on those servers and the same servers have to be used by us inside the campus for similar work. Here, we cannot separate internal network for external network. The model is important so that all people can run whatever configurations they want on their own servers.

Now, if everyone of us who wanted to run a server had to pay so much attention to security, it would just be impossible to do so. By having all the traffic monitored through the firewall, the university computer center can manage the security while we all just run a "supported" OSes and take basic measures like ports which are open etc.

In case my server gets hacked, I don't have to be alert to detect it. The firewalls detect it and they can the administrators can then knock my server off the network and notify me of the same. It will protect my machine (from further harm) and protect the network too.

Without firewalls, every user would be liable for his own machine and too much of our time would be gone in security thing that actual work. With firewall, we don't have to worry too much about firewalls etc and actually, we can do well to keep our servers open for random access from inside the university (for SSH ports) and still not worry about its security from outside the university network which allows lets say only HTTP ports.