1

I must setup a production environnement for my company but I've got a problem on a component of the new architecture.
I made a quick draw to show you how it is done today and how I understand it should be done tomorrow.
Everything uses Debian.
EDIT: Everything is stored in the cloud, actually in a small hoster but I plan to migrate to DO

1. Today

Servers have access to the internet and are behind a NAT.
Servers have a security stack installed as described in the end of that post.
Servers have softwares installed in the VM directly.
The schema here on imgur

2. Tomorow

Servers are in a private network.
Apache and MySQL run within docker containers.
Servers are behind a load balancer / proxy / firewall.
I can run a cluster of PHP containers if needed.
EDIT - The new schema on imgur according to answers

3. Where I am

Old question
The entry point is my problem, I don't know which software to use for proxying / load balancing.
I don't really understand if I should use docker in swarm mode because I have two different containers.
I don't know if I should use nginx as I already have Apache handling HTTP requests.
I don't understand how containers can have access to the internet to download configuration files for exemple (puppet files stored on an external git repo).
I'm actually reading a lot on this and trying to learn good practices (as storing data outside of a container) but it is still lil bit unclear.
I've another app running with node and I would like to apply the same schema, just replacing the Apache/PHP container with a PM2/Node container, this is why I'm asking about load balancing. I could also need load balancing for the PHP App.
Maybe I'm totally wrong with that architecture renewal and I should stay with what is it working today ? But I feel that managing security for 2 servers is more dangerous because I have a larger attack surface.
I've also heard than CSF will block docker by default so I have to write additional rules, I would like to avoid it but will it work out of the box with docker in swarm mode ?

EDIT: answers helped me to come to this point, correct me if I'm wrong

Security

If I understand your messages, I should set things following schema in 2.
First I filter all incoming TCP/UDP connections with the firewall, block IPs which try to scan port, etc...
Then I've got my proxy which will route SSH/HTTP/SFTP to the right container depending on the port I hit.
Since traefik can monitor a swarm, I guess its enough to have an eye on what's going on.
I would like to SSH access to my DB VM only via SSH tunnel to avoid exposing a route from the proxy to the DB VM. (I know how to do it)

Scaling

By using traefik, I'm now able to add an orchestrator server and add as many Apache + PHP Container I need to scale up.

Monitoring

Traefik offers good way to monitor services under management.
Logs are sent to an ELK stack for visualization. I also think about sending security logs to the ELK stack.

Sounds good ?

4. Appendix - The Security stack

Forgive my poor knowledge in security, I tried to do my best.
- CSF + LFD
- unattended-upgrades
- clamav
- rkhunter
- logwatch
- fail2ban
- psad
- no ssh root login + port knocking

1

What is your main goal?

Performance

If your goal is performance and you can spare some offline time, then your current setup is fine, and what Quintiliano commented about using your DBs in containers may stand. While using Apache as reverse proxy is easy to manage, for better performance you can check HAProxy (it's not hard but takes time to get used to it). To increase your performance, you may need 2 or more servers and a Load Balancer (Load Balancer makes no sense with a single server). If you have a good connection you can use network interface binding to double your connection speed.

Security

You can improve your security adding a router before your proxy (professional -expensive- or economic but full featured, such as ubiquiti, mikrotik, etc). You need to know how to set it up and protect your private network. In terms of security, is better to run your databases in containers and loose some performance. You can also set your proxy in containers. I would recommend to check traefik which is an easy-to-use Proxy/Load Balancer. You are right, a single server is easier to maintain than two servers.

Redundancy (failsafe)

If what you want is no-down-time, you will need to change your design. You need at least 2 routers and setup VRRP. You need to setup network interface binding and have at least 2 servers (more is best). You setup in your routers to send odd IPs into one of the servers and even IPs to the other (that way you don't need to synchronize sessions), and will load balance your connections. You will need a way to synchronize files from one server to another (in case of file upload, etc). I recommend to use unison, however I use LXD instead of docker, so I'm not sure if that will work in your case. For the database, its better to have them in a cluster, check Galera with maxscale. You usually need at least 3 servers for that, but it is possible to run it with 2 servers if you use Galera Arbitrator

That is only one way to go, there are many paths and this is my personal solution.

UPDATE (about security)

Seriously, how bad it would be if that business data leaks? You said it would be 'Fatal', but it is so? The reason I'm asking you this is because if its that important, then you shouldn't be doing this by yourself, but either with a specialized team or a security consultant. No matter how much you stick to the book, there is always room for mistakes if you don't have enough experience (and if you are asking in this site, is because you don't have it, is that right?). Don't take me wrong, if that data leaks, who is going to be blamed for it? In my personal case, I don't manage top secrets, so as long as I try to follow and apply above-standard security measures, I think its enough. From what I see in you security list, you are covering pretty much what I usually do, and so far I haven't had any security breach so far (that I'm aware of :P).

I have few more suggestions that could improve your setup:

1) Data protection: Please take a look into Data at rest encryption with MariaDB. That is a pretty easy way to get your data encrypted. The main benefit of it is that if your database files get compromised, they are useless without a key. If you store the key in memory is better as it won't be stored in your snapshots or backups, the downside is that you will have to enter that key each time you start the database. This encryption method won't help you if someone has access to the running instance (which is the most common way to hack into a database).

2) Applications: My guess is that your weakest link is in the application layer. You have to be sure your code is well protected and that is not vulnerable to XSS, SQL injections or alike. If you can't be 100% of it, better to keep highly important data as away as possible from your applications. Be sure you add more protection layers (like extra authentication, encryption of values in database, etc)).

3) Encrypted directories: Encrypting directories is usually helpful for those cases in which the server is stolen or someone gains access with a non-root user.

4) Backups: Keep a look where you store your backups and snapshots. They are as important as your databases. If your servers are infected with a ransomware (yes it could happen in Linux too), your backups can help you to save the day. Its a good idea to keep those in a safe location (restricted, encrypted and if possible immutable).

5) Local network: Be sure your servers are isolated from your local network. Don't assume that your local network is safe, so don't grant special permissions for local clients. Treat your server as if it were isolated in a remote place and only accessible through Internet, that way you will only protect a single entry point.

6) SSH: If you are still using port 22 for ssh, change it. I would recommend to use client keys additional to password for ssh (and additional to knock port).

7) Passwords: Be sure you keep all your server passwords safe with a well-tested password manager application. It doesn't matter how much you add into this security formula, if your personal computer is hacked and those passwords are compromised. So protect your personal computer as well as the servers. (you are using strong passwords, right?)

8) Backdoor: It has happened to me several times that I'm locked out! So keep in mind that it can happen to you too. Be sure that if you open an additional way to access the servers, those are as strong as the main one. If you have physical access to the servers, you don't have to worry too much about it.

9) Antivirus: Don't trust too much clamAV. It's not a bad software, but you just won't have the same protection as with a commercial solution. I use it for my mail servers and in some File servers (Samba), to block well-known Windows viruses, but I don't count on it when it comes to new threats. Its better not to assume that it will protect you. As you know there is a very small list of known Linux malware, so if you are having performance issues due to clamAV, it probably won't change your security too much without it (if you are already protecting your server in many other ways). If you want a commercial solution, Sophos and ESET are good options. If you really want to be on top of all threats for your servers, you can try AlienVault (but you will need an extra server just for that).

Key Questions: (What would happen if someone...)

* gains access to the DB through my applications?
* gains access the containers?
* gains access to the servers as user?
* gains access to the servers as root? <-- usually this is gameover
* stole the server?
* has access to the local network?
* gains access to my 'development' computer?

I'm aware that many of the points covered here, you are already doing them, but I didn't want to assume anything. I hope you find my insight useful and if you have any other recommendations that I didn't cover here, I want to hear it.

| improve this answer | |
  • Thanks for your answer, it really helps and give me several approaches. I'm going to explore it today or tomorrow. – user6465431354 May 16 '17 at 8:05
  • The main goal is clearly Security as the database contains business data, a leak could be fatal for the company. EDITING, I didn't want to post – user6465431354 May 16 '17 at 10:22
  • I would go with "security" setup as down-time are not critical at all. They are disturbing but not dangerous for the business, they not even make us loosing money. If I understand correctly : Firewall (CSF) -> Traefik (Reverse Proxy) -> Apache/PHP ? Everything is hosted in the cloud, I plan to use DO as hoster. – user6465431354 May 16 '17 at 11:00
  • Yes, so far so good with your edited question. As I know now you want to focus in Security, there are more details that we can improve. Let me go back to you later (its too late for me now). – lepe May 16 '17 at 14:55
  • Many Thanks. Be sure to check the new schema as I've added some components such as SFTP, NFS and DB Back Up. – user6465431354 May 16 '17 at 16:17
1

My two cents for you.

First of all I think the second approach is a good one. You ll have only one exposed server and is easier to maintain your environment from a security perspective. Besides, with your second approach, its also easier to scale out.

What I would not do.

Using MySQL in containers. I know that docker is a great solution for deploy and is very fast to get things done with it but my experience with Docker + Database was not good. When you use Docker to deploy MySQL nodes you should ensure an external persistence storage for your MySQL DB files, which with the wrong disks and network hardware, can easily become a problem when your application is under high loads. I prefer to have a good VM (or more if needed ) dedicated to the DB and use docker to spawn tons off applications servers connected to this DB. It also solve your problem with 2 VM types in docker swarm. Besides, since you are using puppet, you can provide a new DB node almost as easier as docker would do.

What I would do.

It depends only in your familiarity whit the tool and time to learn new things. If your site has low traffic < 10k requests/s, out of the box, Apache is a good tool for the job. Its easy to configure and maintain and almost every one knows how to handle it. As your load increase, you should look for more robust solutions. I am a big fan of HAproxy. Been using it for almost 6 months and I am loving it. It servers both as L7 ( as HTTP proxy ) and L4 (as TCP proxy, ex: a MySQL load balancer ) proxy/balancer, has tons of options and its way lighter than Apache. Its not hard as soon as you get the idea and you can easily scale it out to handle tons of connections.

Both with Apache and HAproxy you can have a load balancer algorithm sharing the load between your PHP nodes. You probably will need a Memcache/Redis for session/cache distribution and isn`t hard at all to implement. Using a proxy as choke point, you can also use it as TLS off load, decreasing the usage of your PHP nodes.

As in your PHP node, using Apache and Ngnix is only up to you. Again, use the tool that you have knowledge and experience. Both are great servers for PHP.

As for your problem of nodes getting configurations, your proxys can also work as gateways from your internal network. The increase in load would be minimal (usually you make changes on your site after production time). If you do need to transfer high amount of files/configs, I suggest you to install a GIT/SVN or other repository you use inside your site. With this approach you only need to transfers the files from DEV -> PRODUCTION only one time and have puppet or any other tool serve it for your servers inside your LAN. Depending on the size of your environment, it is a huge win.

And finally, for the security side. You can always use the good and old Iptables for holding the line on your proxys and internal servers plus some WAF(web application firewall). Since iptables only let connections in the web ports (I would filter ssh for set of specific origins / tunnel inside a VPN) your attack surface would be the application. WAF can be really troublesome to implement but its a way to go. You can also look for appliances and proprietary solutions for this one. Centralize your applications logs in something like ELK or Gray Log and above all, monitor everything for performance and issues, since disk, memory and disk usage, to link and connections on each proxy and server. This way you can evaluate your site performance and forecast any needed upgrade (always prepare for the highest load) before your clients start to complaint . I like Zabixx for this role.

I hope it helps you.

[]`s

| improve this answer | |
  • Great answer, thanks. I'm going to explore this today or tomorrow. – user6465431354 May 16 '17 at 8:04
  • I agree with the fact that it will be easy to deploy MySQL outside of a container with puppet. I already know how to do it from scratch without puppet so it won't be an issue. I planned to use DO as hoster and they provide block storage so I won't have to manage them. But I understand, I could absolutely use the storage of the VM directly. I will investigate more on this point. – user6465431354 May 16 '17 at 11:07
0

Both given answers are great.

I would like to add a point to the security aspect here.

Main problem is DOCKER.

With docker responsibility for security moves from you to the development team. This is called DevOps.

The problem might be that the development team is not as security aware regarding patches, current java-version and and and (all the stuff admins care about, but no one ever notices).

The solution would be to train developers for their new role.

Since this an error-prone, lengthy process it is better to support this with according techniques. This is called (in Docker-context) DevSecOps.

There are products on the market that integrate with Docker and look for security - fully automated - up to the point where they discard Docker-containers not providing the security set up by guidelines.

That might help in the transition process.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.