19

My company's product is essentially a Linux box (Ubuntu) sitting in somebody else's network running our software. Up to now we had less than 25 boxes in the wild and used TeamViewer to manage them.

We're now about to ship 1000 of these boxes and TeamViewer is no longer an option. My job is to figure out a way of accessing these boxes and updating the software on them. This solution should be able to punch through firewalls and what have you.

I've considered:

1. Home grown solution (e.g. a Linux service) that establishes an SSH reverse tunnel to a server in the cloud, and another service in the cloud that keeps track of those & lets you connect to them.

This is obviously labour intensive and frankly speaking feels like reinventing the wheel since so many other companies must have already run across this problem. Even so, I'm not sure we'll do a great job at it.

2. Tools such as puppet, chef or OpenVPN

I tried to read as much as possible but I can't seem to penetrate enough through the marketing speak to understand the obvious choice to go with.

No one else except us needs to connect to these boxes. Is there anyone with relevant experience that can give me some pointers?

  • 2
    "We're not about to ship" => "We're now about to ship"? – Bob Aug 1 '16 at 1:34
23

Pull updates, don't push

As you scale, it's going to become unfeasible to do push updates to all your products.

  • You'll have to track every single customer, who might each have a different firewall configuration.
  • You'll have to create incoming connections through the customer's firewall, which would require port-forwarding or some other similar mechanism. This is a security risk to your customers

Instead, have your products 'pull' their updates periodically, and then you can add extra capacity server-side as you grow.

How?

This problem has already been solved, as you suggested. Here's several approaches I can think of.

  • using apt: Use the built-in apt system with a custom PPA and sources list. How do I setup a PPA?
    • Con: Unless you use a public hosting service like launchpad, Setting up your own apt PPA + packaging system is not for the faint of heart.
  • using ssh: Generate an SSH public key for each product, and then add that device's key to your update servers. Then, just have your software rsync / scp the files required.
    • Con: Have to track (and backup!) all the public keys for each product you send out.
    • Pro: More secure than a raw download, since the only devices that can access the updates would be those with the public key installed.
  • raw download + signature check:

    • Post a signed update file somewhere (Amazon S3, FTP server, etc)
    • Your product periodically checks for the update file to be changed, and then downloads / verifies the signature.
    • Con: Depending on how you deploy this, the files may be publicly accessible (which may make your product easier to reverse engineer and hack)
  • ansible: Ansible is a great tool for managing system configurations. It's in the realm of puppet / chef, but is agentless (uses python) and designed to be idempotent. If deploying your software would require a complicated bash script, I'd use a tool like this to make it less complicated to perform your updates.

Of course, there are other ways to do this.. But it brings me to an important point.

Sign / validate your updates!

No matter what you do, it's imperative that you have a mechanism to ensure that your update hasn't been tampered with. A malicious user could impersonate your update server in any of the above configurations. If you don't validate your update, your box is much easier to hack and get into.

A good way to do this is to sign your update files. You'll have to maintain a certificate (or pay someone to do so), but you'll be able to install your fingerprint on each of your devices before you ship them out so that they can reject updates that have been tampered with.

Physical Security

Of course, if someone has physical access to the customer's deployment, they could easily take over the server. But at least they can't attack the other deployments! Physical security is likely the responsibiltiy of your customer.

If you would for a moment, imagine what would happen if you used a large OpenVPN network for updates... They could then use the compromised server to attack every instance on the VPN

Security

Whatever you do, security needs to be built in from the beginning. Don't cut corners here - You'll regret it in the end if you do.

Fully securing this update system is out of scope of this post, and I strongly recommend hiring a consultant if you or someone on your team isn't knowledgeable in this area. It's worth every penny.

  • 2
    I would second the use of Ansible - it's mid way in complexity between shell scripts and full-blown Puppet/Chef style configuration management, and has the sophistication to do more complex things than just update the software (as hinted at by the question saying "manage"). – RichVel Aug 1 '16 at 13:17
  • If you go the route of using Ansible, you can write it to run on 'localhost', and it won't require SSH access to any of the machines being managed. Configure it to be a cronjob, and you're golden. – BobTuckerman Aug 1 '16 at 14:00
  • 1
    BTW: If you want to run your own package server, fpm and aptly are two great tools that make it much easier to build and host your own packages. Just went through this process recently, and it was pretty nice. – BobTuckerman Nov 14 '16 at 20:58
10

Do you actually need to access them?

Or just update them? Because you can have them update themselves, similar to how apt updates on it's own unattended.

If you need to login

Why not an OpenSSH daemon listening via port forwarding? Each customer can have a separate key for security and would only be connected when needed.

Up to your customers

You need to take into consideration what the customer is willing to accept as well. They might not be comfortable with any remote access to their network, or only comfortable with specific technologies/configurations.

  • 4
    this. with 1000 diffrent customer requirements at least some won't want a permanent openvpn connection back to your offices. Ideally you'd try to make them update themselves if/as/when they detect a new version is available (from a file in an AWS S3 bucket, say. That's what we do. – Sirex Jul 31 '16 at 19:57
  • @Sirex - One drawback of using an S3 bucket is that there's no simple IP whitelist that the customer can use to lock down that server so it can only reach the bucket that holds the update. We ended up having to set up an update server with a static public IP address so customers could use IP filters to control what that server can talk to. (AWS does publish all of their IP blocks, so it's theoretically possible to set up a filter that allows access to only AWS resources, but that's overly broad for this use case) – Johnny Jul 31 '16 at 22:39
  • We dont have the updates on S3, but we do have a text file which details what the latest version is - used by the app to provides the 'update available' banner messages. Customers can then trigger (in our case manually) the download of the latest version, in our case from a service called fetchapp. – Sirex Aug 1 '16 at 3:16
9

I suggest an orchestration tool like Puppet or Salt.

Salt is a message queue and can make a persistent outbound connection from your appliance to a master server. You can use this to run arbitrary commands on the appliances... like an apt-get.

The other option is Puppet, where you still have a master server and the clients make outbound connections from their locations.

I use both of these tools for a similar purpose where I may not have administrative control of the firewall.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.