How do you remotely administer your Linux boxes?

Question

NOTE: I asked this question this morning with respect to EC2 boxes, but only got back links to tools to start and stop instances, so I'll rephrase...

I have a few Linux boxes that do nightly processing jobs for one of my projects. From time to time, I'll need to get in, make some code changes, configure some things, move files around, etc.

My toolset for these operations is painfully sparse (SSH into the box, edit files in VIM, WGET remote files that I need), and I suspect there is a much better way to do it. I'm curious to hear what other people in my position are doing.

Are you using some form of Windowing system and remote-desktop equivalent to access the box, or is it all command line? Managing remote Windows boxes is trivial, since you can simply remote desktop in and transfer files over the network. Is there an equivalent to this in the Linux world?

Are you doing your config file changes/script tweaks directly on the machine? Or do you have something set up on your local box to edit these files remotely? Or are you simply editing them remotely then transferring them at each save?

How are you moving files back and forth between the server and your local environment? FTP? Some sort of Mapped Drive via VPN?

I'd really need to get some best practices in place for administering these boxes. Any suggestions to remove some of the pain would be most welcome!

Answer 1

My toolset for these operations is painfully sparse (SSH into the box, edit files in VIM, WGET remote files that I need), and I suspect there is a much better way to do it. I'm curious to hear what other people in my position are doing.

Sparse? What on earth do you mean? Excuse me for ranting, but dismissing ssh, vim and wget as painful is almost insulting. From your question I deduce you are mainly a programmer for your daytime job, so I kinda understand the question. But honestly, I would not hire a Linux admin who is not comfortable with any of the three tools you mentioned.

Are you using some form of Windowing system and remote-desktop equivalent to access the box, or is it all command line? Managing remote Windows boxes is trivial, since you can simply remote desktop in and transfer files over the network. Is there an equivalent to this in the Linux world?

For administrator tasks I never, ever use an X environment. You do not need one, it'll only take up system resources and, for the most of the time, they're a hindrance instead of a help. Most GUI configuration tools (well, practically all, really) only offer a subset of the configuration option you can set in a configuration file with vim.

Managing a Linux box is no less trivial than managing a Windows box. It just takes some time to gain a decent skill set.

And a network file transfer equivalent? Plenty. scp, sftp, ftp, nfs, cifs / smb (Windows file sharing protocols), and then some.

Are you doing your config file changes/script tweaks directly on the machine? Or do you have something set up on your local box to edit these files remotely? Or are you simply editing them remotely then transferring them at each save?

Depends on what I am doing. Most of the things I do directly in the config files on the machine (for development and testing boxes) and then I push the file into a configuration channel on our Satellite server, after which I deploy the file to all servers directly (for production boxes). Really, vim is a treasure. That is, when you find out how to use it properly.

How are you moving files back and forth between the server and your local environment? FTP? Some sort of Mapped Drive via VPN?

scp all the way and maybe some sftp, and I suggest you do too. Never, ever use FTP to move sensitive files (e.g. config files) over a public network. I do not use a mapped network because again, all I need is on the server. If you mean c files and not configuration files here, I usually use something like svn or git and then push my changes to the box.

I'd really need to get some best practices in place for administering these boxes. Any suggestions to remove some of the pain would be most welcome!

You are already using them: ssh, scp, wget and vim. Those are not pain. There might be some teething pains, while you figure out how powerful they are. But, to bring the Windows analogy back, I feel seriously hampered when I have to use a Windows box. For you it's the other way around. It's just what you are used to. So, give it some time and it'll come to you.

Answer 2

You mentioned already ssh, vim and wget which is essential and perfect. Some additional tools that can make life easier:

1. GNU Screen

"GNU Screen is a free terminal multiplexer that allows a user to access multiple separate terminal sessions inside a single terminal window or remote terminal session. It is useful for dealing with multiple programs from the command line, and for separating programs from the shell that started the program." (From the GNU_Screen page on wikipedia)

A main advantage is that you can have one or several virtual terminals that are in the exactly same state as you left them when you come back (i.e. relogin via ssh). This is also good when your connection is broken for some reason.

Screen works indepently from the software you use to connect to the box (it lives on the server), so it combines well with putty or most other terminal software.

This article shows some nice things you can do with it: http://www.pastacode.de/extending-gnu-screen-adding-a-taskbar/en/

---

2. Midnight Commander

A console based graphical-like browsing tool for viewing and manipulation of files and directories. Can also do secure remote transfers.

---

3. rsync

For fast, secure and reliable file transfer and synchronisation between different locations

---

4. VCS

Use of a distributed version control system like bazaar, mercurial or git to update code. Github or Bitbucket offer commecrcial code hosting, but it is not necessary, you can also use it efficiently on your own machines.

Joseph Kern: can you elaborate how you exactly use git for remote config organisation?

---

5. Terminal Clients

On unix-like-systems they are already on board, on Windows you can use Putty, Tera Term, Mind Term or Pandora. Or make a cygwin installation and ssh from the cygwin terminel windows to the remote boxes (which has more advantages but this is a question of what you prefer).

---

6. Tunneling and Port Forwarding

It can be helpful to forward certain ports securely to your local machine. For example you could forward the mysql port TCP 3306 or postgres TCP 5432 and install some database administration tool locally.

You can build tunnels from Windwos machines with putty (or command line based with it's little brother plink), with cygwin and Mindterm also can do port forwarding. If you are locally on a unix-like machine you can use ssh odr plink to create such tunneling.

To create some more stable and permanent tunneling for various ports I recommend OpenVPN. The "pre-shared-key" tunneling method from point to point is not so hard to install.

---

7. Have a local unix-like system

When your local machine is a Mac you have this already, you can open a local shell. When your workstation is windows-based it could be helpful to create a local unix-like server, which is in the same local network. This can be a different machine in a different room connected to the same router or switch. Or if you want only one machine, you can install the free vmware server and make a virtual machine, preferably the same operating system as your remote machine. Install a samba server on it and you can "net use" the samba shares from your desktop.

If you an ssh server on the local server and open port 22 on your router for it you can ssh into your local system when you are outside.

You can build tunnels to remote machines or transfer and synchronise files and whole file trees with rsync. You can use it for testing, for VSC, for local development, as a local webserver, for training purposes.

You can pull backups from remote machines. You can create local cron jobs that do te backups autmatically (e.g. databases you want to save locally regularly)

Answer 3

If you're looking for a nice GUI to work with file management via SSH from Windows boxes, have a look at WinSCP: http://winscp.net

I don't administer any EC2 instances, but in general if I have more than a single machine performing a role I'll try and write a script to perform work on all the like boxes, in lieu of making changes box-for-box.

I'd like to get started using Puppet (http://reductivelabs.com/products/puppet/), because it makes system administration more of a configuration management exercise. I haven't had the spare cycles to have a look at it in detail yet, but I've heard very good things.

Answer 4

You need to consider a system configuration management tool like Chef.

I hardly manage systems manually via SSH sessions anymore. I keep all the code for web applications and projects in a source code repository that I can access from the systems that run them, say web sites on EC2. When working in the cloud, it is essential to have automated infrastructure.

Basic work flow looks like this. First, for the application code and configuration data:

• Clone/checkout code from GitHub or other version control repository.
• Edit code in my favorite editor on my local workstation/laptop.
• Commit changes, push to central repository.

For configuration, add:

• Install configuration files in the location where Chef can serve them.
• Trigger a chef-client run, or wait for the interval, on the edge nodes.

Systems get configured like package installs, user creation, config files generated from templates, etc.

I [almost] never manually edit config files, application data, or anything else on the remote systems directly. Changes are done through my local repository and pushed out to the master. I know that my systems will be configured correctly every time, no matter if I have to kill an instance and restart it.

Files needed by the configuration are served right off the Chef server, which is just a web application (merb, running in Apache + Passenger). Access for clients is controlled via openid authorization.

My configuration includes hooks into Nagios and Munin, so I also get monitoring and trending without having to manually do anything to set those up.

Best practices these days are leaning to automated configuration management. If you're still doing things manually, you're working too hard.

Answer 5

What's wrong with ssh, vim, and wget? Jewels unknown to most Windows users, in my opinion. :-)

Sometimes I use gVim remotely over FTP or SFTP.

If you would prefer using the GUI on the Linux machine, you can run an X Server (see Cygwin for one) on your local host and forward your display back through your ssh connection.

I never considered a VPN solution worth the expense or hassle for administering Linux.

If you're doing development and need to run your dev jobs on the server, just set up a revision control client on the server, log in remotely, sync the client, and run your process.

Answer 6

Managing remote Windows boxes is trivial, since you can simply remote desktop in and transfer files over the network. Is there an equivalent to this in the Linux world?

Is this trolling intended?

Remote admin of windows boxes is not trivial. Remote admin of unix boxes is because they were designed to be managed remotely and to be on a network. For windows it was a bolt on to a sytem that wasn't designed from scratch to live on a network.

This is anecdotal but from experience at various companies, the server to sys admin ratio for linux boxes is much, much higher than for windows. Simply because automation on windows is not trivial and a gui will always be slower than scripting. Having said that, there are windows admins who can caress a windows infrastructure into being easy to manage, however these admins are very rare.

All the tools you need are available by default in every linux distro. If you want a graphical login, go ahead and do so, it just wastes CPU and memory and will give you the same problems you have with windows. And as already said, for more than a handful of boxes you should consider puppet,chef, cfengine or one the many other tools available.

Answer 7

I use:

• SSH - For access
• SCP - Copying files
• SSHFS - If I need to mount a drive or directory
• git - Saving configs

Putting my configs under version control, was probably the smartest thing I have done. I use SSH based tools as often as possible to reduce the administrative footprint and attack surface.

Currently all my git repos are local. In the future I'll be moving the configs to remote repos.

Answer 8

SSH has always been enough for me. There are other options X11 is essentially remote desktop though it's inherently insecure as it alone is not encrypted. However it can be tunneled through an SSH connection (and you get the benefit of not having to open up additional ports). This of course assumes that you have an X environment installed on the server.

SFTP (which is ftp over SSH so once again no additional ports need to be opened) can be used to put files on the server rather than pull them a good windows sftp client can be found at http://filezilla-project.org/

Answer 9

I ssh to the box and make change in command line and with editors like vi/nano when change is simple. For more complex change on file, like editing multiple source code at the same time, I use BBEdit on Mac OS X. It can access to the filesystem using SCP. I'm sure there is such editor for windows and linux. sshfs is a way to access your filesystem remotely using ssh.

Answer 10

Definitely PuTTY; as well as the ssh client, there's a Windows-based SCP client as well.

Answer 11

Command-line over ssh for linux (since they haven't even X11 installed) and a mix of ssh and ARD for my Apple machines.

Note that I never use Apple Remote Desktop to transfer files, because AFP over TCP is ridiculously slow; I just use scp.

Answer 12

Don't forget about using Webmin, a web-based interface for system administration for Unix/Linux.

Answer 13

If you want remote graphical access, you'll almost certainly want to set up VNC.

If you want to edit your files remotely over SSH, you've got some options:

• if your using KDE tools (Kate, KDevelop), use the fish protocol.
• set up FUSE to (auto)mount your remote volume.
• Use muCommander (a cross-platform, graphical mc) to move your files back and forth
• some text editors (ex. TextWrangler on the Mac) support getting files over ssh and moving them back and forth

After you've edited your file, just use SSH to, say, start compiling.

One other thing to consider is using a version control system. bzr is really nice; it is easy to set up, and designed to be easy to use. You could work on one computer, push the update, pull the update on your server, execute commands, and revert if things go awry.

Answer 14

SSH does the job most of the time. Learning the command line tools available to you is worth your time. If there are GUI tools I want to use I just use VNC.

For moving files around I usually mount a CFIS/SAMBA share to my local machine.

Answer 15

For simple changes SSH and SCP do the trick. If needing to make changes on a larger number of machines, you should look at a configuration management system. It will be able to start / stop service, change the contents of config files, and verify that your systems are in a known state.

I prefer to use Bcfg2 but Puppet and CFEngine are also popular.

I have Bcfg2 running hourly on all of my servers, though it can be run manually as well. There's a central version-controlled repository of all the system configuration files so we can keep track of any modifications that are made. The system works extremely well and is a huge step up from logging in to servers manually and tweaking things.

Answer 16

I often use Komodo over a SFTP link.

Answer 17

Administering Windows systems has always been "heavy": that is, a complete windowing environment is necessary, which necessitates either being on-site or a network-heavy application like Remote Desktop or VNC.

UNIX was designed with networking in mind - so the tools are designed for working over the network, not for looking pretty. A UNIX server should never have that windowing environment (X11 usually) set up.

The minimalist interface of SSH, wget, and vim is much more conducive to the network - vi was in fact designed to work on a 300baud modem line.

There are, indeed, graphical tools that will assist you working with remote UNIX and Linux systems (wsp was one; gvim may be another).

The usual way of administering UNIX systems is to log into the machine and work on the machine itself. There is no need to move files around just to edit them. The only thing that one must remember is to make sure the configuration is good and to test it where possible.

Answer 18

SSH to access, scp if I need to transfer a limited amount of files. Shared directories via SMBFS/SAMBA if we REALLY need it. Typically it's a waste of time. Some of our less *nix-competent employees use webmin locked to local subnet access over HTTPS.