We are engaging a consultant in India as our Linux Administrator. We don't know him well and he requires Root access to all our servers to do his job (including a security audit).
What is the best practice for enabling a remote consultant for such work such that we are protected against any malignant activities?
Thanks in advance.
Don't. Also, you're in as much danger of ineptitude as malice from what I've seen of the typical way companies handle this.
I'd like to say, there's probably great system administrators out there in India, but the way many companies do things is terrible.
If you're going through a body shop, you're also likely seeing a pretty big cut go to them, and many of them are unlikely to have properly vetted their employees. I've talked to three, one of whom I worked for and none of them have done any technical interviews.
So, if you must hire someone remotely, for god's sake, interview him yourself and make sure he knows his work. System administration is far too important to hand over to someone blindly
Now that I've handled the "ineptitude" part of it,
Administration is a pretty broad phrase. And someone with root access can do anything. Now, personally I think creating an account for the admin, and giving him the ability to elevate himself through sudo is a better idea (which your config management system should handle if you have many servers). That said, even that relies on a certain amount of trust. There's so many stories out there of the sheer damage a disgruntled sysadmin can do. Change all your passwords? Sure you could get in eventually, but its not trivial, and it would probably cost more than you're saving.
So, consider a local. If not, consider someone you have vetted yourself and have directly hired.
su authority to elevate themselves to root. When someone says they need "root access", this is what they should be asking for. If this person says he needs "the root password", he's not competent to do the job anyway.
Feb 8, 2017 at 22:53
sudo authority to elevate themselves to root? If not, could you outline your best practices for using su to elevate oneself to root without having, and distributing, a root password?
Feb 9, 2017 at 8:08
sudo and su are two completely different things. Thanks for clarifying.
Feb 9, 2017 at 15:45
As has been mentioned, don't do this.
The only way you'll be able to protect yourself is by doing something like this:
As should be clear, this is a very clumsy and inefficient process, but if you insist on accepting work from a non-trusted individual, this is one way to handle things.
As I recommended, though, you're much better off hiring a known, trusted individual.
What is the best practice for enabling a remote consultant for such work such that we are protected against any malignant activities?
From a legal perspective: due diligence beforehand and strict penalties for breach of contract.
You start with the usual good hiring practices that also apply when hiring on-premise staff (and/or service providers) which include fact checking the provided resume, ask for education transcripts and certification numbers, check and call their references, interview, maybe even a background check or security screening etc. etc.
Then apply the carrot: pay fair value, offer attractive work, amazing colleagues, good working conditions and benefits etc. (if you pay peanuts you get monkeys.)
And the stick: breach the terms of your employment/service contract and we'll sick our lawyers on you and leave you bankrupt!
Unfortunately both the above become increasingly more difficult when crossing borders and timezones.
Once you decide on hiring somebody:
This question details what I usually ask my clients to do to establish remote access for me, which might be a starting point for you as well.
There is one systemic method of protecting yourself that comes to mind, which I have not seen mentioned.
Host your Linux instances as VM's on a virtualization hypervisor (VMware, Xenserver, Hyper-V, etc.).
DO NOT give the remote admin administrative access to the hypervisor. The remote admin would only get root access to the VM's themselves.
DO Implement a hypervisor based backup system (Unitrends, Veeam, vSphere Data Protection, etc.)
DO keep at least one snapshot per day of each Linux VM, going back as far in time as you feel is necessary.
DO NOT give the remote admin write access to the backup repositories.
If you do these things, you will have backup snapshots of each Linux instance over which the remote admin has no control. If the remote admin does something hinky, whether intentionally or accidentally, you can always mount a backup from before the hinkeness occurred to evaluate what happened and possibly recover to a clean state.
This won't be proof against a hypervisor side-channel attack, which could potentially be mounted from within a VM that the attacker has root access to.
If your backups don't go far enough back in time, this won't protect you.
You need to thoroughly trust whomever is in control of your hypervisor and the backup infrastructure.
If you're doing this in the cloud (AWS, Azure, etc.), the implementation details will differ, but the general concept would be the same.
In essence, divide responsibilities among parties who are not business partners with each other, in addition to only hiring people you trust.
Give him his own user account. Then find out exactly what he needs access to and grant just that access but nothing else. For example if he needs to reconfigure an Apache web server, use an ACL to give him write access to the Apache configuration files and configure sudo to let him restart the Apache service but not execute any other commands as root. As always, keep backups of anything that you're giving him access to (in this case, your Apache configuration files).
