Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This question already has an answer here:

The company I work for runs a series of ecommerce stores on a VPS. It's a WAMP stack, 50gb storage.

We use an archaic piece of ecommerce software which operates almost entirely client-side. When an order is taken, it writes it to disk and then we schedule a task to download the orders once every 10 minutes.

A few days ago, we ran out of disk space, which caused orders to fail to be written. I quickly hopped on to delete some old logs from the mailserver and freed up a couple of GB pretty quickly, but I wondered how we could fill up 50gb will nothing much more than logs.

Turns out, we didn't. Hidden deep within the c:\System Volume Information directory, we have a stack of pirated videos, which seem to have appeared (looking at the timestamps) over the past three weeks. Porn, American Sports, Australian cooking shows. A very odd collection. Doesn't look like an individual's personal tastes - more like the VPS is being used as a mule.

We have a 5-attempts and you're blocked policy on our FTP server (plus, there is no FTP account with access to that directory), and the windows user account has had it's password changed recently. The main avenues are sealed - and logs can verify that. I thought I'd watch and see if it happened again, and yes, another cooking show has appeared this morning.

I am the only one to know of this problem at my company, and only one of two with access to the VPS (the other being my boss, but no - it's not him).

So how is this happening?

Is there a vulnerability in some of the software on the VPS? Are the VPS owners peddling warez across our rented space? (can they do this?)

I don't want to delete the warez in case it is seen as a hostile action against this outside force, and they choose to retaliate.

What should I do? How do I troubleshoot this? Has this happened to anyone else before?

share|improve this question

marked as duplicate by Tom O'Connor Aug 14 '13 at 21:03

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

Save the movies if any of them are good and then reinstall :p – ThiefMaster Nov 17 '11 at 10:41

without auditing code (are you using custom store software?), you can't know if there's a bug that's being exploited (and even if you didn't find anything that doesn't mean it isn't there). Are you using custom SQL code, for example? With sanity checking and cleaning of input?

I'm assuming all your systems are fully up to date? Malware checks? Antivirus up to date?

Once someone cracks the system, they could have rootkitted it. No matter how you put in password changes or alterations, if something in the system has been changed to allow backdoor access, you're not going to keep it out. On top of that it could be logging your password changes and keystrokes, so you're just feeding more passwords to the attacker.

You can put in auditing to see where connections are coming from, but I doubt that'll help a heckuva lot.

In the end, you need to consider a wipe and reinstall from scratch. It's the only way you can trust the installation again and know that it's clean from trojan code, since the infection can mask itself once it's in control.

Scarier, if you're taking credit cards, this information could be getting swiped and you'll be liable for having customer identities swiped. If you're in the US this has ramifications where you need to notify customers of possible identity theft.

If this is a server that handles anything involving money you might need to consider calling in contractors to audit the system. Take images of the system for forensic use and wipe and reinstall. The longer you wait, the more liability you open yourself to.

To answer how it's happening, if the server is a dedicated server, it could be cracking something in your storefront (SQL injection, for example) vulnerability in Windows not patched anything web browse with that system? "Drive by" downloaders from a website. Run software on it that's not from the system? Could have been infected with something. Weak passwords. Audit them ever? And there's the possibility that you won't easily know how they did it. My bets are on the storefront software, especially if it's niche, as it's easy for developers to not clean input from the URL and open it to injection attacks. Or if it's using PHP open to an outside interface; you do keep that up to date? You don't mention if it uses something like a php administrative interface, but sloppy php coding can add an easy attack vector as well.

If you're at a total loss of how to fight this, seriously, hire outside help. No shame in getting help, and the rule of thumb is that once you're hacked, you CANNOT be sure it's fixed, and if customer data is flowing on that system you're opening yourself to liability and harming innocent customers. Plus if that system isn't partitioned off from other systems on the network, it could try to intercept other system's data.

share|improve this answer
+1 for the advice to go outside. Note the liability side here - your boss better getrs a lawyer involved, he may be in for a LOT of pain if this is not handled by the books, laws and your contracts for example with credit card companies. Criminal / Gross neglect CAN be waiting. THe advice to get outside help basically is 100% on track. As well as the "forenxig copy" advice. – TomTom Apr 28 '10 at 10:01
There are no CC transactions made on the VPS. If I were troubleshooting this on a dedicated server, I would use the tools from sysinternals, such as a procmon and rootkitrevealer. I'd have procmon set up to monitor activity to the directory to locate the offending process. But alas, those tools do not work on a virtual server. The eCommerce product is not niche. There is no DB backend. There is very little going on at the front end. Apache is up to date, windows updates are managed by the VPS provider. – Mikuso Apr 28 '10 at 10:48
That doesn't sound like a VPS to me, more like a shared hosting, or a managed server. In any case, if you're not managing the OS then you have to talk with those who are. – Javier Apr 28 '10 at 15:12

First off, I think you'll find there are at least three people with access to the VPS, not just the two you know about. I believe the system has been hacked and taken over. I also suspect that it is now hosting files for a P2P network, most likely torrents. You could have a search for *.torrent files but they're probably hidden from your view anyway.

You are no longer in control of the system. They have simply left you with that illusion. Had they not been so careless as to use up all that disk space they may well have got away with it for a lot longer. Incidentally, even a rudimentary monitoring system should have alerted you to diminishing disk space.

At this point you need to take Bart's advice and gather forensic evidence in the form of a system snapshot. Then wipe it completely and reinstall from scratch. Your backups at this point are probably worthless, as you have no real way of knowing when the hacking took place.

You need to look at what can be done to better secure your newly rebuilt system. For that matter, I'd consider rebuilding it on a new host and continue to use the existing one till it's ready, then do a cut-over.

share|improve this answer
So it's not worth trying to troubleshoot the weak spot? I don't like the idea of simply reinstalling onto a new server, as the hole may still exist - and then when the DNS switches over to the new IP, it will simply be compromised in exactly the same way. There is no sign of torrent activity. No mass-connections on strange ports. The files all appear to have been created within seconds of each other - meaning they were uploaded and extracted on the server, or downloaded very very quickly. Odd behaviour. Not torrent-like in nature. – Mikuso Apr 28 '10 at 10:42
Once you have taken your forensic image of the machine, you can make a copy of that image and put it on a totally isolated machine on a totally isolated network. Here you can troubleshoot to your heart's content, but don't whatever you do reconnect that image to the internet or your internal network, or ever trust any machine that's been on the same network as it, or any writable media (ie USB stick) that's been attached to it. – GAThrawn Apr 28 '10 at 11:01
@Mikuso, it's certainly worth troubleshooting the weak spot but not on the production machine. GAThrawn has the right idea. I'd go slightly further and set up an isolated network, perhaps using virtual machines. The reason is that you may well see things from another machine on the same network that are not immediately visible on the compromised one. Watch the network traffic to see what's being sent out unsolicited for starters (using Wireshark or similar). – John Gardeniers Apr 28 '10 at 12:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.