Our server has been hacked and I am trying to track down the cause. The server is being for SPAM purposes and it looks like IIS is being using to send the emails as the badmail and queue folder are full to bursting. The emails being sent have hte x-Mailer header set to "The Bat!", any ideas of things I can check for? I've checked the most common incoming ports (0-1055 and only HTTP / HTTPS) are open.
Hire a professional if it's important to find the cause as it could be anything. Pull the network plug, wipe and reload the server as soon as possible - pull the plug instantly if a pristine state for investigation isn't absolutely needed.
Beside the obvious spam, it's impossible to tell what other things like root kits are installed or what communication is really going on from the server to the outside world buried miles deep in anything from dns requests to whatnot.
Inbound open ports aren't usually the attack vector these days but if there's a vulnerability in that version of IIS or whatever is listening on those smtp and http ports then that could easily be it. Patch management problems and weaker internal security like easily compromised workstations with network access to the server seems like other likely causes.
From first wikipedia search result about the header:
First stop the SMTP service, this will stop any further activity. Next, delete the messages in the badmail and queue folders. Lastly, check the configuration of the SMTP service and make sure to close it to open relay. Resart the SMTP service and check for connections to it, and if any exist, see where they're coming from and check the folders again for messages being relayed through it.
Is this a Windows server? What SMTP platform is it, IIS? If so, what version (5,6,7)? Is this an Exchange server?