Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Our provider suspended our account with 5000+ clients and their sites. The reason was that the whole IP range was blacklisted by SpamHouse, but our provider did not explain what emails, form what IP and with what content was sent exactly.

So now we are just suspended and 5000+ sites are not working, and all users moaning. Big problems, as you can see.

First thing I have is that we can (or we must?) buy separate professional SMTP server, fully configurable with GUI, and integrated with lots of spam detection software... so bad outgoing emails will be stopped, and will be able to detect who is doing bad things -> and then close their accounts at our hosting.

Is it good idea, or do you have any other idea to effectively and quickly find solution for this problem? It must be 100% effective, ad it's the only way to convince our provider to un-suspend our account.

Please help urgently as we are losing our business....

Thanks in advance, Anton

P.S. This question duplicated to: http://stackoverflow.com/questions/3921906/im-hosting-reseller-and-i-got-spam-problems-please-help-to-solve

UPD. After all your comments - we are now looking for an expert to hire to server administrator position in our company.

share|improve this question
    
careers.stackoverflow.com - might help you find someone. –  Zypher Oct 13 '10 at 15:21

6 Answers 6

up vote 1 down vote accepted

It is not entirely clear for me what you are asking and you do not provide a lot of details about your current situation. What would help answering your question properly:

  • What OS is your current server running?
  • Do you use some kind of web host management package, like WHM?

When allowing others to send e-mail from or through your server, you should first make sure that:

  • Outgoing and relayed e-mails are tracked
  • Outgoing and relayed e-mails are linked to the user account sending them
  • Preferably: outgoing and relayed e-mail should be throttled or restricted to X messages per hour
  • You frequently scan your server for malicious e-mail sending scripts. There are plenty of good solutions for this. I like ConfigServer eXploit Scanner.
  • Setup SPF records for your clients.
  • Setup DomainKeys.

Judging from your question, I do not think extra servers will be a solution at all.

You have to get this under control and if you cannot, you should definately not be thinking about managing another server as well.

share|improve this answer
    
> What OS is your current server running? - centos. >Do you use some kind of web host management package, like WHM? - yes. –  meglio Oct 13 '10 at 9:00
    
OK, I see - it's more complicated then I expected. I'm not server administrator, I'm software developer only - and our company has VPS with managed support, but it seems like this support is useless as the tech guys have not even proposed to us to setup all this for us. So finally I think that I need to hire one who can set all this up prefossionally, correct? –  meglio Oct 13 '10 at 9:03
2  
If you really have 5000+ running on your system, I really urge to hire someone with enough expertise to handle this for you. Setting up a system is not the hard part, but maintaining it properly is. Do it, for your clients' sake. –  Aron Rotteveel Oct 13 '10 at 9:40
    
offtop: How can I contact people privately in serverfault.com? –  meglio Oct 13 '10 at 11:01
    
+1 "You have to get this under control and if you cannot, you should definitely not be thinking about managing another server as well." –  Chris S Oct 13 '10 at 12:37

No, sortry. YOu need to get your outgoing emails under control. Point.

You do NOT need full anto spam, though - it is good enough if you can mark every (!) outgoing email with an ID you can track back to the customer (account) that did send it, and then follow up legally with that (i.e. shut down your customers, recover expsenses from them).

But you obviouly have a problem here- or your customers have. Likely their sites being abused by bots.

share|improve this answer
    
We can hire person to check emails marked as spam, BUT how can we quickly automate this spam detection process? –  meglio Oct 13 '10 at 8:54
    
So, we need to hire an expert to correctly setup our server? But there is still another problem - nobody can tell us what email was makred as spam and caused us to block the IPs. They just suspended with no arguments, and SpamHouse does not tell us what message and from what email was abused. –  meglio Oct 13 '10 at 9:30
1  
It might not be that cut and dry. Most of the time the problem is so overwhelming that they don't track it to that level, as the originators more often than not are the spammers and ignore any measures. So they just cut you off, and make it YOUR problem, since you're the host and you should have something in place to stop this problem or mitigate it. Factor in the manpower tracking headers that could be fake, and if you aren't already separating queues or tagging outgoing mail... –  Bart Silverstrim Oct 13 '10 at 12:14
    
Exactly. Plus it makes sense like that. Your "problem" is that you dide not do your job - tag outgoing mail and make sure you make it your customers problem. Under normal circumstances you would cut off your customer now, redirect email to go through another email server (you have some in reserve, right?) and start asking your customer for collateral (as in: he has to pay every minute you spend and every dollar you loose). –  TomTom Oct 13 '10 at 13:22
    
Again, unfortunately, I'm not server administrator and it's not my job. We have managed server and nothing of advised above were not done by them. So finally what do I have now - I have now unhappy internet provider, unhappy users, and unhappy boss. The best would be to now hire someone for long-time relationship to sort out all the problems and maintain all our servers... We are open for suggestions. –  meglio Oct 13 '10 at 13:47

Catching a spammer in your network is a handjob more than anything else (speaking from my history). I would use a mail gateway (postfix will doit) with per-client queues. All the client servers should be set to use the mail gateway as the smarthost, so emails sent to localhost (either with sendmail or by connecting to localhost sendmail service) will reach the gateway. Also installing a firewall rule for redirecting external port 25 traffic to localhost mail service (or directly to smarthost) is a must. On the mail gateway you should freeze the emails on arrival for 2-3 minutes and use a cron script to count the number of emails from each queue. Unfreeze the emails from queues that hold a total number of emails lower than a threshold (spammers are speedy, they will usually send tons in seconds). Doing this manually for a while will allow you to spot some patterns in the mail traffic. A spamassassin milter will help as well. The per-client queue setup can be a little bit complex and will surely require some scripting but will help in locating the spammer and with getting things working for the other customers.

On the other hand, if things are really bad, you should disable smtp traffic for everyone and re-enable it per customer request. This means mail service downtime for them but more control on your end. Anyway, given the situation I don't think the newly caused "downtime" means a lot. And yes, you need a sysadmin. Not a whm guru. Not really a clickety-click task ;)

share|improve this answer
    
Is it right place to find one expert here, or it's not allowed to request contact & pricing info etc? –  meglio Oct 13 '10 at 11:29
    
i suppose you can contact someone in private using the information listed in his profile. –  sysfault Oct 13 '10 at 11:47
1  
"On the mail gateway you should freeze the emails on arrival for 2-3 minutes and use a cron script to count the number of emails from each queue." That is a bizarre way of doing it. If you use exim as your gateway, you use ACLs to handle this sort of thing. –  brent Oct 13 '10 at 14:32
    
i don't know much of exim's acl mechanism but as far as i remember acls are for filtering so you must have some rules in mind when writing the acl. if i don't know what's going on and can't come up with some filtering rules other than blinds then i'd go for the most sane: volume-based. since i don't want to send anything from a queue while computing the volume i'll hold the messages. when i have some counters i can take a decision: 3 mails per minute, unfreeze that queue. 5 counter checks OK, unfreeze the queue for good. 10 mails per second, handjob verifications, keep it frozen. –  sysfault Oct 13 '10 at 14:47

If you are looking for a simple to setup and manage mail server with antispam and antivirus + inbound and outbound tracking, setup something running postfix + spamassassin + clamav + MailScanner - a lot of the install work will actually be done by the MailScanner package and it works well with CentOS - I have 8 Mail Servers configured this way. Start out here for MailScanner and here for a howto

share|improve this answer

I am wondering what you're already doing to mitigate the problem.

You're saying Spamhouse won't tell you about the emails. That routine in my experience. Headers can be forged, and it can take a lot of time to trace things back. It's easier just to blacklist the originator of the reported spam and put the onus on them to prove they're cleaned up. Often the originator is A) a spammer that is going to move to another block or B) some unwitting user that infected their system with a bot, so their ISP already mitigates the risk by blocking all outgoing port 25 traffic unless it comes from their mail server's IP.

So what are you doing now to mitigate the problem? Do you tag outgoing mail with a special ID for the customer? Do they get their own queues? Do you monitor outgoing mail volume, and restrict sending rates so they can't send unreasonable volumes of mail out? Do you block outgoing port 25 traffic unless the customer has a specific request for email service? Run monitoring reports on bandwidth used by your customers to find anomalies, in case their host server is compromised? Or are you literally just hosting, letting your users do whatever and however they please?

I would think that if you haven't done anything before now, you're going to need to major reworking of your logical layout to the network. You'll need a mail server (maybe more depending on your customers,) lock outgoing traffic for SMTP to only customers that need email outside your network, put in monitoring scripts and rate limiting, and you might want to consider outgoing spam scanning, although this can lead to headaches for outgoing customer mail if it's incorrectly flagged as spam and their mail just disappears into the ether. I'd personally advise against that or at least set it to catch the most obvious spam, or alternatively offer it as an option for customers to opt in or out of using. With 5,000 customers hosted, you may very well need a system administrator to oversee your spam/email configuration. Following up on reports and auditing your hosted user's bandwidth can take up quite a bit of time to maintain.

share|improve this answer
    
Bart, I understood the goal. We are now looking to hire server administrator for all our servers and related administration needs. Please let me know if you are interested, or if you have anyone interested - but only people with good 5+ years of experience please. Thanks. –  meglio Oct 13 '10 at 13:04
    
@Meglio: That would be nice but I don't even know where you are :-) –  Bart Silverstrim Oct 13 '10 at 13:09
    
Well, this is remote job - not in office. My skype is meglio.public. I cannot find where to put this info in my profile at serverfault.com –  meglio Oct 13 '10 at 13:25
    
You can add information to your profile by clicking your name and editing your profile information. Add info like personal website, description, etc. where you can add what you like. –  Bart Silverstrim Oct 13 '10 at 14:01
    
Thanks, I have added my contacts to the profile. –  meglio Oct 13 '10 at 14:05

This is not an answer to deal with actually scanning the mail but this is a method for potentially catching the source. If your hosting allows PHP and your mail gateway is simply sendmail, you likely don't have any clue what's passing through it or where it's coming from.

Remember this is for PHP + sendmail. It involves setting up a file that is prepended to all PHP scripts that sets environment variables (prepend.php) that are then logged when mail is sent using PHP by calling a wrapper for sendmail (sendmailwrap) rather than calling sendmail directly.

In your php.ini,

auto_prepend_file = /usr/local/etc/prepend.php
sendmail_path = /usr/local/bin/sendmailwrap

Content of /usr/local/etc/prepend.php

<?php
putenv("PHP_HTTP_HOST=".@$_SERVER["HTTP_HOST"]);
putenv("PHP_SCRIPT_NAME=".@$_SERVER["SCRIPT_NAME"]);
putenv("PHP_SCRIPT_FILENAME=".@$_SERVER["SCRIPT_FILENAME"]);
putenv("PHP_DOCUMENT_ROOT=".@$_SERVER["DOCUMENT_ROOT"]);
putenv("PHP_REMOTE_ADDR=".@$_SERVER["REMOTE_ADDR"]);

Content of /usr/local/bin/sendmailwrap:

#!/bin/sh
logger -p mail.info sendmail-wrapper-php: site=${PHP_HTTP_HOST}, client=${PHP_REMOTE_ADDR}, script=${PHP_SCRIPT_NAME}, pwd=${PWD}, uid=${UID}, user=$(whoami)
/usr/sbin/sendmail -t -i $*

The point of this is that your sendmail log file will now have something like the following:

Oct 13 10:10:10 host user: sendmail-wrapper-php: site=www.example.com, client=10.10.10.5, script=/spam.php, pwd=/var/www/example.com/htdocs, uid=1001, user=www

If this is setup incorrectly you could cripple your clients' ability to mail so understand the steps and test before putting in production.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.