0

Ok, first of all let me describe my working situation so that is is clear why this is a question pertaining to system administration. I work at a larger IT solution company that is running operations for a large scale email service (5 million users) for a major telco company. One of the major issues with this portal is users being compromised by bot nets so that they can be used to send spam mails in the order of probably 100 000s per day. The complaint email address for the portal receives ARF reports at a rate of maybe 1 per minute amounting to maybe 1000-2000 per day. Any of these reports would have to be looked at manually, analyzed processed and archived. Since there is no man-power assigned to this tasks, the reports are basically ignored resulting in increase in spam to a level that the portals email server are all blacklisted on time or another. The reason why I would like to have a framework to automate this would be to identify the legit reports to be processed automatically while some reports still have to be analyzed by a human which could be delegated to off-shore people if the tasks are sufficiently easy to describe and to execute. I should stress that I do not need help implementing the steps, most of these are 1 line shells scripts plus a few ones accessing external tools, I am searching for a tool to string together the different steps.

I was thinking of something along the lines of Gearman meets procmail meets Jenkins meets RequestTracker meets Ansible (meets Actor model if you like), but I assume that such a tool is not available. There have to be some tools to automate at least some parts of this, since the complaint processing has to be an issue on any medium to major scale email portal where more than one person is running the email system.

(I hope this explanation is sufficient to motivate the relevance of the question, if not, I really do not what is. Or maybe this question is place wrong in this SO portal, but if so which one is applicable?)

I would to automate a few tasks that process files in some way, do some decisions, call some scripts and in some cases have some tasks that require human intervention.

My example use case is the following (when I do the whole thing by hand)

  • receive an email spam complaint, which I open in editor to be able to search through
  • decide if the email fits some criteria (e.g. it is really sent by a user in our systems, isn't older than 5 days etc)
  • extract the username of the user that has sent the mail
  • search the customer database if the user is already disabled
  • disable the user if still active
  • search mail queues for other offending emails by the same user
  • move the mails to a holding area to analyze later
  • send the customer service people an email that a user was disabled (probably with a collect list of disabled users once a day)

Most of these steps could be easily automated with e.g. perl scripts, but I have no idea how to automate the consecutive steps, especially if there are conditions and preconditions and some tasks could be collected together and run if there are a few todos already pending (e.g. search through the queues only if either 10 users have been found or the task wasn't run in the last hour or so).

Some decisions may require human intervention (e.g. looking at the mails that could not be identified automatically) or tuning of the decision process so that it would be helpful to be able to run some steps again with a changed script.

I assume that it would be possible to have some kind of job scheduler, but I wonder how to define the necessary steps and conditions and preconditions.

(Please note that I do not search for a tool that can do the described tasks, most of these are easily implemented as scripts, but a tool to organize the sequence of tasks that have to be done after each other for a given task data)

I would prefer a FOSS solution for this running on Linux.

3
  • 1
    Seems to me that you are asking us how to write to design and write a program. Which IMO is mostly out of scope here. It is very unlikely that there is any pre-existing software that you will find that can do this for you. You may be better off writing a spec here and just hiring a programmer.
    – Zoredache
    Dec 28, 2013 at 1:16
  • If there is no such program available, I'm out of luck, however I figure this must be a problem that has come up in one way or another. Actually it could be a worthwhile open source project to start, if there is none available (if I can find the time ...) Dec 28, 2013 at 1:29
  • I think you've already highlighted the solutions (Gearman, Jenkins, RT, Ansible) but all you need to do is glue them all together. Welcome to DevOps. -- Perhaps what you're looking for is a workflow engine (good google keywords, those). Jan 4, 2014 at 23:21

1 Answer 1

1

This is tough to answer definitively because there's a few moving parts and some places where human intervention is probably needed. I'd recommend starting at the top with a scripting language you're either comfortable with or is technically preferred (e.g. if you're a Windows shop then PowerShell, if you're just about anything else -- use anything else). Group your logical parts that don't need to be looked at by a human and script it. Generate reports or populate a database somewhere and slowly refine the process. I don't know of one single utility that you could use outside of a carefully written script for your particular environment.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .