Logging and Log-viewing tool for cloud?

Question

I have an app which simply logs to a file. I am planning to deploy this app on cloud. If I deploy it on AmazonEC2 instances (multiple), I would want to monitor logs of every instance. Something which shows the logs in a structured manner and alerts if anything grave is logged.

any ideas ?

ps : if this is a repeat question, feel free to link that question here

Answer 1

I'm using syslog-ng for that, you can configure syslog-ng on your local machine to read from the destination file (or you can make your app log to syslog) and then send it to a destination machine where all the logs are either merged into one or divided in different files and even a directory structure.

On the client side

source yourservice {
    file("/var/log/xxx.log" follow_freq(1) flags(no-parse));
};

destination yourcollector {
    tcp("xx.xx.xx.xx" port(65140));
};

log { source(yourservice); destination(yourcollector); };

Then on the collector server

source yourservice {
    tcp(port(65140) keep-alive(yes));
};

destination yourservice_dir { file("/var/log/yourservice/$YEAR/$MONTH/yourlog-$YEAR-$MONTH-$DAY" create_dirs(yes) template("$MESSAGE\n") template_escape(no));};

log { source(yourservice); destination(yourservice_dir); };

You can change the parameters on the destination to add or remove variables to divide it in one or more logs