Log which combines log files from three different servers

Question

We have a three-server config with each server running nginx, and rails via Mongrel. Each rails installation has its own set of log files, in it's own 'log' folder, which is just a regular folder on the server. The log file i'm interested in is <RAILS APP>/log/production.log

Any request hitting the site is sent randomly to either of the three servers, and so the logging can be on any one of the three logs: effectively each log file has the log output for a third of the requests on the site.

Sometimes i need to look through the log for a specific request, to see how it was handled, and i always have to have all three open in less, and search for my request, which is a bit of a pita. What would be much nicer is to have a single log file which combines the three.

Now, you're probably thinking "Put them all in a shared folder and just have one log file". But, we don't want to do that, for fear of efficiency hits: we do use a shared tmp folder for certain things but it's a bit slow, and we don't want to have all of the logging going over http (which is what would happen if we used a shared folder). So, that isn't an option.

Any other suggestions? I'm thinking of a cron task which runs eg once a minute, copies the logs from server 1, 2 and 3 into the same place, and combines them IN A CLEVER WAY so that the ordering of the lines in the log is the same as if all three servers wrote out to the same log in the first place.

thanks, Max

EDIT - example log file output looks like this:

Feb 06 13:15:31 ec2-rails rails[21419]: ESC[4;36;1mUser Update (3.3ms)ESC[0m   ESC[0;1mUPDATE `users` SET `custom_signup_fields` = NULL, `all_license_sku_ids` = '--- \n- 1\n',
 `rank_criteria` = '--- \n:has_attached_a_doc_to_a_lesson?: false\n:has_downloaded_a_premade_lesson?: false\n:has_logged_into_lesson_viewer?: false\n:has_downloaded_a_custom_l
esson?: false\n:has_logged_in?: true\n:has_created_pupil_access?: false\n:has_created_a_lesson?: false\n:has_favorited_a_lesson?: false\n:has_viewed_pupil_usage_data?: false\n
:has_viewed_a_lesson?: true\n', `preferences` = '--- \n/: \n current-scheme-step: 5931\n current_product_id: 21\n' WHERE `id` = 52331ESC[0m
Feb 06 13:15:31 ec2-rails rails[21779]: ESC[4;36;1mInstrumentFamily Load (2.6ms)ESC[0m   ESC[0;1mSELECT `instrument_families`.id FROM `instrument_families` INNER JOIN users_in
strument_families ON instrument_families.id = users_instrument_families.instrument_family_id WHERE ((`users_instrument_families`.user_id = 51668)) ESC[0m
Feb 06 13:15:31 ec2-rails rails[21419]: ferret_create/update: User : 52331
Feb 06 13:15:31 ec2-rails rails[21419]: creating doc for class: User, id: 52331

One thing to note here is the number in square braces - this is the pid of the mongrel running rails - there are 16 of these on each server.

Answer 1

Whilst I understand that you are looking for a quick way to get 3 log files..."merged" in a way, you might be best just setting yourself up with a Log server and do more cool things with your logs. Like you stated, you wanted to check things in a log, no problem with a central logging server.

The logs can also monitor your server health.

I use Graylog2, it's sleek, simple to use and really effective to search in. (Graylog2 is a free, open-source alternative to Splunk).

1. https://www.graylog.org/download/

This would then accept log pushes from your server and you would see requests easily. (They even have an easy install script!)

Provided that you still want to do your original suggestion, then I recommend you use sort on the timestamps every minute. (Provided you have timestamps!)

Answer 2

In a lot of cluster environments you (additionally) send your log messages to a logging server with syslog.

The syslog will integrate the stream of messages in the order which they arrive and presto.

If the service doesn't support syslog natively as a logging protocol, you may be able to pipe to the logger program.

Answer 3

Can you make do with something like this on the central folder where the files has been gathered or is it to heavy?

cat *.log | sort | less

or

cat *.log | sort > combined.log

(Using sort to put them in order according to timestamp)

or maybe: Tail multiple remote files