1

We have a three-server config with each server running nginx, and rails via Mongrel. Each rails installation has its own set of log files, in it's own 'log' folder, which is just a regular folder on the server. The log file i'm interested in is <RAILS APP>/log/production.log

Any request hitting the site is sent randomly to either of the three servers, and so the logging can be on any one of the three logs: effectively each log file has the log output for a third of the requests on the site.

Sometimes i need to look through the log for a specific request, to see how it was handled, and i always have to have all three open in less, and search for my request, which is a bit of a pita. What would be much nicer is to have a single log file which combines the three.

Now, you're probably thinking "Put them all in a shared folder and just have one log file". But, we don't want to do that, for fear of efficiency hits: we do use a shared tmp folder for certain things but it's a bit slow, and we don't want to have all of the logging going over http (which is what would happen if we used a shared folder). So, that isn't an option.

Any other suggestions? I'm thinking of a cron task which runs eg once a minute, copies the logs from server 1, 2 and 3 into the same place, and combines them IN A CLEVER WAY so that the ordering of the lines in the log is the same as if all three servers wrote out to the same log in the first place.

thanks, Max

EDIT - example log file output looks like this:

Feb 06 13:15:31 ec2-rails rails[21419]: ESC[4;36;1mUser Update (3.3ms)ESC[0m   ESC[0;1mUPDATE `users` SET `custom_signup_fields` = NULL, `all_license_sku_ids` = '--- \n- 1\n',
 `rank_criteria` = '--- \n:has_attached_a_doc_to_a_lesson?: false\n:has_downloaded_a_premade_lesson?: false\n:has_logged_into_lesson_viewer?: false\n:has_downloaded_a_custom_l
esson?: false\n:has_logged_in?: true\n:has_created_pupil_access?: false\n:has_created_a_lesson?: false\n:has_favorited_a_lesson?: false\n:has_viewed_pupil_usage_data?: false\n
:has_viewed_a_lesson?: true\n', `preferences` = '--- \n/: \n current-scheme-step: 5931\n current_product_id: 21\n' WHERE `id` = 52331ESC[0m
Feb 06 13:15:31 ec2-rails rails[21779]: ESC[4;36;1mInstrumentFamily Load (2.6ms)ESC[0m   ESC[0;1mSELECT `instrument_families`.id FROM `instrument_families` INNER JOIN users_in
strument_families ON instrument_families.id = users_instrument_families.instrument_family_id WHERE ((`users_instrument_families`.user_id = 51668)) ESC[0m
Feb 06 13:15:31 ec2-rails rails[21419]: ferret_create/update: User : 52331
Feb 06 13:15:31 ec2-rails rails[21419]: creating doc for class: User, id: 52331

One thing to note here is the number in square braces - this is the pid of the mongrel running rails - there are 16 of these on each server.

3

Whilst I understand that you are looking for a quick way to get 3 log files..."merged" in a way, you might be best just setting yourself up with a Log server and do more cool things with your logs. Like you stated, you wanted to check things in a log, no problem with a central logging server.

The logs can also monitor your server health.

I use Graylog2, it's sleek, simple to use and really effective to search in. (Graylog2 is a free, open-source alternative to Splunk).

  1. https://www.graylog.org/download/

This would then accept log pushes from your server and you would see requests easily. (They even have an easy install script!)

Provided that you still want to do your original suggestion, then I recommend you use sort on the timestamps every minute. (Provided you have timestamps!)

| improve this answer | |
1

In a lot of cluster environments you (additionally) send your log messages to a logging server with syslog.

The syslog will integrate the stream of messages in the order which they arrive and presto.

If the service doesn't support syslog natively as a logging protocol, you may be able to pipe to the logger program.

| improve this answer | |
0

Can you make do with something like this on the central folder where the files has been gathered or is it to heavy?

cat *.log | sort | less

or

cat *.log | sort > combined.log

(Using sort to put them in order according to timestamp)

or maybe: Tail multiple remote files

| improve this answer | |
  • Sorting isn't easy as there's many lines per entry, and it's only the first line in each entry that has a timestamp. I will add some info about the format of the logs into my question, thanks for reading. – Max Williams Feb 6 '15 at 12:45
  • Not timestamps on each line? Sounds silly. Not just lines wrapping? Can't you configure it to put TS on each line? YW – kidmose Feb 6 '15 at 12:48
  • Actually, sorry, there is a timestamp on each line – Max Williams Feb 6 '15 at 13:14
  • fwiw I do cat *.log | cut -c 4- | sort -- not a great hack due to the multi-line issue but it works to find the event uuid then you can grep for it to see the full thing. – Meekohi Apr 30 '19 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.