We have a three-server config with each server running nginx, and rails via Mongrel. Each rails installation has its own set of log files, in it's own 'log' folder, which is just a regular folder on the server. The log file i'm interested in is
Any request hitting the site is sent randomly to either of the three servers, and so the logging can be on any one of the three logs: effectively each log file has the log output for a third of the requests on the site.
Sometimes i need to look through the log for a specific request, to see how it was handled, and i always have to have all three open in less, and search for my request, which is a bit of a pita. What would be much nicer is to have a single log file which combines the three.
Now, you're probably thinking "Put them all in a shared folder and just have one log file". But, we don't want to do that, for fear of efficiency hits: we do use a shared tmp folder for certain things but it's a bit slow, and we don't want to have all of the logging going over http (which is what would happen if we used a shared folder). So, that isn't an option.
Any other suggestions? I'm thinking of a cron task which runs eg once a minute, copies the logs from server 1, 2 and 3 into the same place, and combines them IN A CLEVER WAY so that the ordering of the lines in the log is the same as if all three servers wrote out to the same log in the first place.
EDIT - example log file output looks like this:
Feb 06 13:15:31 ec2-rails rails: ESC[4;36;1mUser Update (3.3ms)ESC[0m ESC[0;1mUPDATE `users` SET `custom_signup_fields` = NULL, `all_license_sku_ids` = '--- \n- 1\n', `rank_criteria` = '--- \n:has_attached_a_doc_to_a_lesson?: false\n:has_downloaded_a_premade_lesson?: false\n:has_logged_into_lesson_viewer?: false\n:has_downloaded_a_custom_l esson?: false\n:has_logged_in?: true\n:has_created_pupil_access?: false\n:has_created_a_lesson?: false\n:has_favorited_a_lesson?: false\n:has_viewed_pupil_usage_data?: false\n :has_viewed_a_lesson?: true\n', `preferences` = '--- \n/: \n current-scheme-step: 5931\n current_product_id: 21\n' WHERE `id` = 52331ESC[0m Feb 06 13:15:31 ec2-rails rails: ESC[4;36;1mInstrumentFamily Load (2.6ms)ESC[0m ESC[0;1mSELECT `instrument_families`.id FROM `instrument_families` INNER JOIN users_in strument_families ON instrument_families.id = users_instrument_families.instrument_family_id WHERE ((`users_instrument_families`.user_id = 51668)) ESC[0m Feb 06 13:15:31 ec2-rails rails: ferret_create/update: User : 52331 Feb 06 13:15:31 ec2-rails rails: creating doc for class: User, id: 52331
One thing to note here is the number in square braces - this is the pid of the mongrel running rails - there are 16 of these on each server.