I'm looking for a solution that would allow me to automatically capture the STDOUT/STDERR of a process running on Amazon EC2, and send it (remotely) to another server.
Sound simple, except:
- I will be using spot-instances, which means I don't control exactly when they start, and they can terminate at any minute (without proper shutdown)
- Because there's no shutdown, I can't write to a local file and transmit it (e.g. to s3) when the process is done.
- The output is not well structured (e.g. no tabulated fields in a log file), so "Standard" cloud logging solutions aren't trivial, and using one of the cloud databases is not ideal.
Couple of ideas I considered, but each has a problem:
- Appending to a file on "s3" is not possible, and rewriting files is too slow for logging.
- Sharing EBS volumes (as drives) is not possible to the best of my knowledge.
- Using "simple_db" is sounds too slow (and "simple_db" has been in Beta for years, so I'm not sure it's usable).
- Using SQS (e.g. one message per line of output?) is very slow.
- Redirecting to a network socket will fail if the connection drops for a second (e.g "myprogram 2>&1 | nc my.log.server 7070"
Perhaps there's a "syslog" solution with remote logging? but will that require a separate "on demand" instance to collect the information?
Any tips and ideas would be appreciated.