Hot answers tagged flume
I am the creator of LogZilla and we are just around the corner from releasing an Amazon EC2 Cloud solution of our software. I would love the opportunity to discuss your goals and the possibility to provide this solution for you. If you are interested, feel free to contact me. Although I'm sure you could use rsyslog, we are using syslog-ng with tcp (you ...
Yes, this is the standard use case for flume. The server with the log files will run a flume-node and another (or potentially the same) server will run a flume-master. The flume-nodes will discover the flume-master and from the flume-master you can execute commands like: exec config my-config 'tail("/path/to/logfile")' ...
Have you considered using Hadoop Archives? Think of them as tar files for HDFS. http://hadoop.apache.org/common/docs/r0.20.2/hadoop_archives.html
Media6degrees has come up with a fairy good solution to combine small files in Hadoop. You can use their jar straight out. http://www.jointhegrid.com/hadoop_filecrush/index.jsp
By far Flume is a better alternative.. Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.. Flume provides Continuous, high-performance, scalability Reliable data loading and data flow monitoring from feeds Logging directly to HDFS securely (made for hadoop) Flexible ...
Only top voted, non community-wiki answers of a minimum length are eligible