Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a large set of log files that I need to extract data from. Is it possible to use Flume to read these files and dump them into an HDFS (Cassandra, or another data source) which I can then query?

The documentation seems to suggest it's all live event based log processing. I'm wondering if I'm missing some obvious process to just have flume read and process static log files from a directory.

share|improve this question
add comment

1 Answer 1

Yes, this is the standard use case for flume.

The server with the log files will run a flume-node and another (or potentially the same) server will run a flume-master. The flume-nodes will discover the flume-master and from the flume-master you can execute commands like:

exec config my-config 'tail("/path/to/logfile")' 'collectorSink("hdfs://path/to/hdfs-folder", [options])'

This creates a configuration that tells flume how to access the file (it can tail or read the entire file, other options are available) and where to put it.

Then it is a matter of pointing the configuration at a particular server:

exec map (server-hostname) my-config

There is more information in the flume user guide: http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.