Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have generated a couple of logs across different machines. Each log representing an action performed by user. The log is a text file generated in php by appending using file_put_contents . A line in the log represent action performed by user, time etc. There are many logs like this for each and every user action. Now I want to make a user profile which contain what all actions he did by going through these logs in ascending order of time. How do I do this?

Also I am planning to store these user profile in a MYSQL db. We have a large data of over 50GB of text from which I have to make these profiles. Is this ok?

Is there any tool that let me get business intelligence from these logs. (Suppose if i want to see how many users did a specific action today etc).

share|improve this question

1 Answer 1

up vote 1 down vote accepted

50GB might be a bit much for a random MySQL server. Depending on what you are planning on doing, you might want to consider multiple MySQL servers (shards). It's definitely possible to get a server with enough memory to keep your entire dataset in memory, which would help things.

If you are just storing these in text files, you would need some custom application to go through the logs for you. Nothing off the shelf will directly work for that.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.