Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to upload my audit files to S3. Using s3cmd tools.

I just run this command :

sudo s3cmd put /var/log/audit/audit.log s3://audit-files

But I get a error message like this :

/var/log/audit/audit.log -> s3://audit-files/audit.log  [1 of 4]
 5120000 of 5116306   100% in   88s    56.80 kB/s  done
WARNING: MD5 Sums don't match!
WARNING: Retrying upload of /var/log/audit/audit.log
/var/log/audit/audit.log -> s3://audit-files/audit.log  [1 of 4]
  180224 of 5116306     3% in    4s    41.30 kB/s^CSee ya!

Why this is happening? I'm pretty new to Amazon S3.

share|improve this question
add comment

1 Answer

up vote 5 down vote accepted

Once the file has been uploaded, S3 computes the MD5 hash of the data. s3cmd computes the MD5 hash of the local file, and compares it to the hash declared by S3. These two hashes are not matching in your case, which means that the file stored in S3 is not the same as the local file.

My guess is that the log file is being appended to during the upload, and that this is causing the mismatch. Try first copying audit.log to another file, and upload the copy with s3cmd. Alternatively, only upload log files which have already been rotated.

share|improve this answer
    
Yes your correct. I wish to clarify other doubt: I have uploaded a auditd.log file to S3 its size is say 4Mb. After some time, automatically certain events are logged into auditd.log file. Now I wish that there are any commands to push only the content into S3 that is new in auditd.log file(rather than the whole file)? Is that possible through s3cmd? –  batman Jul 19 '12 at 8:00
1  
@batman It's not possible with S3 at all, because S3 doesn't support modifying or even appending files. I suggest that you setup log rotation (maybe on an hourly basis), and upload the log files after they're rotated. –  mgorven Jul 19 '12 at 16:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.