0

I've been using the s3 logs to track downloads for a podcast (mp3 files) I currently have 23 files I actually care about and I only need to know information about GET operations (download requests being fulfilled.)

This works fine as is, but I'm also getting additional log files generated from asking for the files I care about. Is there a way to configure the bucket logging policy such that only the GET requests are recorded? Even better, can I limit based on some type of url request pattern? (e.g. *.mp3)

The default configuration is giving me 10k+ extra files telling me that I asked for the log files and/or that a log file was created (very meta!) Downloading and filtering out all these extra files takes a significant amount of time.

1

There isn't a way to restrict what is logged.

The documentation mentions the behavior it sounds like you are describing, as a reason why you shouldn't configure a bucket to write its logs to itself.

When your source bucket and target bucket are the same bucket, additional logs are created for the logs that are written to the bucket. This behavior might not be ideal for your use case because it could result in a small increase in your storage billing. In addition, the extra logs about logs might make it harder to find the log that you're looking for.

https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html

  • That makes a lot of sense... e.g. to a bucket that doesn't have logging enabled? – Dave K Sep 25 '19 at 16:07
  • 1
    @DaveK Correct. I typically create a bucket per region for log collection and then set the log prefix to s3/source-bucket-name/ for each source bucket that writes logs there, to keep the logs for each bucket under a different key prefix. – Michael - sqlbot Sep 25 '19 at 16:46
  • Just documenting how this ended up working out for me: Bucket 1 has my MP3 files (files I'm trying to track.) Bucket 2 only has my logs. Bucket 1 has logging enabled, but logs are written to Bucket 2. Bucket 2 does NOT have logging enabled which prevents creating extraneous logs. – Dave K Oct 16 '19 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.