7

I have a large S3 bucket with a nested "folder" structure containing (among other things) static .json and .md files. Theses files are being served by S3 as text/plain rather than the correct application/json and text/markdown.

I have updated the bucket defaults so that new uploads will have the correct content type.

What is the best way to walk the "tree" and update the content type for files matching a certain extension?

16

Here is an example how to do this with the aws cli tool. The cp tool allows the use of a recursive options, which I don't think the s3api tool can do. In this case, I'm fixing a bunch of SVGs. Remove the --dryrun options when you are ready to unleash it.

aws s3 cp \
      --exclude "*" \
      --include "*.svg" \
      --content-type="image/svg+xml"  \
      --metadata-directive="REPLACE" \
      --recursive \
      --dryrun \
       s3://mybucket/static/ \
       s3://mybucket/static/
  • I understand the underlying concept of the command. But will this delete the versions of the file since it is a replace directive? My bucket is configured with versioning and each file has some versions. Replacing it will delete all versions and make a new file or will it take into account all the versions during the replace? – Joze Jun 5 '18 at 8:17
  • That’s a good question. Definitely worth a test in a throwaway bucket. My hope is that the request to replace just metadata wouldn’t blow away versions but... – t1m0 Jun 5 '18 at 12:37
  • I did a test, and turns out that versions stay, however the metadata modified version becomes the latest version, the older versions have the previous metadata. I am trying to find a way to recursively modify the metadata of all versions now, that is without making a complicated script that will list every version and then modify its metadata... – Joze Jun 5 '18 at 13:35
  • This command resets acl. If you need non-default value, add --acl=... – Vasiliy Zverev Oct 21 '18 at 0:15
3

Unless you want to get your hands dirty using boto/python or another AWS API, I suspect your best bet would be to use $ aws s3api copy-object with the --content-type flag to copy an object to itself, setting the new content type.

Here's the documentation for s3api.

It goes without saying that you should test this first, before running it recursively on your whole bucket.

  • 1
    Regarding testing, use the "dry run" flag in Amazon CLI. – bart Dec 11 '16 at 3:10
3

As mentioned earlier, you can use aws tool by Amazon and use s3api to copy-object onto itself, and use metadata-directive=REPLACE to change the content-type.

I am putting this here, because sometimes you would want to iterate on filenames that are stored in database, and this is how you can do it through cli.

aws s3api copy-object \
          --content-type="application/vnd.android.package-archive" \
          --metadata-directive="REPLACE" \
          --copy-source "MYBUCKET/FILE.apk" \
          --bucket "MYBUCKET" \
          --key "FILE.apk" \
          --acl public-read
  • This replaces just one file. How do you execute this command against all files of type .apk? – bart Dec 11 '16 at 3:11
  • @bart see my answer for recursive way. – t1m0 Dec 14 '16 at 15:39
1

After some digging, I found that the s3cmd tool can do this. For example, to set JSON files to application/json:

s3cmd --recursive modify --acl-public \
       --add-header='content-type':'application/json' \
       --exclude '' --include '.json' \
       s3://bucket/

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.