Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am looking to use S3 as data storage service for a mail system.
The idea is to allow users to upload files that serve as attachments to their emails, which are then available to download from the web-service via a link to the file hosted on S3.

The issues I am facing:

  1. As users choose their file names, there will be collisions, so files on S3 must have uniquely generated filenames, or stored in separate folders, can users download the files with the original filename still using some API or configuration?
    (example: user uploads file dog.gif, stored on bucket as A3f23_dog.gif, download link returns file as dog.gif, possibly using HTTP headers)

  2. Do the number of files/folders contained in a single bucket (on the root) have any performance impact? Or do I need to do some distribution of files to folders etc.

  3. Can I have the files publicly available using URLs while still NOT allow enumeration of files in the bucket? (i.e file listing)

Thanks a lot, I hope this makes sense.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

I recently designed a similar system for storing reports -- here's what we found to work well:

  1. Use '/' in your keys to create a structured hierarchy. In your case, I would do something like: <user-id>/<year>/<month>/<day>/<message-id>/<user-defined-filename>.
  2. Use a pre-signed url to provide access to the file in S3 only to users that have access to the delivered email.

The main point of (1) is to ensure that you don't have conflicts between users -- you can restructure the hierarchy however you like, but the key point is to have unique identifiers as apart of the key structure (in this case, user-id and message-id), while maintaining the original filename as the last part of the structure; that last part of the structure is what most browsers use as the name for the downloaded file.

Notice that I included the date as part of the structure -- if you're going to have lots of objects, you're going to want to separate them into virtual "folders" in the S3 key space so you don't have to wait forever every time you need to look for files manually.

Point (2) provides access to the files only to the recipients of the message. Only the owner of the bucket (you) will be able to list files (unless you explicitly grant different permissions), but anyone with the link to a file can access it.

If you want to add more control over the links, create links that direct a user through your application, then if your app decides the user is actually authorized, create a pre-signed url and give it to the user. (This way you could hand out time-limited pre-signed urls, so you don't just give complete ever-lasting access via a link.)

share|improve this answer
    
Thanks, this was basically what i had in mind, thank you! –  Amit Ben Shahar Dec 1 '11 at 23:43
    
I would use a content hash (SHA1) for the file name, with parts of the hash as "folders". Email attachments get forwarded around a lot, so this will deduplicate them. Like this: /4a/56/28/46/3487654364325809bc326754df103864.txt –  rmalayter Jan 22 '12 at 12:13
    
@rmalayter - Clever idea ! –  Amit Ben Shahar Dec 4 '12 at 10:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.